At your age I also was yet to experience my first crush - it happened at 22.
No surprise you don't fall in love with people who ask you out - it doesn't work this way. A crush is when you see someone or get to know someone and at some point of time you realize you want them to yourself/want to have sex with them. It might happen fast (love at first sight/met) or gradually (from friendship to love, love through attachment).
Once someone falls in love they are in the difficult position of making the crush love them back (that's why they say "whoever falls in love first - loses", but I don't believe it - being in love feels awesome and "hunting" the crush is a fun game, although it comes with the risk of broken heart).
Your previous dates were not able to accomplish that, probably because they hurried too much or couldn't communicate with you properly. With enough patience and building the relationship slowly you would eventually feel love towards them.
BTW. What's wrong with taking you to their house? I invited my current crush into my house on our 2nd or 3rd date. We ate, watched some movies and that's it. No big deal. Truth is I wanted more but the mood wasn't right so I did nothing. Aren't you a little bit too paranoid?