Insecure women who need frequent affirmation may feel neglected and alone in an aspie relationship; being there for someone emotionally can be a second language for many of us. Id recommend watching a movie or reading some literature on aspergers before getting too involved. The movie "Adam" is a great movie that shows some of the challenges and pitfalls of aspie relationships, but the character in the movie cannot represent all aspies, as we are all different.
Based on his behavior, such as leaving home and joining you for drinks after be thought he wasn't invited, I'd say this guy likes you very much and would probably walk miles to have a drink with you. At the same time, I can't say I know he likes you for the person you truly are. There will be many things about you he may not currently, or ever, fully understand. If you don't mind someone loving the mystery that is you, maybe this could work. If you really need to be validated and understood on a regular basis, you're probably going to find this prospect doesn't fit the bill.
If you can though, I can tell you that you'll never know someone more honest, loyal, and more commited (within their ability) to you than an aspie; that's just my opinion though. Good luck either way!
_________________
We are not so different from potted plants in that, if given everything we need to be properly nourished, the outcome can be incredibly contrary to when we are not. A flower won't grow in flour, and neither can we.