I would also be a bit narked if someone kept fiddling with their phone while we were together -- I think that's just plain old rude. But did he have a good excuse, such as the bloke he was meeting was running late, or asking for directions, or they had to change venue at short notice and were trying to fix a place to meet, or something like that? If so, I'd probably make an exception (but I'd still be annoyed, secretly, because if it were me in that position, I'd make my apologies and try to sort it out in one phone call, then return to paying attention to my companion -- if only others were all as considerate as me ...
).
The phoning you at two am is more than a bit odd, unless it was an emergency (a mental health crisis or something like that). Like you, I would assume a fair degree of intimacy with someone before I felt able to do that.
All in all, and given his confession that he's a 'loner', he sounds like one of those people who will dance unpredictably in and out of your life as it suits him. If you can accept that or think that you can adjust your expectations of 'friendship' in this case (and some people are just like that, and don't mean any harm by it), then great.
I've had acquaintances like this, and they're puzzling at first, seeming to 'blow hot and cold', until you come to appreciate that this is their form of friendship and then you can start to regard it as a random but positive occurrence in your life. It is difficult for AS people to tolerate such capriciousness, though, but sometimes (and depending on the person) an element of the unexpected can be fun.
Otherwise if you really can't tolerate it then cut him loose: He probably won't feel the loss and nor (by the sound of it) will you.