Plainly you're able to interpret this with a level head and have plenty of overarching concern for all your friends so yeah I'd say a four year age gap dating at that age is pretty crazy. As a rule, as long as there's any willingness on any aspie's part to please others they'll usually be willing to hear you out; I don't see why he'd necessarily be any different. Jobs don't absolutely change anybody's identity, they usually just mess with that. I can't exactly understand anxiety over older women since I get along just fine with them but I don't think you'd have to rope anybody else into the situation to get that phobia resolved.
While I get the sense you're seeking validation for your & your friend's apprehension, I will say that in the present situation I wouldn't say anybody's in the wrong so much as in the now - best to be upfront & reconciled with each other. For what it's worth, I think picking the brain of anybody that strange stands to teach everybody a lot about their respective human nature.
I can't represent any of you three but platonic = great as far as I'm concerned. With the kind of work I'm into, not to mention my batshit insane past, trusting anybody takes quite a lot of deliberation and marijuana. If I don't trust someone to that degree, chances are I never really met them according to my standards in square one.
_________________
"Standing on a well-chilled cinder, we see the fading of the suns, and try to recall the vanished brilliance of the origin of the worlds."
-Georges Lemaitre
"I fly through hyperspace, in my green computer interface"
-Gem Tos