When I was a kid and needed alone time, I would just space out and daydream where ever I was right next to family or whoever else I was with. From about age 12 and into my 20's I needed plenty of alone time even when being around loved ones. I withdrew to my room or other place where I could be alone. My need to be alone when being around family wasn't really due to being drained by them, it was due to wanting (needing) to be on my own and think about stuff and daydream in peace. But if I couldn't, then I'd feel drained. And get irritable.
Somewhere in my 30's I only needed to have alone time when with loved ones (my mother) if I was very agitated about something, which happened very rarely. I don't tend to need alone time anymore when I'm home. Her presence doesn't drain me, while my father's sometimes can.
It's only around non-immediate family that I easily feel drained, and I have since my teen years. I never got drained from simple interaction (like a typical school day) as a kid. I have experienced several times throughout the last decade that people can drain me as easily online as IRL. But these aren't loved ones, so off topic.
I did have one person once who did drain me despite how I felt about him. I don't know if it was his personality or the time we spent together actively being together (as opposed to the easy going being with family where we just go about our every day lives), but after some hours with him, I was almost always ready for bed, dead tired.
So it depends on who it is, how they affect me.
tl;dr: My mom, never. My father, sometimes. Relatives, friends, acquaintances, yes.
I think it might be the case because when at home we're relaxing and doing chores, and interests and whatnot, and only sometimes actively interacting and even when we do, we know each other so well, it's just easy going. But with everyone else it's direct interaction without stop, and they're always focused on you even when the focus isn't meant to be negative.