Sounds like he was just kinda leading you on, reminds me a bit of the first guy I dated when I was in college...or was lead on to think he was dating me. I guess he never said he was my boyfriend, but he was acting like it very initially....but after like the first few times we met he'd only want to hang out for a while to do sex stuff, and whenever I texted to see if he wanted to like see a movie or just go out somewhere. He always had a lame excuse why he couldn't.
At the time though I really wanted a boyfriend and I was into him, so even when he had me walk to his apartment in the snow(I figured I'd be staying overnight at his place) and we had sex and he sent me on my way even later at night to walk in now very deep snow back to my dorm I still persisted in putting in effort to see him. It didn't all really dawn on me till communication got less frequent till the point he just sorta ghosted me. I certainly felt stupid for spending so much time thinking about him and such when he didn't even care about me.
I don't want to derail at all with my life story, but yeah better to move on sooner then keep putting energy into a person that doesn't reciprocate. So I think you made the right choice breaking it off.
_________________
We won't go back.