You are definitely not alone, I too have sat alone while at school for as long as I can remember, I would usually just sit and watch the other kids play. If i wasnt being bullied (which was pretty much all the time) a couple of kids would come to me and include me in their games, but it didn't happen very often. I've always felt like the outcast kid, especially when I got to high school, it was still the same thing, I would sit in the hallway close to the cafeteria and watch all the students walk by me as if I were a ghost. The bullying never stopped either until I started to defend myself, I found that once I started to defend myself against the bullying that people started to respect me a bit more. I had a couple close friends in high school, moreso than what I have had at one given time throughout my life. After a while I found myself hanging out with a few more people and by the end of my four years of highschool I had a nice little circle of friends, but only hung out with 2 or 3 after school every once in a while.
My home life was pretty much the same growing up, I would always be alone in my room playing with my toys or playing my video games. My mom would tell me to go outside to play and get some fresh air, she also told me to go out and play with other kids and make some friends. She still tells me that I should get out and socialize. Today I live with my gf and we go out a fair bit, she has helped me out and will continue to help more, I feel she is the only person in my life that understands me the most. If it wasnt for her I would still be stuck in my room doing nothing. I still find it very difficult to communicate and socialize, but with her help i think it will get easier.