rarely speaking may be a clue. If you don't speak to the ones you are trying to add , then you are a stranger, and speaking as a woman, I am very cautious about adding strangers to my contacts, especially if they are somebody I see around me but really do not know. It is a dangerous world "out there" and "stranger danger" is a real "thing". don't take it personally! Finding others who share the same interests ( what are your most intense and enjoyable hobbies, likes, etc?) can build common ground and trust to go forward with. this goes for making friends first (of all sorts) before trying to build closer relationships. It all takes time, and many many small steps to build trust and confidence, to understand each other.
( I'm old, it has taken a lot of practice, it is a learning curve thing, and I am finally doing better in social stuff)