I've only been asked a few times, but back in highschool, people just assumed/accused me of it, and that was long before I ever started smoking. It used to bother me, but now if anyone ever does ask, I'm just like "hell yes, I do." Well, it depends on who asks, of course. Like, if my boss or potential employer were to ask me, I wouldn't say yes. But I don't really care who else knows.
Weed is great. It relieves stress and helps me unwind at the end of the day. It doesn't make me have to take a piss every 10 minutes like alcohol does, and it will never kill me.