It might (hypothetically) stem from how you communicate, not that I'm making an accusation given that I'm not familiar with how you communicate.
I also get accused of arguing or trying to argue in situations where I don't really feel that's true. I also notice people respond as though they're seeking to argue in situations where I wasn't really intending on getting drawn into a debate, but if I accuse them of seeking to argue it's possible that I'm just doing the same thing.
Laying one's thoughts and the reasoning behind them out in a clear manner is often grounds for being accused of trying argue and it's not entirely unfair given that's how one constructs a reasonable argument.
If both parties in a discussion are communicating their perspectives in that manner it might appear their discussion is an argument or a debate. Ultimately the boundary between an argument/debate and a discussion isn't discrete. Beyond that, sometimes if someone's making an argument poorly and your response eviscerates their poor argument, accusing you of trying to argue can serve to put you on the back foot because now you're worried about being rude instead of being correct and now you're less likely to address their humbug (aka BS).
_________________
The Party told you to reject the evidence of your eyes and ears. It was their final, most essential command.
You can't advance to the next level without stomping on a few Koopas.