For an Aspie saying "sorry" can be difficult, especially when you KNOW you haven't done anything wrong.
But I've realized something lately after dealing with my ex (who I also think has Aspergers).
Apologizing isn't just done when you're wrong. Sometimes when you do the right thing, it still hurts people. We can't always assume others will "get over it" just because what you did wasn't logically WRONG per say. It can be hard for an Aspie to think of what others are feeling, but if you know you've hurt someone, it's always better to apologize and show you care. My ex crushed my heart over and over (he would never realize what he was doing hurt me, though, until I told him). I can understand that since he also has Aspergers, but it was so frustrating because he would just defend himself rather than aknowledge the pain I was in after confronting him about it.
This is a very common Aspie trait, and one of the most crippling ones. Personally, I've been trying to be more sympathetic towards others and see things from their perspectives. At first, you'll feel like you're faking it, but after long enough it'll become easier and more natural.