I had a semi-relationship with an Aspie, and it was very difficult. Of course, part of this had to do with issues that didn't directly relate to the Asperger's. He was having a tough time with his social life for reasons beyond his control, though his AS didn't help matters. He was feeling very confused and depressed. But there were also some Asperger's traits that made matters harder. I've learnt to read NT facial expressions, but I couldn't read his. It was hard to know when he wanted my company and when he was getting fed up. He talked a great deal about his own feelings but would become distant or coldly logical when I talked about mine. It was certainly a challenge, though he was intrinsically a kind person and I think it was worth it.
I think NTs are generally a little easier, because you can use social conventions to predict their behavior even if you don't follow such conventions yourself. Then again, everyone is an individual whether NT or AS and I'm sure there are exceptions.