There's nothing wrong with getting annoyed by people, whether they are autistic or not. I get annoyed by people who gets too much up in my personal space, and some aspies don't get how uncomfortable it is when they're standing too close. Especially my brother, who are very intrusive in that way. I love him and all, but sometimes I just want to put him in a cage.
I am one of those people who can talk way too much, and about things that are only interesting to me. I encourage people to let me know if I do it, because I'm not being annoying on purpose, and it's nice if people can point it out, so that I can be less annoying to others in the future.
It might be that your coworker have the same desire to be more likeable, or he doesn't care at all. Either way, try to tell him to lay off in a kind way. Just let him know that it's really hard for you to focus when he's talking all the time. If you just say "you're really annoying", he might get defensive about it and it won't lead anywhere.