Sorry to hear you have had a difficult time. I am 35, and only got my diagnosis a year ago so I know how it feels to think, "why didn't that occur to anyone sooner?" In my case, nobody was diagnosing girls with Asperger's in my hometown when I was growing up, so by the time I was a teenager, bipolar was the dx that I got. It took me fourteen years to get my bachelors, because of social dysfunction and difficulty with organization and "life skills". So don't give up on grad school just yet, if it is what you want--take some time and look for a program which better meets your needs. It helps to be able to outline what you need, so you can give the list to a prospective advisor. If they say they can't work with you to meet your needs, find another program with faculty that will. Smaller schools tend to have more flexibility than larger universities.
Welcome & good luck!