Psychologists and psychiatrists can be very intimidating. Don't let them break or deter you. If you feel their view is incorrect seek a second opinion. Anyways they are known to be wrong.
Since you are an adult an interview with family and school officials along with education records should be included in your assessment. Asperger's, autism, HFA etc don't just appear in one's life. Being on the spectrum is evident in early childhood. This needs to be considered in determining whether your problems are an ASD and/or psychiatric nature. I personally would not take medication until then. Any responsible clinician would never ever make a diagnosis such as this in one appointment, ever. An assessment should be at least 5 to 10 visits.
5 psychologists, yikes!
Good luck.