I don't think that generically having a disability is enough, though I wouldn't limit it strictly to having physical disability, either. I.e. someone could have problems with balance & vertigo, but otherwise be physically fine. And, so on.
I see it is as being at the discretion of common sense & decency. If there's a reason beyond being disabled in an unspecified way, then I say that's fine. And, if someone else needs it more, then let them have it.
Of course, working out who needs it, or needs it more, may be difficult/treacherous to work out in real-time.
As far as the spectrum... trying to think up some scenarios...
1) for those with major tactile issues: maybe the bus is crowded and the only way to sit without bumping shoulders (and bare skin -- say, it's a hot day), is in one of the handicapped seats
2) have a hard time determining when to get off the bus, and so need to be in a seat where one can see out the front
3) maybe it's hot and the windows are down, and that's letting a lot of exhaust smell into the back
4) someone in a nearby seat has overpowering perfume on
5) different vibrations in different seats
Those seem legitimate to me, though I can imagine them not being believed by other passengers. So, that could be difficult.
It just occurred to me, about having trouble knowing when to get off -- what do blind people do? Does the driver let them know? (Another reason to sit up front.)