I have a disease called spinocerebellar ataxia. The onset of my disease wasn't until my late-20s, but I always knew I had a 50/50 chance of having it. So I have experienced some of what you're talking about, but some of it I haven't.
When you have a serious neurological disease, you just can't do as much stuff as other people. You have to rest, sleep more, give yourself more time to do stuff, and so on. You shouldn't hate your family for trying to manage your expectations. My son now has the same 50/50 chance of having my disease. I'm going to let him mostly do what he wants, but not everything. I'm not going to let him play (american) football or hockey for example. No reason to risk head injuries. Life is different when you are a person with a significant risk of serious neurological problems. You can't have the same expectations as a regular healthy person.
Not having a part-time work history is fairly common with Millenials. I think when people see your intelligence, drive, and commitment, you won't have a problem. Yeah, there will be lots of interesting stuff. Start looking around now and see what kind of career plan you can make for yourself.