After I had some not-easy-explainable symptoms and wanted to be "clean" I left out more and more kinds of food from my "average" diet. Currently I am (a kind of) vegan (no meat, no fish, no egg, sometimes dairy from non-cow milk) and eat as few as possible refined foods. No sugar (seldom "unrefined" sugar), no yeast. But much cereals (millet, buckwheat, rice, corn, sometimes normal wheat or spelt (Triticum aestivum spelta), much of oats) and much fruits and vegetables (often raw), and sweeteners like honey or sirups. And xylitol, stevia. No soy products. Not much fats or oils (including seeds). I did once a "80-10-10" diet for over a month but not experienced any special benefits that did go away when I stopped with it. There are many controversies about food healthyness.
I do not know about intolerance to any of the mentioned things (gluten, casein), I have no special problems after eating some goat cheese for example. After eating egg and I did not eat egg before that time for more months I had some kind of bad feeling or "sadness". After all, I am feeling somehow more "clear" since the time when I introduced veganism.