The hard part of being intolerant or allergic to something is not really not being able to be in contact with something or to eat it. No. The hard part is knowing it and feeling bad because of it. Feeling 'defekt', wrong, even stupid sometimes.
With some people is just the same. With people that are wrong for you.