I really do agree with you in the normative sense but the reality is that anti discrimination laws exist and are enforced. So that being the case why shouldn't we apply these anti discrimination laws to our favor?
Because those anti-discrimination laws don't apply to permit holders and the solution lies in less government intrusion, not more.