If it is acceptable for Americans to say “no” to Muslims on the grounds that we don’t want any more Muslims, then we’re back to discussing the limits, if there are any, to the freedom of association. Put another way, if we don’t need a reason to say “no” to Mohamed, then we don’t need to ask for permission in order to say “no” to diversity.