It's important that people have the *legal* right to say anything. People shouldn't be prosecuted for it.

That doesn't mean any of those people need to be permitted at a conference. Their inclusion necessarily means the exclusion of many other people. Their rights have not been affected; they simply have not been allowed to attend or speak at a conference.