punishment for unpopular speech and the mob doing the same by imposing economic sanctions.

I find it regrettable that so many no longer believe that the best cure for bad speech is good speech, but that is not my point here.  My point is that even many if not most free speech advocates seem to believe it is not censorship unless the government does it.  But if the government ruins your life because you said something seems ugly or we free citizens do the same, how is one different from the other?  But are acts to stop speech deemed unacceptable.

But don't we have the right to do business with whomever we choose for whatever reasons we choose, including objecting to their political positions?

Of course.  And if Richard Spencer opened a rib joint in Houston I would not go regardless of how good the ribs were.  That is my choice as an individual.

It seems to me that the distinction arises when we attempt to coordinate actions in order to impose society sanctions. At that point we are acting like a government in exacting punishment, only without the accountability.

That guy who was fired from his job at a hot dog stand after being outed for marching in Charlottesville, would he be blackballed from every job until he becomes homeless?  Because he believes whitey rights are under attack?  Because he didn't keep that opinion to himself?