To me, "legitimacy" is a made-up concept that doesn't have any applicability in real life. He was legally elected, he can be legally removed. All else is posturing. Introducing a subjective concept like "legitimate" would inevitably get extended to "his campaign manager beat his wife therefore the election was illegitimate" somewhere down the road.

There are mechanisms to deal with whatever has come up thus far. Use them or don't, the rest is just posturing and wishing for a different outcome than the one that occurred.