I'd suggest leaving them... they will also grow flowers when the rest of the plant does, and make an even better display. Certainly does no harm to leave them. But I'd rather have more flowers on a bigger plant than just a few on lots of little plants. My prejudice.