The other effect that can be seen in some small 'speakers played at typical levels, usually not there in their bigger brothers, is moderate level compression. There's an example
here (chart4 up to 500Hz) for a 5.25-inch woofer. All being equal, a smaller drive unit has to operate with larger excursion and so reaches its linearity limits earlier.
Other may hear it differently, but to my ears this effect makes a small 'speaker seem louder than a bigger 'speaker played at the same volume. I hypothesise that this may encourage an impression that larger 'speakers don't sound good at lower volume levels.
What someone likes is what someone likes and there's no gainsaying that. But to me, reproducing dynamic range well is essential for a 'speaker to be good. Bigger drivers typically do this better than smaller. I wish we could see more evidence of this effect in reviews that provide technical data.