That's not correct. Speakers cover a relatively narrow bandwidth before they start to break up. This means that the cone starts to buckle and twist, which affects how they measure and how they sound.
Even a 6.5" driver is breaking up at around 1.4 kHz and this is obvious when you measure them. Designers have devoted huge energy to smoothing these problems out and most are okay to crossover in the 2.5 to 4 kHz region.
5" drivers break up at higher frequency, but lack dynamic range and bass extension.
However the bigger a driver the lower the break up. A 15" might at 250 Hz, a 12" 350 Hz and a 10" a bit higher.
In the old days when powerful amplifiers didn't exist, designers had to use larger drivers with lighter diaphragms to increase sensitivity, which meant less bass and more break up as well as a low crossover point. They still do this for PA and for Tannoys, but in terms of high fidelity sound reproduction, neither are as good as smaller modern drivers.
There have been big improvement in driver design in recent years, which means bigger amps are more important that they used to be. Old fashioned speakers boomed, so that the harshness of clipping was sort of balanced up. Boom and Tizz. Now they are controlled over very large excursions, so no boom, just harshness.
Ash