Unless the amplifier is grossly underpowered for your speakers, or unless you are playing your music at disco levels it is highly unlikely in a domestic situation you would ever hear the amplifier clip. It is much more likely that if you are hearing distortion, it will be coming from the speaker itself.
Well known "deleterious effects" to who? Certainly not to those who know how to design a good passive loudspeaker. Take a design like a DeVore, or an Audionote and you will find that these mostly use first order crossovers, which are very simple, i.e. 2 components, namely a capacitor and inductor in the circuit, thats it! And seeing as they are being driven by a power amp with plenty of voltage and current any component losses will be negligible assuming good quality parts are used.
There are speaker manufacturers that use very complicated crossovers with poor components and these will have a bearing on the sound, but less so than say all the parts and circuitry needed for an electronic crossover, which is why most of the top speaker companies today make passive designs. There are a few that use an active element for bass below 100Hz but the rest of the audio range is passive.
I remember back in the early 1980s I used to rebuild speaker crossovers and the results were amazing. Certainly component compromising was responsible but if the parts were changed for better designs the speaker could be significantly improved.
On several of my favourite albums, and mainly on transient peaks, such as a rim shot, or crashing cymbals on a drumkit, I could hear mild distortion through my Naim SBL's driven by my then NAC72/Hi-Cap/NAP180 - at around 9 o'clock on the volume - enough to be realistically 'loud' but not giving it 'ginger' either.
As many of the CD's were AAD I assumed it was tape saturation on the master tape. However, when I swapped out the Naim for my Sony TA-DA9000ES, and played the same CD's at similar Db levels (using a SPL meter) low and behold the 'fizz' distortion that I assumed was tape saturation was gone. Only real explanation that I could think of was amplifier momentarily clipping on peaks - certainly driving to considerably louder levels with the Sony elicited no distress from the SBL's - but then it was and is a much more powerful amplifier than Naims finest bar the later Statement amp. Interestingly, the extra 'fizz' of distortion on the peaks added a sense of superficial excitement that was otherwise missed on the much smoother and more detailed Sony! Perhaps part of the appeal of the Naim sound?
The closest I got to auditioning speakers with a simple 1st order crossover and quality components was a pair of Epos 14's that I had home on demo whilst assessing speakers to pair with my then new Naim kit, before settling on SBL's. Certainly the simple crossover aids dynamics and detail if the drivers are designed around it.
However many passive speaker designs have complicated crossovers with high quality components due phase alignment of drivers etc - the B&W 800 series would come to mind.
Companies such as ATC, Dynaudio, PMC, B&O etc have all made speakers in both passive and active versions, and certainly ACT categorically state the engineering superiority of active over passive. Indeed, companies such as Naim and Linn still put active operation (with multiple external amps + active crossover) at the top of their performance tree. No doubt with good reason.
I would argue that most companies make passive speakers specifically for the audiophile market, where punters like to mix and match and choose amps and cables etc.
As such, the fully self contained active speaker, as used by most recording studios and professional sound engineers and top domestic speaker companies such as ATC, Dynaudio, B&O, Meridian, MBL Radialstrahler, Steinway Lyngdorf etc, would be anathema to an audiophile and likely a marketing disaster.
And of course they are not popular with most dealers because of the self contained nature, and commensurate lack of opportunity to on sell racks, cables, amplifier 'upgrades' and other audio accessories.
Cheers