My own take on this, (in which I'll generalise and gloss over stuff in a big way) is nothing like as clear cut.
IME they are responsible for different aspects of overall fidelity... but with very considerable overlap, as you would expect!
Speakers (and cart for vinyl lovers) are largely responsible for the tonal balance, colouration/neutrality, bass extension etc and of course the way a speaker interfaces with a room has a huge effect here. In terms of potentially gross defects in sound quality speakers must be the most important then, ie if there is really boomy bass or a shouty mid from the speakers then this is going to dominate everything else...
Assuming the use of pretty good and revealing speakers of decent neutrality and which are "happy" in the room they are in though I hear huge differences between amplifiers in terms of front to back depth, "spaciousness" and "air", separation between instruments, "texture", whether the sound is "organic" or "sterile/mechanical" which to me can make or break the overall sound quality. Power amps seem to be largely responsible for these particular differences but phono stages can sound very different from one another as well... all IMHO and YMMV of course
Typically a "poor" (or even "average"?) amp will present a fairly "2D" soundstage with "pop up, shooting gallery" lateral imaging and a lack of space, air and separation. Loud sounds seem to make quieter less obvious stuff "disappear into the mix" and you are no longer aware of them.... It can sound rather "lifeless" and "grey" and attempting to get past this by increasing the volume just results in a "wall of sound" which can be quite oppressive.
A really good amp though will create a soundstage where you feel you could walk into it and say go 10' back and 8' to the left and touch a specific instrument. That quiet acoustic guitar picking out a few simple notes very much in the background becomes really obvious in spite of all the much more dominant sounds. It doesn't stop playing when the bass, drums and synths all kick in after the quiet intro, as seemed to be the case with the less good amp.
There is a sense of ease and effortless to the sound... it seems to "fall out of the speakers" rather than being "forced out against it's will" with the less good amp.
And all this without any real difference between the two amps in terms of bass extension, HF extension, colouration etc which are more the speakers department.
Hence I would say that
once the speakers are good enough, and well integrated with the room (which is a huge caveat, granted) then the amp becomes just as important.
