I agonised over this a few years ago, and did some blind testing on my wife, who could identify differences between the original 24/96 recordings, a 16/44 downsample, and a 128k MP3 quite reliably (though not between Flac and WAV), and rock and classical about the same. When she did the same tests on me, I could reliably pick out the MP3, but 24/96 vs 16/44 was better than 50:50, but not much.
One thing we did discover is what we called the talk-over test. We found that if we played music over dinner at full resolution through our pretty decent kitchen system, we would tend to listen to the music, but if we played the same music as MP3 we carried on with our conversation. Tried it out on friends (without their knowledge) and it seemed to work on them too, so at one stage we kept MP3 playlists for dinner parties.
However, the differences are quite subtle, and much smaller than the huge differences in recording quality we notice especially though our main system. Had a whole evening recently where I was convinced there was something wrong with our system, then suddenly a track came up on Radio Paradise (Flac) that showed that the others were just poorly recorded, or encoded, or something.
There are so many other variables that drawing conclusions that have any validity is almost impossible, so we just don't worry and enjoy the music. I do buy the occasional 24/96 recording, but most of our listening is via Tidal limited to 16/44, and we don't lose any sleep over the bit rate.