There are various kinds of elitism at play here and if people reading what I write think I am saying you cannot enjoy music (or even understand) a piece of without musical training or knowledge then read again, and if you still think I am saying so read it a third time. I think I have been clear.
There is also the elitism of "I know what I like and I don't need no education" and while that is true it is much much harder to understand fully the language and structure of music, its architectural edifice is seen only from the outside and the inside, but the way the edifice is built is based on assumption and (in my case with jazz, reverse engineering, looking at the edifice and stripping it down)
This is why I compose, this is why I build thing, this is why I am useless at everything in life except this strange marriage of the two.
But I will not step down against anti-intellectualism, the "my opinion matters as much as yours" approach to this, music is formal, defined structure that has the added advantage of appealing to the emotions, if you write about the emotional aspects of music only, fine, but that places you in uncertain terrain, terrain that is assumptive and quite possibly wrong. If you know that a modal change creates an uplifting effect if the minor key subdues the piano while the sax takes over, if the expression is notated just so, then you gain insight into the mind of the composer, the inner workings of the music are revealed in unpretentious and obvious ways.
Uninformed "assumptive" music reviews are hand waving at best, poetical near nonsense at worst and not fit for publishing, but they are useful for artists to get eyeballs to hover around their work for even a dew seconds. The basic fundamental errors in music reviews that mention key changes when they are modal changes, when the tempo is not changed but the beaming across the barre is changed is not important to an emotionally driven reader, but it is a factual mistake and it is just plain wrong and in allowing that wrong to go uncorrected you do the artist and the music a great disservice.
Sorry, I know I said I was done but misrepresentation and not really reading carefully is in and of itself wilful neglect.
tl;dr. Its perfectly fine to and completely right to not need any education to enjoy music but you will better understand the basis of your enjoyment if you do learn a bit.