This is true of my experience too. I miss crashing out with a bunch of mates listening to music. We used to take turns to pick tracks and take music to each other's houses. Had to listen to a lot of heavy metal! ;0)
My wife's just not interested. She doesn't 'get' most of the music I like and certainly has little musical sensitivity. She's learning though. I'd be listening to something, and it would be just getting to the best bit, and she'd barge in, turn it down and ask me something. I took the head off her! I said 'How would you like it if you were watching a good film, it was just getting to the climax and I switched the TV off and started talking to you? Or yanked the book you were reading out of your hand?'. It sounds daft but she just did not get how involved in music other people can be. It's just background to her. I tend to get piece now.
If your wife and kids watch TV it's a good idea to get a TV in another room. If the bedroom is a nice place to watch TV she'll maybe be quite happy to disappear off there to watch it leaving you the stereo. My wife rarely watches TV, I reckon the TV is on for maybe half-an-hour a day in our house, some days it's not on at all. I had it on for a few minutes earlier to see if there was anything worth watching on. Wasn't.