Right, booting a different OS to a phone is a lot more involved. I've never done it myself. It usually involves booting into a "maintenance mode" (I can't remember the actual name...something like that) and then flashing the OS image to the ROM. Some bootloaders are locked and require extra hoops to jump through. Most importantly, every phone is different, so specific images and specific instructions are required for each device. This is a big reason why I suggested PinePhone in the first place, because the whole point is letting the community do what they want with it. For Fairphone, you would need the company's cooperation to develop an OS image for the device, so options are limited. Some companies, like Sony, are actually good about it (Sony has an "Open Devices" program and they provide AOSP images and bootloader unlocking, which makes it easier for alternative OSes like Linneage to develop images).
All told, any avenue will require careful research: the right device, the right software, the right level of required work for your interest & time.
I think the most suitable options are:
1. Fairphone with /e/ (a de-Googled, FOSS Android fork) and a FOSS music player for Android. Job done, but user-experience-wise it's still very much a phone playing music rather than a DAP.
2. PinePhone with one of the supported OSes, modem disabled via hardware kill switch, & a Linux music player. Your best bet may well be, say, Arch Linux ARM (for tweak-ability),
Phosh UI (based on Gnome but built for phones), and Lollypop as a music player, which has an adaptive interface for working on small phone-sized screens. Set Lollypop to launch at startup and, voila, DAP. You might also throw in Pulseeffects if you want to have easy DSP capabilities (parametric EQ etc).
3. Old phone with PostmarketOS, Phosh UI and Lollypop. Possibly more work to get installed and up-and-running.
I personally think #2 is the way to go given the general hack-ability of the device. It's also cheaper than the Fairphone. It might take a month or two to receive though.