The optical cable is fine, it's the toslink transmitters - barely good for 5Mhz* . Given that SPDIF is 2.8Mhz that means a very rounded shape to the manchester-encoded data; and that is rubbish for accurate clock reclaimation in a non-reclocking dac.
GTM is right, 'Digital' audio is prey to many analogue failings.
*it's a cheapo LED and a photo diode.
Some e.g DPA for the PDM2 and later DACs went to very fast AT&T optical transmitters for the task instead at greater expense, to gain c. 10x the bandwidth.