Your maths is wrong as you're just looking at the sampling rate. Here you go, a lesson for you:
"The S/PDIF digital audio interconnect format is described by a sampling rate of the 64-bit audio data frames. Each 64 bit frame contains a single audio sample for both the left and right channels along with various status bits. Therefore for the sampling rates of 48kHz and 96 kHz discussed here, the information bit rates are 3.072 and 6.144 Mbit/sec respectively. However since the encoding used in the S/PDIF format is biphase mark code, a 1 bit would have 2 transitions in the bit interval. This means that the clock rate is twice the information bit rate. The coaxial S/PDIF standard specifies an electrical level of 0.5Vpp at a termination impedance of 75 ohm. What receiver bandwidth is required for proper detection of bit streams for 48kHz and 96 kHz? From basic binary data recovery analysis, it is known that the required bandwidth is HALF the transmission bit rate.
Therefore, the receiver must have a minimum bandwidth of about 3 MHz to properly detect a 48 kHz S/PDIF stream, or 6 MHz for the 96 kHz case."
http://www.jensign.com/SPDIFLink/
In any case, it's more than just impedance mismatches, slower rise times need to be considered as well as as return loss...