The connectors do not define the impedance of the interface - that is largely driven by the cable/conductors used. The single-ended or unbalanced coaxial electrical interface spec for S/PDIF calls for a 75ohm impedance across the interconnect (from one connector via the cable to the other connector) and also allows for a choice of connectors - either RCA or BNC.
BNC connectors are more popular in pro-audio than RCA as they are a locking design and when inserted require a twist to engage the locking pins (RCA connectors can be pulled out if the cable is kicked). As far as sonic differences are concerned between connectors, if all else is equal (ie: the cable/conductor make-up) then there are unlikely to be any significant audible differences.
The areas where cable differences can be introduced are:
1) The quality of connector used (material, build-quality and conductive plating)
2) The quality of conductor used (material, impedance matching to standard 75-ohm spec)
3) The jointing between conductor and connector (soldering, crimping, etc)
4) Cable geometry (how conductors are assembled to minimise cross-talk)
So, for a given "model" from a reputable cable manufacturer, you should not encounter any significant audible differences between RCA and BNC terminations.
Dave