I say go by the evidence, and the evidence all points to the fact that talking on a phone is more distracting than talking to someone in person. This has been known for a long time even outside their use in cars. The nature of the voice being listened to via a speaker requires our brains to concentrate harder on the voice than a live voice because of the loss of a lot of auditory clues.
Humans are capable of multitasking. So even if driving in a city it's perfectly possible for us to do so safely whilst conversing with passengers or listening to music or the radio etc. As long as we remain consciously focused on driving as our primary objective. The problem is many people see the car as a "second nature" activity and give their primary focus to the conversation or passengers rather than the driving, which is when it becomes a safety issue.
There is technology that tracks eye movement and can tell where someone is looking. Maybe the way forward is to implement it as standard in all cars and pass out automatic fines/points etc if your eyes aren't focused out the windscreen or side windows for a given percentage of time whilst on the move.
To be honest, I would implement technology to just make phones not work in cars full stop. I don't believe people need to be contactable for the time they are in a car. Millions of people manage without being contactable whilst on the London Underground every day, what's the difference? We all managed to survive with periods of being incommunicado in the past we can survive now in the same way. The 24/7 connected way of life leads to higher stress levels and all the health issues that creates.