I would suggest otherwise, the people behind Leave.EU and Farage &Co knew their audience. The conflation of Freedom Of Movement with a refugee crisis was perfect timing for them. There is still a refugee crisis and we hear precious little about it now it has served it's purpose.
The antipathy to immigration was not "generally at lower levels than most of the EU" (see table). That's why it was central to the Brexit campaign. The numerous vox pops and QT audiences provided ample demonstration of hostile environment that had been cultivated - beginning with Cameron's absurd 'targets' which then led to May's "F off home or we're coming for you" trucks among other measures such as the fortunes squandered on vexatious automatic appealing of residence applications - seeking to make appeals fail for lack of ability to pay to fight them, rather than the justice or legitimacy of the claim.
Sustained media and particularly newspaper owner's xenphobic campaigning paid off bigly. It reached fever pitch during 2015 into 2016 - that is no coincidence.
Only when it was obvious that it was having the right effect and not putting off people who really should have known better, did Johnson join in with his own disgraceful contributions. It really became open season in the way that many Republicans now ape Trump.
Support for EU freedom of movement by country
https://blogs.lse.ac.uk/brexit/2019...hreat-how-europeans-view-freedom-of-movement/
Fact is, it's easy to whip up xenophobia as a scape goat for your own failings and the UK Government were all over it once they realised it didn't upset too many people - shame on them. Hardly suggestive of your contention that such attitudes are were somehow at a lower level in the UK than the EU countries.