John Bowler swore and used racist language towards the couple and their children when he visited his granddaughter on Marlborough Road, between June 2016 and September 2017.
Prosecutor Robert Carr said Bowler had been a regular visitor and used to say hello to the family, but "suddenly turned nasty" around the time of the Brexit vote.
He said the abuse led to the parents being prescribed sedatives.
"The family decided to move away for the sake of harmony," he said. "They were not happy but they did move away.
Most Popular
-
1
Fire crews deployed to ‘large’ blaze in Mansfield – with people evacuated as flames spread to nearby properties
-
2
Letter: Unfair that disabled can no longer access water park
-
3
Creswell properties evacuated as fire crews tackle field blaze
-
4
Large fire in Kirkby may have been a 'deliberate or a careless act'
-
5
New council boss reveals ambitious plans for Mansfield
"When police interviewed him, he said he was racist at the time, but not now. He accepted he could have dealt with any issues in a different way."
Amy Godson, mitigating, said Bowler's granddaughter had "significant issues" with her neighbours, who put padlocks on communal gates, and the landlord confirmed this.
She said another neighbour, who had been the victim of racism, had written a letter to the court explaining that Bowler had defended her.
She said the 68-year-old, of previous good character, was in very poor health, on 16 types of medication, and had to cope with 19 bereavements over the last few years.
Bowler, of Marlborough Road, admitted causing racially aggravated harassment, when he appeared at Mansfield Magistrates Court, on Friday.
He was fined £765, and ordered to pay £85 costs and a £75 government surcharge.