Oli Hawkins headed the hosts in front on 44 minutes from Kellan Gordon’s corner.
Rovers levelled after Cian Harries smashed a brilliant first-time volley past Nathan Bishop on 61 minutes.
But the game turned when Paul Coutts was sent off for an off the ball incident involving Danny Johnson.
Stags won it six minutes into stoppage time after Johnson scored from the spot after Mark Hughes had tugged Hawkins.
“We are really disappointed to concede a penalty that late,” said Barton.
“We feel massively aggrieved by it. The ref has made the wrong decision and it sums up his afternoon.
“He had a really poor afternoon, I felt. I felt we were good value for a point.
“We are miles off where we want to be, but we will just keep getting better and better.
“Mansfield have invested heavily and they will be there or thereabouts.
“They have got a good manager and players who know how to work the referee.
“Johnson has punched Coutts in the chest and managed to get him sent off.
“They are smart enough to foul our player. Oli Hawkins has fouled Mark Hughes and he has given a penalty the other way.
“You can make your mind up when you watch that back. I don’t want to be bleating about referees, but when they decide games like this.
“He has made two big decisions and it has cost us the game.”