Oli Hawkins headed the hosts in front on 44 minutes from Kellan Gordon’s corner.
Rovers levelled after Cian Harries smashed a brilliant first-time volley past Nathan Bishop on 61 minutes.
But the game turned when Paul Coutts was sent off for an off the ball incident involving Danny Johnson.
Stags won it six minutes into stoppage time after Johnson scored from the spot after Mark Hughes had tugged Hawkins.
Clough said: “It was a dramatic game to win it like that, I thought we deserved it on the balance of play.
“It was only the wondergoal which got them back into the game. When they went down to ten men we put the pressure on them.
“We weren’t absolutely at our best today, but I thought we did enough to win the game.
“We won't know how important the three points will be until nine months, but it is great for belief and for the fans.
“Oli Hawkins scored and won a penalty, he delivered today and is building a good partnership with Danny Johnson.
“They need to play games together and spend time together on the pitch.
“That is why we were not tempted to change it late on, we felt those two were as likely to score as anyone.
“We have a settled back four and it helps that the goalkeeper has come in and settled straight away.”
Clough also getting the refereeing decisions going there way was a just reward for Mansfield’s perseverance.
“I didn't see the red card, I half glanced and saw the hand coming down,” he said. The old saying is not to raise your hands.
“The referee was right there. He was three yards away for the penalty, I have not seen it back but the tugging was happening all game.
“You have to keep putting the ball in the box and if they keep tugging you get your reward.
“We are not going to be able to play brilliant football every week so it is about character and perseverance.
“We showed a will to win today. The squad and the fans came together and we need that through the season if we are to be successful.
“It was good to get the first win of the season, especially for those lads who were here last season.
“It gives us a good platform. The other nice thing was that the vast majority of the crowd was still here, I didn’t see many leaving early.
“The noise they made when the goal went in sounded much more than 6,000.”