Dominic Telford’s 35th minute finish proved enough for the Shakers after they dominated the first half and they were value for their win.
Stags were improved after three substitutions at the break but could not find a way through the visitors, who also had former Stags midfielder Stephen Dawson sent off five minutes from time for his late challenge on Tyler Walker.
Mansfield tried to be brave and push numbers forward which left holes at the back for Bury to exploit and two minutes from time, Conrad Logan conceded a penalty from which he made a good save to deny Caolan Lavery.
While disappointed to be out the competition, the silver lining could come in that Stags have had 90 minutes to shake off the rust from a 10-day lay-off that clearly showed on them and the reason boss David Flitcroft played such a strong side ahead of Saturday’s big home derby with Notts County.
See who you can pick out in our fans gallery from Mansfield Town v Derby County
Supercomputer predicts the most likely result for Leyton Orient v Mansfield Town, Bradford City v Newport County, Stockport County v Colchester United and every other League Two game
Former Mansfield Town boss Mick Jones dies aged 75
One more new face in for Mansfield Town as early injuries mount?
Mansfield Town boss Nigel Clough proud of players and fans as Stags are pipped by Derby County in Carabao Cup classic
On a cold night, the dire first half was the worst they have played all season and the least chances they have created.
The team showed three changes from the side that drew at Lincoln City a a week last Saturday with Logan, Hayden White and Otis Khan in for Bobby Olejnik, Will Atkinson and CJ Hamilton, who all dropped to the bench.
Mansfield were almost caught out in the first minute as Bury carved them open with Logan saving from Telford before Sweeney cleared Dagnall’s finish off the line with the goal gaping.
Then a McFadzean cross from the left clipped the top of the netting and a low Cooney cross just evaded Telford in front of goal as the home side lived dangerously in the opening stages.
Mansfield were struggling to get out their half and, when they got as far as the halfway line on seven minutes, Miller brought down Rose to earn a booking.
Logan was lucky when he came for a Barjonas free kick and failed to get there first, the ball flicked over the empty goal on 18 minutes.
Stags were struggling to find any rhythm or sustained possession though forced a couple of corners.
Aimson headed another Barjonas free kick over the top on 23 minutes as Bury continued to dominate.
On 33 minutes MacFadzean drilled the ball in from the left and it flew across the face of goal for a goalkick with Logan looking like he had it covered.
A minute later Mansfield finally had a shot on goal as a loose clearance reached Bishop 20 yards out and his shot flew straight to keeper Hudson.
But Bury were deservedly ahead on 35 minutes. Benning and Dawson challenged for the ball and the defender managed to help it into the path of Telford, who slipped it beneath Logan from 10 yards.
Walker was too high from distance as Stags looked for a quick reply.
MacFadzean went down in the box under Pearce’s challenge on 40 minutes, but instead of a penalty he was shown a yellow card for simulation.
Walker stabbed another half-chance at goal after good control, but again straight at Hudson, as the Shakers went in deservedly in control.
After a dismal first half, home boss Flitcroft made big changes, withdrawing Pearce, Khan and Rose and sending on Hamilton, Davies and Elsnik, switching to a flat back four.
It almost paid off within three minutes as White’s square ball was smashed at goal first time by Bishop from 20 yards, but Hudson was there to make the save.
Bury were quickly finding the pace of Hamilton hard to deal with Aimson brought him down for a 53rd minute booking.
Hamilton was unlucky when his powerful shot was blocked team mate Walker on 54 minutes and the ball ran out to Mellis who drilled just over the top.
Telford should have doubled his tally on 64 minutes as Stags were caught pressing and Lavery’s pass slipped him behind Sweeney and through on goal.
He lifted a clever finish over Logan but could only look on in agony as it bounced wide.
Mellis was wide with Stags’ next shot as Bury continued to slow the game at every opportunity.
On 72 minutes Hamilton superbly controlled a high Mellis ball and got away a shot that was well saved.
As Stags pushed, Bury were looking dangerous on the counter and Logan’s outstretched foot denied Telford from Adams’ ball in on 76 minutes.
On 78 minutes Hudson was booked for timewasting.
Logan had to dive to his left to keep out a header from his own defender, Preston, as he tried to cut out a low cross.
Bishop was added to the book for hacking down Adams on 82 minutes and seconds later Telford was too high after swapping passes with Omotayo.
Dawson was very late with a flying tackle on Walker, the Bury man staying down for treatment himself before being shown the red card on 85 minutes.
With just two minutes to go it all looked lost for Mansfield as a Bury break saw Barjonas through on goal and Logan bring him down for a yellow card and a penalty. However, the keeper got down superbly full length to his left to keep out Lavery’s low spot kick.
However, six added minutes could not come to Mansfield’s rescue as they bowed out.
STAGS: Logan, White, Benning, Preston, Pearce (Hamilton HT), Bishop, Mellis, Khan (Elsnik HT), Sweeney, Walker, Rose (Davies HT). Subs not used: Olejnik, Atkinson, Butcher, Graham.
BURY: Hudson, Miller, Aimson, Thompson (Lavery 57), Dawson, Dagnall (Omotayo 76), Styles, Cooney, Telford, Barjonas, McFadzean (Adams HT). Subs not used: Maloney, Hulme, Nyaupembe, Seriki.
REFEREE: Robert Lewis of Shrewsbury.
ATTENDANCE: 1,029 (27 away). CHAD STAGS MAN OF THE MATCH: CJ Hamilton.