The Stags had avoided a first round exit in the coveted cup competition for three years in a row.
Rhys Oates’ early effort proved decisive as the League One Black Cats suffered a third defeat on the spin in all competitions.
The Stags made the perfect start to proceedings as they took the lead inside five minutes.
Oates did well to close down Sunderland stopper Lee Burge before firing home off the inside of the post for his second goal in as many games.
The Black Cats came roaring back into the game and could have levelled after 15 minutes.
Leon Dajaku headed over the bar and should have done better as the hosts threatened.
The Stags squandered a glorious opportunity of their own as Harry Charsley missed the target after being picked out by George Lapslie’s pin-point pass.
And another opportunity fell to Charsley mere minutes later as he fired over Burge’s bar.
Stags’ stopper Nathan Bishop produced a super save to ensure the visitors led at the break as he kept out Nathan Broadhead’s point-blank header.
And Nigel Clough’s charges were frustrated by the furniture seconds later when Oli Hawkins was denied by the bar in a frantic end to the opening period.
Raging Sunderland boss Lee Johnson made a quadruple substitution in a desperate bid to turn things around at half-time.
Daniel Neill had the first sight of goal after the break as he blasted wide of the mark after being picked out by substitute Lynden Gooch.
Defender Elliott Hewitt produced a great block as the visitors held on for dear life at the death.
Clough’s side breathed a huge sigh of relief as Neil’s effort crashed against the bar in stoppage time.
Sunderland: Burge, Alves (Winchester, 45), Wright, Flanagan (Doyle, 45), Cirkin, Evans, Neil, Dajaku (Pritchard, 45), Embleton, O'Brien (Gooch, 45), Broadhead.
Unused subs: Hoffmann, Harris, Taylor, Younger.
Mansfield: Bishop, Hewitt, O'Toole, Hawkins, McLaughlin, Charsley (Forrester 90+4), O.Clarke, Maris, Quinn, Lapslie (Sinclair, 84), Oates (Bowery, 63).
Unused subs: Ward, Law, Clarke, Burke, Shelvey, Johnson.
Referee: James Oldham
Attendance: Not announced