Two goals in four first half minutes put Vale in charge and Stags' hopes faded further when two bookings in five minutes saw Oli Hawkins red-carded on 35 minutes.
“Today ranks alongside relegation with Burton on the last day of the season with four minutes to go as my worst managerial moments,” said Clough.
“Football is a horrible game sometimes and this is a horrible place sometimes.
“There may have been some nerves. I am not sure if we were ready for it from where we have come from.
“We were joint bottom of the league seven months ago so it's a big step to come to Wembley.
“We were hoping the experience we have in the side would have been able to cope with it.
“But I don't think we did ourselves justice today with what you've seen over the last seven months. That is the most frustrating part of it – we didn't give ourselves a chance.”
Jamie Murphy did fail to put away an early headed chance and Clough said: “It was fine margins. We missed a good chance early on to go 1-0 up. He should score there.
“Rhys Oates also had one that was kicked off the line. I think at least one of those is entitled to go in.
“But a deflected cross went straight to the lad and the second one came off the bar.
“It could have been different, but they punished us and going down to 10 men was the icing on the cake for the first half.”
Clough said he had not yet spoken to Hawkins about the sending-off.
“I only saw the second one,” he said. “People tell me the first was soft.
“I think the fourth official said the assistant had given it from 40 yards away. So it will be interesting to see if it was a yellow.
“But he shouldn't have then committed the second foul. It's happened one or two times this season and it's probably symptomatic of one or two of the things we did today in the first half.”
Clough continued: “I told them at half-time to stay solid for another 15-20 minutes as we couldn't concede another goal, and just try to slowly work our way back into the game. I thought we did okay.
“We restricted them, they just had to defend. We wanted to nick one and make it a close game in the last 10 minutes. But once they got their third late on it's over.
“But 2-0 down with 10 men at Wembley after half-an-hour, it's almost impossible.”
The final whistle saw Vale boss and ex-Stags star Darrell Clarke overcome by emotion as his side completed the job just months after the death of his 18-year-old daughter.
“I class Darrell as a friend and that is the only decent thing to come out of the day for us. I am delighted for him and many congratulations to him and Port Vale,” said Clough.
“It has still been a successful season for us, even though it wasn't the ultimate success we wanted.
“To get to this stage has to be classed as successful to a certain degree and we will try to build on that for next season.
“We start again nine weeks today so we have got to get some rest and holidays in. The only thing you can do is put it behind you and get ready for next season.”
Looking to the future, he said: “We don't have too many players out of contract. It's hard to recruit when you don't know what league you're going to be in.
“Hopefully a few will be staying and we will try to get five or six new players in.
“When you have only lost six league games since October I think it's fair to say we have a good group of players and they are as low and disappointed as anything that they didn't perform as they would have liked today.
“We will delay our pre-season return date after today. It will be a shortened pre-season as we try to give the players the maximum amount of timer we can.
“They have put an incredible amount of effort in over 11 months. We have just fallen short at the final hurdle. But we will go again next season.”