The Cobblers gave up ground in the promotion race after Oli Hawkin’s 21st minute header proved decisive.
But it was a goal which Brady felt should not have been allowed to stand.
"I think the referee got some of the big calls wrong,” he said. “The corner from which they score, Fraser (Horsfall) gets rugby tackled by (Lucas) Akins and it should be a foul for us.
“I'm still scratching his head as to why he is allowed to bring Fraser to the ground like that.
Their cause wasn’t helped when Paul Lewis was shown two quick yellow cards and dismissed for a push on George Lapslie, another call that Brady felt was harsh.
"Pablo (Lewis) makes a tackle and the ref gives him a booking for that but as he goes to get up, (George) Lapslie pulls Pablo down and it's just the usual coming together,” he said.
"Pablo just pushes him away but the lad milks it, throws himself to the floor and the referee buys it and gives a red card. It's soft."
But the Australian boss says his side are still keeping the faith and have full belief that they can seal promotion.
“There are going to be swings and roundabouts and ups and downs, results are down at the moment but we will pick ourselves up,” he added.
“I thought, with ten men, we really pushed them and we were the only team that looked like we would get the next goal.
“The boys put their bodies on the line and the effort was fantastic. Overall it was a good game, but it was just a shame that some of the key moments were not officiated better in my opinion.
“Overall they came out on top, with a controversial decision.
“There are still lots of points to play for and we are in the mix. At the start of the season a lot of people would not have expected us to be where we are now.
“We have got the utmost belief in the group and we know there is plenty of football to play.”