The Shaymen were quick off the mark with an inswinging corner finding captain Shaun Garnett but his side-footed shot was parried away by Forest Green keeper Steve Perrin.
Like Tuesday against Woking, Rovers looked a much improved side when taking the lead after 24 minutes. Martin Foster's inswinging free-kick went past everybody in the box and Steve Jones' cross back in was flicked on by Neil Grayson and Foster's deflected shot was tapped in by Scott Rogers.
Rovers' Gary Owers came close on the half hour when he latched onto a weak Halifax clearance but fired the ball wide of the right-hand post from the edge of the area.
The home side felt they should have had a penalty six minutes before half-time when Darren Hockenhull's wayward back-pass put Steve Brodie clean through and he looked as though he had been brought down in the box by Garnett only for the referee to wave play on.
However if the spectators thought that was controversial they hadn't seen anything yet. What happened in injury time was quite unbelievable, Halifax's Ryan Mullen got into the Rovers penalty area only to be scythed down by Scott Morgan and the referee pointed straight to the spot.
Rovers captain Martin Foster raced up to Morgan and let his feelings be known to his team-mate about the tackle. Whatever he said Morgan reacted with a head-butt and attempted punch at his captain and as the other Forest Green players tried to separate them the referee promptly gave Morgan his marching orders for violent conduct.
After the melee Steve Bushell stepped up to take the Halifax penalty only for Steve Perrin to guess correctly diving full stretch to his right to tip the ball round his post.
With the extra man Halifax upped the pressure and equalised nine minutes into the second half when Hockenhull's inswinging corner found Lee Elam unmarked at the near post to head in the simplest of goals.
Rovers had another penalty claim turned down just after the hour with Neil Grayson going down in the box. The linesman flagged for the penalty but the referee went against his assistant's advice deciding that Grayson had gone down too easily and gave him a yellow card for diving.
Halifax's winner came eight minutes from time, again from a set-piece when this time Hockenhull's free-kick was headed into the left hand corner by second half substitute Andy Farrell.