Despite the freezing conditions at Champion Hill, 725 supporters turned up and saw Dumebi Dumaka give Hamlet a 23rd-minute lead.
Nathan Green doubled the advantage three minutes later before Jake Reid halved the deficit before the break.
Reid grabbed his second to equalise before Vas Karagiannis put the home side back in front just after the hour.
And it stayed that way until stoppage time when Green gave away a foul in the box and Roman Michael-Percil kept his cool to fire home from the spot.
Meanwhile, Green’s goal last Saturday was enough to see Hamlet past Winchester City and into the last 64 of the Buildbase FA Trophy.
They will face either Royston Town or Hampton & Richmond Borough in the next round.