An AFP correspondent who managed to enter Bani Walid for a short time said thick smoke billowed into the sky, while the identity of those present was unclear and there was limited evidence of the new Libyan authorities on the roads outside the town.
While local officials said the town was attacked by Kadhafi's men, Interior Minister Fawzi Abdelali dismissed these claims, saying the firefight was caused by "internal problems" in the town.
He told Libyan television that the fighting was among the people of Bani Walid, and linked to "the issue of compensation for those affected by last year's war."
"The information we have from inside the city does not say that there are green flags (hoisted on town buildings) and there is nothing in relation to the former regime."
But Abdelali confirmed that five people were killed in the fighting as claimed by local officials.
"The loyalists of Kadhafi took control of the entire city of Bani Walid," said M'barek al-Fotmani, a former member of the ruling National Transitional Council (NTC) in the desert oasis, 170 kilometres (110 miles) south of Tripoli as he led the claims of the so-called attack.
Fotmani said the daylight attack started with a assault on a base of former rebels which killed "five thuwar (anti-Kadhafi revolutionaries) including a commander." Around 30 former rebels were also wounded, he said from inside the base which he later fled.
Mahmud Warfelli, spokesman of Bani Walid local council, too said that the attack was launched by "a group of remnants of the old regime," and called for outside help against a feared "massacre."
"There are around 100 and 150 men armed with heavy weapons who are attacking. We have asked for the army to intervene, but the defence ministry and NTC have let us down," he said.
"(The gunmen) took control and hoisted the green flag on some districts, some important districts in the centre of the city," Warfelli added.
A senior NTC member, Fathi Baja, said reinforcements had been sent to protect the town, adding the "fighting is between some Kadhafi supporters and thuwar."
Fotmani said the assailants had surrounded the base, which belonged to the May 28 Brigade, a unit attached to the defence ministry.
But Salem al-Ouaer, a military commander from the town told AFP the Brigade of May 28 itself had caused Monday's clashes.
"Recently the brigade arrested two persons from the tribe of Tlatem and after negotiations it was decided it would release them today. But when members of the tribe came to take them, the brigade refused and clashes ensued," he said.
"People may have spoken about being surrounded by pro-Kadhafi elements to get reinforcements...," he said, while a source close to the tribe said that some pro-Kadhafi elements in the town may have "exploited the situation to their advantage."
Another AFP photograher who has also visited the town said that the situation was back to normal later on Monday, but the base remained surrounded and added that he did not see green flags.
Monday's firefight follows an outburst of opposition to the ruling National Transitional Council in the eastern city of Benghazi last week that prompted its chairman, Mustafa Abdel Jalil, to warn of possible "civil war" in post-conflict Libya.
Speaking on Libya al-Hurra television on Sunday, Abdel Jalil warned the new Libya would fall into a "civil war" unless protests against the NTC ended.
Crowds of protesters in Benghazi -- the city which first rebelled against Kadhafi last year -- had earlier thrown home-made grenades at and stormed the NTC office before setting it ablaze, witnesses said.
The demonstrators denounced the interim government for its lack of transparency and accused the NTC of marginalising some wounded veterans of the uprising in favour of people previously loyal to the slain dictator.
In recent months Libya has also seen clashes between rival militias, comprised of the former rebels.
Bani Walid was one of the last pro-Kadhafi bastions to fall in the bloody uprising against Kadhafi.
Its capture was followed days later by the fall of his hometown Sirte in a battle which also led to his killing and marked the "liberation" of Libya.