Network managers have been warned that certified firewalls cannot be completely trusted.
Security consultant NTA Monitor has questioned the quality of testing carried out by the International Computer Security Association (ICSA) and Information Technology Security Evaluation Criteria (ITSEC) after the bodies missed two flaws which NTA believes should have been identified.
Roy Hills, managing director at NTA, explained that more openness is needed in the testing procedures.
"The first thing that would help is some sort of full disclosure," he said. "One of the ways to improve a process is to be able to learn from mistakes. If they don't do this, how many more missed flaws will it take before people ask what certification means?"
NTA has found two significant flaws in ICSA-certified firewalls: the predictable TCP sequence numbers in Borderware Firewall in September 1998; and the FTP Bounce issue in Raptor Firewall in April 2002.
It maintained that both flaws are very well known to the security community and proper testing should have detected them.
Security organisation the Computer Emergency Response Team issued an advisory about predictable TCP sequence numbers in January 1995, three years before the Borderware flaw was discovered.
It also warned of the FTP bounce attack back in December 1997, four years before the problem cropped up on the Raptor firewall.
A vulnerability in Checkpoint's Firewall-1, an ITSEC-certified system, was also discovered recently. It allowed external internet users to make connections to an internal private network.
Steve Barnett, UK managing director at Checkpoint, dismissed NTA's claims that certification processes are flawed and insisted that certification is still essential to the industry.
"I have been personally involved in the ITSEC and it is extremely rigorous," he said. "It is unlikely that they would miss anything significant."
Security experts pointed out that companies which value security usually use firewalls from different vendors, one in front of the other, for added protection.
Using a pair is considered safer as they are unlikely to have the same vulnerability at the same time.
"People who understand IT security really well will put in architectures that are appropriate to cover unknown risks," explained Richard Barber, security consultant at Integralis.
"A certification company that has missed a particular flaw for whatever reason is in effect an unknown risk that ought to be taken into account."
He added that certification still works, as it stopped disreputable companies from claiming that insecure products are safe.
"As a filter for cowboys, it is extremely useful when you are talking about something as fundamental to network security as a firewall," said Barber.
Eleven 'normal' outer moons, and one described as 'oddball' found circling Jupiter
Scientific discovery has found a quadrillion tonnes of diamonds in the earth's mantle
Mobile payment app makes users' details public by default
2,400 signatures gathered against the development and production of lethal robots