Microsoft's attempts to provide compatibility for older applications in its forthcoming Windows Vista operating system is opening the door for attackers, security vendor Symantec alleged in a study about the software's user account protection scheme.

Matthew Conover, a principal security researcher with Symantec, wrote in a whitepaper that he "expects several […] privilege escalation vulnerabilities to be discovered."

"Windows Vista's developers had to choose the best way to improve the overall security model while still retaining the most backward compatibility. While most of their decisions seem reasonable, two particular decisions lead to several seemingly intractable implementation flaws."

The forthcoming operating system follows the 'rule of least privilege', which commands that users should only gain access to items that they require to perform tasks. It implies that access to additional features will only increase the risk of exploits.

A likely way for malware to circumvent the new security scheme is by obtaining additional privileges, better known as privilege escalation.

Attackers targeting the Internet Explorer browser, for instance, will be restricted by a low rights clearance. Even if they succeed in installing spyware, their malware won't be able to access system elements such as the registry or network resources. This effectively neutralises the malware threat.

However, in his white paper, Symantec's Conover described several ways that allow applications to obtain additional privileges in early versions of the Windows Vista Beta. Those security vulnerabilities have since been patched, but he added that security researchers and malware creators are likely to find new security vulnerabilities.

He also cautioned for potential security bugs in a key element of Vista's security that is designed to prompt the user for consent if an application requires additional privileges. If flawed, it could allow attackers to circumvent all the operating system's security features.

"It's just a matter of an attacker finding one that can be abused," Conover concluded.

The least privilege rule in Windows Vista directs that user will be provided with a limited rights user account.

Each Windows Vista system has at least one Protected Administrator account. All processes that are launched by this user run with minimal privileges and the user is prompted when applications require additional rights.

The software also provides for unrestricted administrator access as well as standard user accounts.

Previous versions of Windows offered administrator and standard user accounts, but users were forced to run the administrator mode to get access to standard features such as the ability to change the system clock.

Symantec in its study asserted that most Windows Vista users will be running in an administrator account because they are easier to set up than standard accounts.

The operating system also introduces a process that's referred to as " mandatory integrity control". It provides processes with an integrity ranking and prevents it from interacting with a process with a higher integrity ranking. A low ranking process such as Internet Explorer, for example, will fail to access system memory or change registry keys.

Similarly, a low ranking process is unable to communicate with a high ranking process. Malware in the past used this method to execute arbitrary code.

Symantec's white paper is available as a PDF download from the company's website.