Black Hat: FireEye researchers chronicle the rise and fall of Rustock

Two of the researchers who helped to bring down the notorious Rustock botnet have provided a definitive history of the malware at the Black Hat conference.

FireEye researchers Julia Wolf and Alex Lanstein described how Rustock grew more sophisticated and efficient over the course of several years, thwarting efforts by security professionals to halt its prolific spamming operation.

"You can really see Rustock evolve through the years as the security community and security tools evolved," said Lanstein.

Law enforcement groups in the US took Rustock down earlier this year when a series of suspected command-and-control servers were taken offline and seized by investigators.

The move crippled the sophisticated malware network which had proved so hard for researchers to track down.

Many of Rustock’s tricks involved duping researchers and administrators into not only overlooking Rustock traffic, but in some cases consciously allowing it.

The researchers described infections which would encrypt themselves as .rar files with titles such as 'backup'.

"If you were an analyst watching for interesting things you would see this .rar file called 'backup'," said Lanstein. "It's not just that you would think you can't tell what it is, you will go the opposite direction and think it is legit."

Ironically, it was the same spoofing tendencies Rustock relied on to stay under the radar that brought down its spam operations.

When Microsoft and FireEye researchers had tracked the botnet’s command-and-control operation to a series of servers, they were then tasked with convincing a judge to allow law enforcement to take down the machines in a single co-ordinated operation.

The key, explained the researchers, was a provision in the little-known US Lanham Trademark Act which allows trademark holders to seize counterfeit goods.

Rustock spam had at various times posed as emails from Microsoft, Pfizer and other companies, which allowed the companies to take down the botnet and seize data on the servers.

The lesson, according to Lanstein, is that major botnet takedowns can happen with enough time and resources.

"When you are working with the courts and ISPs and companies that are willing to spend a ton of money, it can work if you go about it the right way," he said.