Search engines cleared of bias favouring big sites

Search engines are not biased towards popular websites, and may even be egalitarian in the way they direct traffic, computer scientists claimed today.

An Indiana University (IU) School of Informatics study, entitled 'Topical Interests and the Mitigation of Search Engine Bias', challenges the view of a web-dominating 'Googlearchy' in which search engines are accused of pushing all internet traffic to established, mainstream websites.

"Empirical data do not support the idea of a vicious cycle amplifying the rich-get-richer dynamic of the web," said Filippo Menczer, associate professor of informatics and computer science at Indiana University School of Informatics.

"Our study demonstrates that popular sites receive on average far less traffic than predicted by the Googlearchy theory and that the playing field is more even."

Menczer was joined in the study by IU post-doctoral fellow Santo Fortunato; Alessandro Flammini, assistant professor of informatics; and Alessandro Vespignani, professor of informatics.

The IU team aimed to collect empirical data from various search engines. In one scenario, users browsed the web using only random links. In another, users visited only pages returned by the search engines. The researchers also studied the way in which search engines have influenced the web's evolution.

"A simple ranking mechanism provides an elegant model to understand the genesis of a broad class of complex systems, including social and technological networks such as the internet and the world wide web," Fortunato said. "These networks possess a peculiar 'long-tail' TM structure in which a few nodes attract a great majority of connections."

The long tail structure of the web is commonly explained through rich-get-richer models that require knowledge of the prestige of each node in the network. However, those who create and link web pages may not know the prestige values of target pages.

In another study, Scale-Free Network Growth by Ranking, Menczer, Fortunato, and Flammini claim that for a search engine to give rise to a long tail network, it must simply sort nodes according to any prestige measure, even if the exact values are unknown. If new nodes are linked to old ones according to their ranking order, a long tail emerges.

"By sorting results, search engines give us a simple mechanism to interpret how the web grows and how traffic is distributed among websites," said Menczer.