Developers of high traffic Web sites have to be ready to engage in 'Internet DIY', according to a panel of chief technology officers on the final day of the Online Developers conference in San Francisco last week. Commercial products such as Web servers and video streaming software too often do not scale up to handle volume traffic.
As part of a discussion group entitled 'Building it right, managing it effectively', representatives of three organisations that have built transaction-intensive Web sites offered some advice and some warnings to those in the audience likely to follow in their footsteps.
First off was Bharath Kadaba, president of News Internet Service, part of Rupert Murdoch's media empire, which operates a series of content rich, high traffic sites. "Our traffic on the sites is growing at a rate of around 20 per cent a month, which is becoming a real challenge to manage," he commented. Among examples of NIS' Internet work is the Fox Sports Web site, the online US 'TV Guide' and the BT-News International joint venture, Line One.
Kadaba identified three main elements that all complex Web sites will have in common - content, community and commerce. Content, he said, was seen by NIS as a major differentiating factor. Differentiators are depth - a movie database with listings going back to 1977, for example; being current - important for news service sites; and the ability to be personalised - the online 'TV Guide' can be customised to push forward information about an individual's favorite programmes during the week.
A sense of community will be created through the inclusion of chat services, bulletin boards, buddy lists and email. These can be used to increase traffic - Kadaba cited the example of 'TV Guide', whose traffic increases when celebrities take part in online question and answer sessions on bulletin boards.
Industrial strength sites should be constructed with a number of components built into the planning. For example, it is essential to have two Internet service providers to allow for redundancy and increased scalability over time. Kadaba cited the recent Webcast funeral of Diana, princess of Wales, as being responsible for a 900 per cent increase in video streaming traffic on NIS news sites.
A sophisticated production environment will be needed. For example, on the Fox Sports site a number of sports results tickers have to be integrated into the main site as well as legacy data held in a database. As a result, advised Kadaba, it is impossible to overemphasise the importance of systems management or the inadequacy of most of the solutions available on the market today. "You need to build a systems management solution yourself," he warned.
This was true of most aspects of Web site development, he added, with commercial servers and tools typically unable to scale. "There are some sophisticated tools that work well in low traffic sites, but no technology is available for high end, high volume sites," he observed. "Bulletin boards are designed for light traffic and won't support high traffic."
He was supported in his claims by Tom Eastwood, chief technology officer of sporting results Web provider Sportsline USA, who recalled that his company had been burned by the claims of technology suppliers and ended up doing fixes itself. "There are new products out there that come out and turn out not to be so strong," he said. "But you've got to go with them and try. The Internet is growing in dog years. You could miss the time to market if you delay. In addition, some of the companies you deal with will be start-ups, which carries risks in itself."
Alan Fisher, chief technology officer of online auctioneers Onsale, agreed with the need for a sophisticated production environment. Onsale does not need to accommodate news feeds, but does have multiple sales and supply chain management channels to integrate into its main site.
The Onsale site's business processes are fully automated. This means that the company only needs to employ around 100 people, making its sales per employee ratio very high. As a result, the supplier believes that it can offer lower prices than conventional retailers through the use of two-way electronic data interchange.
All these suppliers are aiming to achieve real competitive advantage through their heavy duty Web sites - but these aims have brought with them the burden of being pioneers and having to do a lot of the technology work themselves.
Loon's balloons will bring the internet to remote areas of the country
New clues into the biosphere on Earth in the lead up to the emergence of animal life
Planetary collision might shed light on the chaotic processes behind a star's early development
Success boosted by streamer Ninja and celebrity gamers