The market for tools to collect, process and analyse Web site data is growing as sites move from simply displaying electronic brochures to become more complex revenue generators.
But while the Web site tracking market is still not very large compared with the billions of dollars spent on-line, it has shown remarkable growth in the past five years since it was born.
Stefan Smith, Internet analyst at Dataquest, a division of Gartner Group, says: "We see this as a market of about $200 million next year. It has become a bi-modal market, with products at the low end and high end, but not many in the middle."
He defines the low end as first-generation offerings that use log file analysis and carry a lower price tag. High end or second generation products actively examine packets on the network, respond to changing conditions and help with "one-to-one marketing, such as serving up ads to recognised visitors."
But Harley Manning, an analyst at Forrester Research, describes it as "a fairly confused marketplace" because there are different approaches to such analysis.
"Essentially, the products are software, which sit on a server and count traffic. But there are services, which place a piece of software of theirs on the server, and it sends them the information from the user's log. Still others have proprietary solutions and provide a report," he explains.
He points out that there are fundamentally two problems with trying to analyse Web data, however.
First, the Internet was not designed to do a lot of the things it is now being required to do and second, it was not designed to be fast or to maintain constant connections.
"The computer suffers from Alzheimer's - it keeps forgetting it ever talked to you before. The Web tools market is still in a primitive state and it's incredibly difficult to build a business on this," he says.
As a result, tremendous amounts of data exist about Web sites and their users, but it is trapped in useless Web server log files.
Every time customers visit a site, the server enters a record of the HyperText Transport Protocol (HTTP) transaction in a log file. However, although Web server log files are rich in information, the data is abbreviated and cryptic, which makes it difficult and time consuming to data mine.
The information is also overwhelming in its volume, with a 1Mb log file typically containing 4-5,000 page requests.
In the past, the most common way to measure and analyse Web traffic was to study the log files that the Web servers create, usually in a standard format.
But Matt Cutler, cofounder and director of market development at net.Genesis Corporation, which was one of the first companies in the Web analysis market, now says: "There has been a transformation of market perception. Web traffic analysis has moved from being a techie tool to being part of a serious Web site strategy, which can offer tangible results."
While a few Web masters are able to peer into the raw data in these ASCII files and reach conclusions about users' behaviour, most people have to date relied on one of the many log analysis tools to provide them with summarised activity reports. These provide information about page views, top entry and exit pages, most requested files and such.
The problem with log analysis, however, is that the insights provided are limited to user activities that generate Web server entries in the log file. Other important information about visitors? behaviour never appears in the log file and is, therefore, not available for analysis through the log.
Web site analysis tools, on the other hand, typically import the log-file data into a built-in database, which, in turn, transforms it into readable, graphical reports that can be distributed throughout an organisation, and fine-tuned to meet particular needs.
Web administrators typically want to know that each click a user makes will lead to documents, and that images, multimedia files, scripts and applets have been loaded and displayed properly.
Web designers, however, need to know how visitors navigate the site, what their preferred paths are, and which points are the most popular for customers to jump to another site.
Accrue Software has been in the Web analysis business since it was set up in February, 1996, and shipped its first product, Accrue Insight 1.0, in December of that year.
According to a report by the Patricia Seybold Group, the company is "evolving its focus from Web traffic analysis to analysis of business processes as they occur in the Web environment."
Michael Goulde, a Seybold analyst, explains: "Accrue Insight is an analysis tool that gathers data from the network and from Web servers. It is designed to handle sites that receive a very high volume of visits, which are often distributed across multiple servers at many locations."
He points out that Accrue has a multi-stage architecture, beginning with the Data Collection stage that passes data to the Analyzer. This prepares it for Data Storage and finally passes it to the Reporting stage.
But there are many other Web traffic analysis products on the market, including Web Trends, MarketWave?s Hit Lit, net.Genesis' net.Analysis, Everyware's Beolero, Microsoft Site Analyzer and Andromedia?s Aria.
Each of the offerings, apart from Andromedia?s, either only analyse log file data or log files in conjunction with enterprise data, but Accrue, Aria and new entrant, SilkStream, also look at network traffic.
net.Genesis' net.Analysis 4.0, however, is aimed at enterprise users and is based on a new net.Stream architecture. This correlates enterprise data with information about customers' on-line behavior, enables realtime reporting and integrates the realtime data with historical reports. It also integrates Web analysis data with corporate legacy data.
In addition, net.Genesis has rolled out a new methodology, dubbed Design for Analysis, which is meant to help maximise on-line business.
Larry Bohn, the firm?s chief executive, explains: "The key thing about Design for Analysis is it is a combination of site-design principles and analysis principals based around commerce, so that customers can gain the most value in this technology area. The methodology is a continuous structured evolution to adopt site analysis in context of your on-line business."
Analysts such as The Aberdeen Group, Forrester Research and the Meta Group have also singled out Andromedia for its analysis solutions. The privately-held company was founded in 1995 and has just merged with LikeMinds, a supplier of patented collaborative filtering technologies.
Kent Godfrey, Andromedia?s chief executive, claims: "The merger of Andromedia and LikeMinds redefines Web site activity analysis and personalisation. It takes Web visitor behaviour data and turns it into marketing insights and automated programs that can drive revenue and increase profits."
Users are put into categories such as bargain shopper, and those categories are fed in real time to personalisation software, which delivers pages specific to that customer in real time.
The good news for Web site analysis tools is that they provide users with a powerful way to get a handle on Web sites, enabling them to quickly identify links and status, to index and analyse content, and track problems and site usage.
The bad news is that today's tools are hampered by limited scalability and reporting and link repair capabilities. Also while the expected market shakeout will leave some very sophisticated offerings available, analysts agree that the choice of vendor will be limited.
Claims to have "the most competitive logic density" in the industry
Dell's high-end mobile workstations upgraded with Intel Coffee Lake CPUs
Webstresser admins were also arrested in the UK, Croatia, Canada and Serbia
Security firm claims that 117,638 sites out of 135,035 analysed contain serious security flaws