Web site analysis - Fingers on the pulse

scrutinising a Web site's traffic is no small task. A busy server can attract quite a bit of traffic, inundating a Webmaster with cryptic log file data as the only indication of where visitors have come from and what they might be doing on the Web site.

Web site analysis software can transform all that raw data into meaningful information.

We've tested seven software packages that all process the log files collected by every Web server, converting the basic data into tables, charts and graphs. Because other types of Web site analysis applications exist, including programs that map out a site's topology, log file analyser is a more accurate term for the utilities that interpret the data servers collect.

Anyone running a personal, low-traffic Web page probably doesn't need anything more complex than a Web counter that tallies the number of hits to their site. But that isn't the case for corporate or commercial Web sites - bandwidth is costly, and the personnel cost to keep a dynamic, healthy Web site operating is even more expensive.

Get to know your visitors

In order to discern whether a Web page is pulling in its desired target audience you need to know who is visiting the site. Understanding which pages receive the most hits can tell you whether the intended messages are being viewed and can offer some clues for revising a site map to lure visitors to top-priority pages. Knowing when your site is most popular and where visitors are coming from can help you understand what made them turn to your URL in the first place.

Although the activity may not be visible to Web cruisers, browsers and other Internet clients constantly bombard a Web server with requests for HTML pages, graphics or other files. As the server provides this information it logs information about each transaction such as page accessed and the date/time. It also logs data about the visitor, ranging from the URL from which the visitor accessed your page to the kind of browser used. Any errors encountered during the visit are also logged. This information can be useful - unfortunately, there's an enormous amount of it and it's hard to interpret.

Logging data

Web servers simplify things by storing different kinds of information in separate files, often using an ASCII format common to different server packages. The transfer or access log tracks every hit on the server recording the date/time, IP address, the HTTP text of the request and the number of bytes transferred. Keep in mind, a request for a page containing six graphics counts as six hits and will produce six lines in the transfer/access log if the page is fed to the visitor's browser. This log file can quickly grow to huge proportions.

The error log records lost connections, time outs and other failures like the ever-popular "404 Not Found". While most errors are listed in the transfer/access log, anyone without log file analysis software will find it easier to peruse a separate listing of just the failures.

Web servers can also record a "referer" log, which details the URL from which your visitor jumped to your page. This log can be useful in determining what links are proving most fruitful. The agent log tracks information about the browser, FTP client or search engine robot that accessed your page. If you want to know whether AltaVista has indexed your site or whether visitors are using the latest browser, then this is the place to look.

While upwards of 30 different formats for storing log files exist, most servers adhere to NCSA/CERN-compatible common log format or combined log format styles, or newer configurations like extended log format. None of the log file analysers we looked at had difficulty with any of these formats.

Log file analysis software tallies the prodigious amounts of data in your log files by categories, spitting the information out in reports that summarise the results. Instead of poring through 50Mb ASCII files, you can view a top 10 list of the most popular pages on your site, or scan a graph formatted to show visitor demographics at a glance. The simplest of these programs may do nothing more than create ranked lists. Others, intended for large commercial sites, can interface with database engines like Microsoft's SQL Server.

While all log file analysers share certain features, for instance, the ability to process log files from a range of servers, they differ dramatically in how they are used and in their ability to analyse files. Some programs operate as a service like Price Waterhouse's NetCount, which analyses log files you transfer to the vendor.

Some organisations want all confidential information to remain in-house and prefer to own and operate their own log file analysis software. Some packages don't have to reside on the Web server to operate and can process log files from another computer behind the corporate firewall.

Although some packages are intended for Unix servers only, others are available for multiple platforms, including Windows NT and Windows 95, or are platform-independent like Aquas' Bazaar Analyzer Pro 1.0, which is written in Java. The most important criteria is whether a log file analyser can handle a Web site's log files (some choke on files over 50Mb), whether it provides the kind of reports you need and how easy the analyser is to learn and use.

There are other key points as well. Must the tool run locally or can it access log files on other sites? This can be important for corporations wanting to manage Web sites running from other company locations or, for smaller setups, those hosted on third-party servers such as those operated by an ISP. Does the package have a built-in database or, if not, can it export in formats readable by Microsoft Access, Excel, Lotus 1-2-3 or other applications? Can the analyser resolve numeric IP addresses for you and provide more detailed information about who is visiting your site?

Finally, what does the tool cost? Shareware, freeware and low-cost solutions are available that may suit the needs of small businesses that can slide by with bare-bones reports and limited capacity. More advanced analysis software is highly expensive, but larger organisations that require this kind of horsepower will likely see these costs as a drop in the bucket.

In short, log file analysers can tell you:

• How to increase the number of visits to your site. Once you determine which links lead desirable visitors to your Web pages, you can generate more visits by using banner ads on the site.

• Which pages are being read the most. The raw number of hits to a page is not as important as how many visitors are sticking around to read the whole thing. You may find that important messages are being ignored.

• Visitor demographics. Are you attracting the kinds of visitors you can benefit from, or are your pages simply a haven for those looking for free software to download? Log file analysis tools can tell you what kind of organisations are visiting your site. They can even tell you what country they come from.

• Which files are being downloaded. Are visitors grabbing your product demos? Is most of the interest concentrated on press releases? Do your graphics offer the most allure? Considering the cost of bandwidth, not knowing can be costly. Having the information available can help you offer more of what visitors want.

• Who is using older browsers. Your site may intentionally attract academics or other kinds of visitors using older browsers. If so, you may want to curtail all that effort put into HTML 3 features from which only a small number of visitors can benefit. Or, it might make sense to offer pages in dual versions.

• If you have any missing files. You may not know that many visitors are having trouble accessing a page until you scan the error reports produced by your analyser.

• When your page is most popular. There may be a reason - one which you can exploit - why you receive more hits on a certain day or at a certain time of day.

Reviews of the seven products that underwent hands-on testing follow.

We also looked at other packages which weren't on the market when we conducted our tests. Several, like Accrue Software's Insight or Andromedia's ARIA Recorder Reporter, required a Sun Solaris server which we didn't have access to, so be sure to check out the table on page 96, comparing those we couldn't test with those that follow.

Bazaar Analyzer

Why build a Web traffic monitoring tool in Java? First, platform independence.

The same software can run on any hardware platform that supports a Java interpreter, including Mac OS, Windows 95 and NT and six varieties of Unix. Second, since the user interface is the browser, any authorised person can access the data without special software. As the only Java-based Web traffic monitoring tool we know of, Aquas's Bazaar Analyzer Pro 1.0 (BAP) is alone in offering these advantages.

Once BAP is installed, a Webmaster can monitor a site's activity from any location that can access the server using a Java-enabled browser.

This is good news for administrators who want to track traffic from a remote location, or for organisations that have a Web site located externally on an ISP's server.

BAP can generate reports on demand or according to a schedule. It will email activity reports to you so you don't have to keep checking the server.

Simply check your mail and download the info you need. In fact, anyone in the organisation with permission to access log file data can set up a personal view that displays only the reports they need. BAP excels at real-time tracking. You can monitor visitors online, while they are accessing pages from your Web site. Alerts can be installed to let you know when specific users are visiting your site.

With this analyser, you can generate graphs and tables suitable for sales, marketing and systems administration. These include bandwidth reports, demographics, and most/least requested pages. You can even view which pages a visitor used as the entry/exit to your site.

BAP shrewdly keeps its older reports on file so you can track current and past performance using a history feature. It has a great feature for Web administrators who allow users to change their own pages within your Web site. You can provide BAP with a listing of URLs to monitor, and when changes are made to those URLs, the program will notify you by email.

Price: $999

Contact: 00 1 408 737 7122

URL: www.aquas.com

Pros: As far as we know, it's the only Java-based Web monitoring tool on the market

Cons: Needs a Java interpreter

Conclusion: Great features that make monitoring sites from a remote location very easy.

Market Focus

Interse's Market Focus 3 gained unrivaled marketing muscle and credibility when Microsoft scooped it up earlier this year. While it runs only under Windows 95 and NT, it can work with more than two dozen types of Web server log files, in addition to a proprietary extended log format that can be installed with many Web servers.

Hidden under a deceptively simple point-and-click interface is a host of powerful features. For example, built into the package are dozens of default report formats using HTML and Word file formats. There's a heavy emphasis on information likely to be of interest to marketing departments such as tallies of repeat visitors, how many visits to a site are made before reaching the order page, which ads attract the most visitors, and referring URLs of visitors who actually place orders.

Move beyond those reports and the heavy-duty customised analysis features of the developer's edition are a statistician's dream. Data can be grouped, filtered, averaged and compared with results from multiple servers. Market Focus can collect data from several Web sites hosted on different servers and provide a snapshot comparing them at a given point in time or using other criteria you set up.

A marketing manager for a European region could create a report listing all the users in the EC who had accessed an internationally oriented Web page in the past month from a particular search engine. Those figures might justify placing a banner ad on the search engine to attract even more visits.

Users can even set up multidimensional views of data to examine the relationships between several different factors. Market Focus also allows you to map log data to any terminology you choose, so reports can be formatted using corporate nomenclature (clients instead of customers, for example) rather than classifications the programmers thought you should use.

Where some log file analysers choke on files larger than 50Mb, the upscale versions of Market Focus handle 1Gb files with aplomb. As you've probably figured out by now, this program calls for substantial hardware - a Pentium PC with 32Mb to 64Mb of RAM. But, if you need the kind of in-depth marketing analysis that Market Focus provides, neither the cost of the hardware requirements nor price of this software is likely to be of much concern to you.

Price: $695 for standard edition; $3,500 - $7,000 for developer's edition

Contact: 00 1 408 732 0932

URL: www.interse.com

Pros: Upscale version can handle mammoth files

Cons: Can be expensive - the program needs expensive hardware to run

Conclusion: Top-of-the-line features and support.

MKStats

Matt Kruse's MKStats 2.2 is a low-cost log file analyser that provides a basic selection of services for individual Web page operators, non-profit organisations, small businesses and ISPs that don't have big budgets.

It offers easy installation and a full complement of the most commonly used reports - although they're limited to simple ranked lists.

MKStats is typical of the shareware, freeware and low-end commercial Web-site analysis tools you'll find available. At the high end of the scale, there's a $300 licensing fee for ISPs that host Web sites for more than six businesses. At the low end, it's free to certain individual - non-commercial users such as college students. But that doesn't mean this package is suitable for only very small sites.

You won't find a lot of fancy charts or automated reporting tools in MKStats. Instead, the product reads log files - up to about 50Mb - and produces summaries organised by activity or volume, along with a handful of simple graphs.

You can get lists of where your visitors are coming from and which pages are being accessed, and view hit trends as they change. One listing, for example, provides a 24-hour average of hits by hour, so you can see what time of day your site is busiest.

The utility lists top 10 pages, domains, countries, hours of the day or days of the month, and can also provide raw data with a complete list of any page or graphic on your site with 10 or more accesses.

If you have a small site and don't need elaborate reports or a lot of handholding for installation and support, MKStats may be for you. Non-profit organisations, including educational and governmental institutions, can purchase a license for as little as $50. Small businesses and ISPs pay between $100 and $300, depending on their size.

Price: $50 - $300

Contact: No phone number available but you can download a trial copy at www.mkstats.com/download

URL: www.mkstats.com

Pros: Attractively priced. Suitable for small sites as well as large Web hosting providers

Cons: Doesn't contain any automated reporting tools

Conclusion: Installation is fairly simple. A basic utility with simple graphs and reports. Basically a low-end commercial Web site analysis tool.

Net.Analysis Desktop

Net.Genesis's Net.Analysis Desktop 2.0 is a standalone log file scrutiniser for Windows 95 and NT platforms and has its own built-in relational database engine. It can either operate on your server processing local files or download log files from remote servers such as Unix boxes that can't run Net.Analysis Desktop on their own by means of its built-in FTP feature.

Although Net.Analysis Desktop is a capable program, it lacks the speed and power necessary to crunch really large log files. Perhaps its main strength is the flexibility of the application's database filters which let you create reports that capture information about such details as specific users, referring URLs, time ranges or domains, while ignoring everything else. This lets you, for example, look at nothing but downloads of particular kinds of resources such as audio files, CGI programs, HTML pages or video files and create your own classes that encompass one or more of these. Snapshots of this type can show you information such as which banner ads attract the most click-throughs.

You'll also find that Net.Analysis streamlines setup. Define a log file and its location once, and the program memorises the file and places it in a log manager list for retrieval at any subsequent time.

If you've had experience with other analysers you may find Net.Analysis Desktop a bit quirky. Unlike most other log file processors Net.Analysis doesn't automatically generate external reports. It uses its own database capabilities to analyse your information and display the results on-screen.

You can scan these on-screen reports or generate a graph and export them as HTML, Microsoft Word or Excel. Webmasters with security concerns may find Net.Analysis Desktop's standalone configuration comforting, but this program isn't a great choice for those who want to retrieve and examine log file reports from multiple locations.

Price: u199

Contact: 01638 569600

URL: www.netgen.com

Pros: Flexibility of the application's database filters Cons: Lacks the speed/power needed for large log files

Conclusion: A capable program with basic features.

NetIntellect

WebManage Technologies' NetIntellect 2.1 has all the refinements you'd expect from a mature product, including flash and substance such as the colourful 2D and 3D charts. It's graphs are impressive yet make it easy to interpret complex relationships at a glance.

NetIntellect's automated tools for creating databases and retrieving log files from remote sites with built-in FTP services are flashy, but make this is one of the most useful log file analysers we tested.

Although NetIntellect is a 32-bit Windows 95 and NT application, it smoothly integrates log files produced by a variety of Web servers on Windows, Unix or Mac platforms. It doesn't need to be installed on the Web server itself or interface directly with the server's software. Simply tell it where to find the log files you want to analyse; the program fetches them and uploads finished reports to the address you specify via FTP.

NetIntellect has a built-in scheduler for generating reports and can email them automatically. Reports can be saved in HTML pages you view in your browser or stored as plain text, Word files, Excel or Lotus formats using your own custom spreadsheets. These choices are all clearly presented in lists displayed in a three-paned window, with toolbars for fast access to common features.

Since massive amounts of raw data are often hard to interpret, NetIntellect provides more than 80 predefined charts, graphs and tables, and presents multiple reports in different windows so you can compare results over time or correlate information from several reports.

We were impressed with NetIntellect's sophisticated wizards, which lead you through creating new databases from log files. You can also tell the program to ignore certain file types when compiling statistics.

Early versions of NetIntellect were reported as being a bit slow, so WebManage released a March upgrade with two processing modes: standard processing mode generates a predefined set of reports at high speed regardless of the size of your log file; an optional extended processing mode compiles an indexed database at a more leisurely pace, but it can later access information from the indexed database quickly using any of the many report formats available.

NetIntellect may be your choice if you want a combination of a simple interface, automated retrieval and reporting tools, and a large selection of reporting formats and graphical displays. You'll need a fairly powerful Windows 95 and NT machine to run it, but a Pentium 100 with 32Mb of RAM and 250Mb of virtual memory are recommended.

Price: $230

Contact: 00 1 503 294 7025

URL: www.webmanage.com

Pros: It smoothly integrates log files produced by a variety of Web servers Cons: You will need a powerful Windows 95/NT machine to run the program

Conclusion: A useful log file analyser.

NetTracker

If you want to install a log file analysis application once then view reports from any location without special software, Sane Solutions' NetTracker 2.0 is a top choice. NetTracker installs on your server but can deliver textual and graphical summaries remotely using your Java-capable browser as an interface.

Connect to this modestly priced analyser through your intranet or Web site and its standard reports - detailing page usage, referring URLs and profiles of visitors - are available from the dynamically generated HTML page. Because all the number-crunching takes place on the server and only the reports are transmitted to your desktop, this method of working with NetTracker is well suited to slow connections.

NetTracker's easy access to reports, without requiring extra software, plus its simple browser interface, make this analyser a good choice for non-technical marketing and product managers who need to monitor activity on a Web site quickly with a few clicks. Data can also be exported in formats compatible with Microsoft Access, Excel and Word.

Although not written in Java, NetTracker uses Java applets and JavaScript to display results in your browser which must be a Java-capable version of Netscape Navigator or Microsoft Internet Explorer. It doesn't have the cross-platform universality of Java-based Bazaar Analyzer Pro, but Sane Solutions does offer versions for both Windows NT and Unix servers.

Price: $295

Contact: 00 1 914 697 7555

URL: www.sane.com

Pros: Uses JavaScript

Cons: It doesn't have cross-platform universality

Conclusion: Suitable for non-technical professionals.

WebTrends

Hidden behind a lean interface that contains only seven tool buttons and two drop-down menus lies what is arguably the industry leader in Web traffic analysis tools, e.g.Software's WebTrends 3.0. Although it has been on the market for over a year, WebTrends has evolved through three full versions and ranks as one of the most widely used log file analysers available. So what makes the package so popular? Speed.

WebTrends can often do the same chore in a matter of minutes. It analyses log files in real-time, in the background under Windows 95 or as an NT service, maintaining an up-to-date index that it can search through. So, if you decide you need a ranked list of pages accessed by visitors from Eastern Europe in the past week, WebTrends doesn't have to evaluate your log file from scratch. It can be set to create reports every five to 10 minutes, if you like, and email them to the addresses you specify. Reports can be formatted as HTML pages for retrieval by your browser from another location, as Microsoft Word or Excel files, or in other formats. It gives you more than 60 tables and graphs to choose from. Included in this number is a clutch of views that will delight marketing moguls - things like reports on scripts and files executed when visitors access advertisements.

WebTrends was by far the easiest analyser to install and use. During setup, it recognised Microsoft's IIS running on our Windows NT machine and offered to activate extended logging. It located our current log files and added them to its list and was the only Web-site tool to virtually configure itself. While WebTrends works best running on your server, it can analyse log files from non-Windows platforms, including Unix and Mac servers.

Price: u290

Contact: 01488 681743

URL: www.webtrends.com

Pros: Easy to install. Speed and powerful features

Cons: Too good to be true?

Conclusion: Industry leader in Web traffic analysis tools. PICK YOUR FANCY

While we did sample a good cross-section of industry-leading packages, we didn't have the facilities to test a number of other worthy products.

Here's a rundown of four that didn't get tested, two of which are Unix-based.

ARIA Recorder Reporter

This analyser from Andromedia runs on Sun Solaris (although the publisher plans to support other Unix flavours and Windows NT in the future). ARIA takes advantage of Unix's threading capabilities to perform simultaneous application threads for optimal speed, which helps this tool scale up easily to accommodate peak periods. It can generate HTML-based reports for viewing in a browser. (www.andromedia.com)

Insight

Accrue Software's Insight is a Unix-based multipurpose site-analysis tool that not only provides marketing-style information about visitors, but key measurements of how your Web site is performing. It can track multiple visits to a site by the same user as easily as it measures the time required to deliver a given page, effective line speeds and other factors. It uses a Java-enabled Web browser to manage and present reports.

(www.gauge.com)

Hit List Standard 2.5/Pro 2.5

MarketWave's Hit List Standard 2.5 and Hit List Pro 2.5 are Windows 95 and NT applications that provide essential information for Web site operators who depend on advertising, click-throughs or sales to justify their operations.

Pro 2.5 can trace which search engine was used to locate your site and retrieve the actual keywords used by a visitor to locate a page. Pro 2.5 offers dozens of reports. Its advertising report generates information about visitors' use of ad banners, gauging the number of impressions and click-through rates. The advertising performance report offers insight on how your ads are performing on other sites. Standard 2.5 is a scaled-down version for smaller sites that creates fully formatted Microsoft Word reports but offers only 10 pre-defined formats. A third version, Hit List Enterprise 2.5, is available for high-volume sites that generate more than 100Mb of log files per day. (www.marketwave.com)

Statisphere

O'Reilly's Statisphere is a 32-bit Windows 95 and NT application, but it can analyse logs from virtually any Web server. Its strength is its ability to create dynamic, real-time graphs that can be updated every 60 seconds, showing current system status without the fuss of generating a report.

After constructing an initial database for a site, Statisphere updates itself regularly using only the new data. Because the entire log file doesn't have to be downloaded or interpreted each time a new report is requested, this analyser can generate HTML reports quickly from remote locations. Statisphere makes heavy use of wizards to lead you through complex tasks. It's suitable for large sites and truly mammoth log files.

(software.ora.com)

The best of the test

While you can buy more sophisticated and/or more versatile Web site analysis tools, e.g.Software's WebTrends 3.0 offers the best combination of features and the best value.

Despite its bargain price, WebTrends is fully capable of handling log files from the largest sites. Since it does its work in the background, you never have to wait for results. At the same time, WebTrends is simple enough to operate so that even new Webmeisters putting up their first sites will have no difficulty installing or using the package. Indeed, its ability to locate and identify your server software and configure itself makes WebTrends the logical choice for any Web site neophyte.

Honourable mention must go to Bazaar Analyzer Pro 1.0 and NetTracker 2.0. Both boast the kinds of features that all log file analysers will have in the future, for instance, the ability to retrieve reports remotely using ordinary browser software and a heavy reliance on platform-neutral tools like Java and JavaScript.