Despite the UK?s lack of enthusiasm for the euro, few can doubt that the fate of the nation now rests largely within Europe ? our days as an island race are fading faster than the survival prospects of sterling.

But for the UK?s former public utilities, the advent of deregulation and open markets can be viewed as both a curse and a blessing ? ending the comfortable days of customer monopolies, but affording unimagined opportunities for growth throughout the EC.

Although the battle lines are still being drawn, it will only be a matter of time before UK utilities clash with their counterparts in the rest of Europe for the lion?s share of the market ? a battle in which IT is likely to play a decisive part.

Scottish Power executives see the company positioned as a leading European supplier. In recent years, the company?s main role as an electricity supplier has been complemented by gas, water and telecommunications. Deregulation is making these markets into a battleground in which only the most efficient suppliers ? with the most effective marketing ? can expect to survive.

Complex tariff and service messages need to be communicated to audiences which are neither tuned to receive them, nor experienced in evaluating them. Only in telecoms is the new open market just beginning to make an impact on the residential customer. To make competitive and profitable offers to consumers, Scottish Power must be able to understand their requirements in as much detail as possible, using the latest data-mining techniques to comb through millions of customer records.

At the heart of Scottish Power?s customer evaluation activities is a WhiteCross WX9010 data exploration system ? a strategic system whose use can be traced back to the company?s purchase of another utility, Manweb, some three years ago.

The acquisition was intended to give Scottish Power an early presence and credibility in non-Scottish markets because Manweb was not short of computing resources ? with mainframe systems dedicated to tasks, which included accounting, maintenance and retail control. What was also clear to Manweb, however, was that running a complex query exercise ? analysing payment point data for example ? would always be difficult.

Special programs would need to be written; data would need to be extracted from production systems and recompiled; and considerable machine time would be needed for queries to run, preventing the mainframe from performing its usual tasks.

Manweb?s IT team realised that the most efficient solution would be to put in place a separate system to take and manipulate data. Six different options were considered before the team decided to use a computer bureau running a mainframe similar to its own. The solution had a lot to be desired. Data had to be agglomerated and summarised before it could be downloaded; assumptions made in specifying which data should be used restricted the questions that could be asked, and the depth to which queries could go; and as with Manweb?s own mainframes, a degree of programming was necessary, so this meant that spontaneous queries were not possible. Ultimately, because it wasn?t possible to gain full-time access to the machine, the ability to follow lines of enquiry to their final conclusion were compromised by availability.

Manweb considered building its own data mining engine, using a powerful, multi-processor PC with lots of memory and storage, but it be- came clear that such a system would still need summarised data, custom programs and considerable support. At this point, Manweb opted for the WhiteCross WX9010 data exploration system.

Tony Harper, the then database and research manager for Manweb ? and now fulfilling the same role for Scottish Power ? says that the WX9010 has proved ideal. ?We were able to quickly load masses of unprocessed data from our mainframe systems, then make joins and scan millions of rows of information in a few seconds. We could look in great depth at the data ? down to individual customer level if need be ? to identify which of our customers preferred to deal with us over a counter, and the locations they liked to use.

?In just a short time we provided the business with the knowledge it needed to put in place an alternative network of outlets.?

Today, Manweb customers have a choice of almost 200 bill-payment points and 600 purchase points for electricity tokens. In the busiest locations, Manweb opened 16 of its own customer service centres, and a trading agreement with the Midland Bank made available 140 of the bank?s branches as payment points. In more remote areas, local retailers now accept payments and throughout the region tokens are on sale in hundreds of newsagents, petrol stations and other retailers. Database exploration made it possible for Manweb to close stores yet improve customer service.

When Scottish Power acquired Manweb in 1995, it decided to take advantage of Manweb?s database and research staff, as well as WhiteCross? data exploration team ? transferring them to its Glasgow headquarters to help in its new data-mining exercise.

Harper says: ?We are starting with baseline information by looking at data on our existing customers. First, there?s the geo-demographic information that?s inherent in customer address codes. Then we can look at billing data to find out how many units of electricity, gas, water or telecoms they use at the moment. With that as a starting point, the power of the WhiteCross system lets us look at other factors ? some more obvious than others.?

Harper continues: ?One thing we are considering is whether first names can help us. If the head of a household is called Harold it?s more than likely that he is aged over 40. If Harold has a family, it?s also probable that his children are of school age. We might similarly deduce that his home would be unoccupied or lightly occupied from nine until five on weekdays, except during school holidays.

The WhiteCross system lets the team rapidly test assumptions such as these by correlating consumption against name data. If there appears to be some justification to the team?s reasoning then it can verify it by picking typical Harolds to research in detail.

One benefit of the system is that it can ask oblique train-of-thought questions such as these without any pre-programming or the necessity to pre-select data. ?We get the answers while we still remember why we asked the question,? says Harper.

He adds: ?By doing the groundwork of segmenting our customers into groups now, we allow our product managers to put together a wide variety of single and multi-utility packages representing both good value for prospective customers, and profitable business for us.

?A new breed of utility broker will soon be cherry-picking the highest profit consumers. Our stance, of understanding exactly who wants what services and when, will let us compete very effectively, defending our existing customer base, and winning new ones.?

Digging data mining: early

Data mining is currently winning new adherents because the three technologies necessary for its support ? massive data collection, powerful multi-processor computers, and the advent of sophisticated, data mining algorithms ? have all reached maturity. As a result, companies are now presented with an opportunity to generate new business through automated prediction of trends and behaviours, and the automated discovery of previously hidden patterns. For example, databases can be larger in depth and width ? enabling analysts to overcome previous limits to the number of variables and explore larger samples with correspondingly lower errors and variance. However, a recent data-mining report from analyst Ovum warns that data mining can also be a confusing technology. ?Users must learn new concepts and procedures,? says senior analyst Eric Woods, who co-authored the report. He adds: ?Data mining requires a commitment and desire to use information in new ways. It has huge potential, but it is not a magic bullet. ?It requires an investment of time and resources to ensure that the correct data is available for analysis; that the data mining algorithms are used effectively, and that results are interpreted correctly.? For these reasons, data mining is still the preserve of a few early adopters, many of whom are leading organisations in the finance, retail and telecoms industries. These organisations are attempting to find new ways to use and analyse the vast amount of data they have collected for so long and with so little benefit. ?While the benefits are real, there is still work to be done for data mining to become a mainstream technology,? says Woods. According to Ovum, the key areas of data mining which need to be improved are better integration of different data-mining techniques and greater automation of the modelling process; more imaginative and informative presentation of results; tools offering more explicit help for users setting up and managing a data mining project; and more flexible deployment options, including support for ActiveX, Java and HTML.