CIO

The Numbers Game

In addition to the difficulty of generating accurate predictions from minimal historical data, e-commerce forecasts have foundered on the definition problem. Individual research organisations use different yardsticks to measure the impact of e-commerce. They range from e-spend to e-business transactions to e-commerce transactions to Internet-based commerce. Each term comes shrink-wrapped in its own subtle distinctions, which often elude the uninitiated while producing dramatic changes in final figures. The haze deepens when reports mix business-to-business (B2B) stats with business-to-consumer (B2C) figures.

This definitional dilemma can descend to very basic levels. IDC, for example, defines "user" as anyone who accesses the Internet at least once a month. If others base the term on once a week use, or once a year, the potential for confusion becomes obvious.

Not surprisingly, the definition of e-commerce itself is flexible. IDC defines it as "an online commitment to buy a product". That includes placing an order online (though it doesn't embrace actual payment). Under other definitions, simply browsing for products online, with no order placement, qualifies as e-commerce.

Clued-up customers of research firm products have their own rules for extracting value from the swamp of ambiguity.

"We tend to look at all the reports and take a view based on a number of them [the research organisations]," says Oracle Australia MD Brian Mitchell. "Gartner and IDC tend to have a broader perspective and Forrester has good information. Because of the different definitions, there are a myriad of ways in which these numbers can be viewed. We don't pay much attention to the figures in a micro sense. It is more in the macro sense of just seeing if the numbers are still big and going up."

Ramin Marzbani, the often-controversial CEO of market research firm ACNeilson.consult, echoes that sentiment. Forecasts at most should be treated as signposts telling you whether to turn left or right, Marzbani says. More than that, such as relying on them to tell you how quickly to turn, is putting more weight on them than they are designed to support.

"There are 20 numbers out [purporting to describe the dimensions of Australian e-commerce] and almost all of them are useless because they don't make clear what is being included and what isn't," Marzbani says. "Taking the total value of electronic transactions in Australia, you could argue it is worth trillions already because all the payments and clearances between banks are electronic transactions."

Page Break

IDC Asia Pacific research vice president Dane Anderson admits the excesses of the Internet bubble were reflected in the "exuberance of some boutique [market research] firms". But he defends IDC's electronic commerce forecasting record, describing it as "conservative".

The basic tool IDC depends on for broad perspective forecasts of e-commerce hasn't changed, Anderson says. It is the Internet Commerce Market Model (ICCM) developed in the US in late 1995 and rolled out in the rest of the world the following year. The model's strength, he says, rests on the assumption that conducting Internet commerce requires connected devices; and tracking populations of installed devices is IDC's traditional forte.

Marrying user surveys and vendor interviews to the results cranked out by ICCM yields what Anderson calls "a powerful view grounded in reality". Inputs from regional analysts round off the forecasts and given them localised credence. "As time progresses, our forecasts have become more confident because the environment becomes more stable and more predictable," Anderson says.

Signs are appearing of a change in the forecasting landscape as the terra nullius of electronic commerce fills up with population centres. Gartner for one is moving away from general e-commerce predictions to focus on more tightly defined segments. "We found those sorts of broadly based forecasts have been unhelpful to our clients over the past year or so," says Richard Harris, vice president Asia Pacific for Gartner G2, the group's new strategic business planning research service.

At the start of any new era or technology, broad-brush estimates are necessary evils, he says; but generalist figures induce a false sense of security because "they give people the idea they know what is happening when they really don't". As the new era fragments into differentiated zones of activity, "unless you drill down to look at specific areas, the [generalised] figures become quite meaningless", Harris says.

Unfortunately, even in more specialised areas, the pundits' predictions haven't proved particularly precise. The e-marketplace segment, for example, was tipped to spawn anywhere from 900 to 5000 electronic exchanges, says Accenture's Gattorna. "We never got anywhere near that. In Australia, there are probably no more public markets than you can count on one hand. And the people who talked the numbers up have been very slow to track the failures," he says.

E-tail forecasts went wildly wrong because pundits were wowed by the front-end systems that made it possible for customers to visit Web sites and place orders.

What the experts missed was the slower rate and higher cost of developing the back-end systems needed to deliver the goods being ordered.

As to whether the quality of e-commerce forecasts is improving with experience, "I haven't seen any credible figures lately in the supply chain area," says Gattorna. "I think we are in an interim period. People are licking their wounds and reconsidering the feasibility."

Businesses engaged in the supply chain sector will be relying less on input from the e-commerce research companies and more on their own experience, he suggests. "Frankly, macro hype is useful to talk things up in the beginning. But it doesn't mean a thing to a company which is at the stage of fine-tuning online relationships between buyers and suppliers."

A bellwether player in the e-procurement space, Cyberlynx, whose stakeholders include the Commonwealth Bank, retail colossus Woolworths and brewer Lion Nathan, pays very little attention to e-commerce forecasts. It depends far more on input from its customers to shape forward strategy, says business development manager Steve Harmer.

In the early days of e-business, numbers that gave a pointer to market sizes were valuable but they have been transcended in importance, he says. "What we want now are things that can't be set up in numbers such as implementation issues within customer environments.

"The market forecast groups tend to take a sample and project it up. We get back down to our customers, " says Harmer. "We have a tight target market and we talk to a lot of customers on a day-to-day basis. That gives us a good understanding of what they want and their speed of acceptance of new technologies."

Some companies never felt compelled to touch base with e-commerce projections before taking the Internet plunge. When Amway Australia launched itself into Web commerce in 1998, "to be honest, we didn't look out into the marketplace and say this is going to be worth it to spend this kind of money", says Greg Bowman, the company's sales and marketing director.

The mechanics of the Web seemed to fit naturally with Amway's business, which in Australia embraces 100,000 self-employed distributors of its health, home and beauty products. "It wasn't a question of show us the numbers to prove whether we should make the decision or not," says Bowman. "We recognised right away this was a channel we wanted to be involved in. We felt if there was one company in the world well suited for e-commerce activity, it was us."

And so it has proved. Some 47 per cent of Amway's revenues now flow through e-commerce and it has spent next to no money advertising its Web site, www.a2k.com.au. By contrast, early online rivals such as dstore splashed out millions of dollars to attract customers.

The difference is that dstore, like many of the pundits' reports which made it a poster boy of the dotcom industry, is now mulch.

The Intelligent Users Guide to Online Business ForecastsFive tips from the pros on avoiding e-forecast poisoning:

1. Clarify and understand all definitions.

2. Ignore forecasts which look out more than six to 12 months.

3. Don't grab forecasts based on global figures that have been scaled down to regional dimensions. Localised political, social and pricing issues make the numbers a poor bet.

4. Forget macro and focus on the niche where your company is positioned.

5. Numbers, even if accurate, are only a starting point. They give the "what" but custom-tailored answers on "how" and "why" and "when" should precede any large investment.

Page Break

It's a question that every e-commerce wannabe faces:

How much can I trust the deluge of predictions on how fast Internet commerce will grow?

Shred all the electronic commerce forecasts made so far. Now sprinkle them over the planet and watch them cover it to a depth of one metre. Among the visionary-for-hire brigade producing these forecasts, that figure will be attacked as a vicious exaggeration.

Okay, make it half a metre .

The precise depth is irrelevant. What's important is that e-commerce predictions actually may be more useful as global mulch than as guides to corporate action.

"Meaningless" is the word most heard these days about the prognostications pumped out several years ago as the curtain was rising on e-business mania. There's evidence to suggest such an assessment is as valid for 2001 as it was for 1995. Forecasts by different market research groups over the same time frame have varied by half a trillion dollars. For example, a Gartner forecast issued last year predicted Asia Pacific e-business transactions in 2004 will reach $US1 trillion. By Forrester Research calculations, the value of the same transactions zooms to $US1.6 trillion.

Nor do you have to journey four years into the future to strike serious contradictions. A recent Cisco-commissioned study by The Allen Consulting Group cited "Internet economy revenues" of $28 billion in Australia for 2000-01, International Data Corp (IDC) estimates Australian "e-commerce transactions" of $4.6 billion in the same period, while the National Office for the Information Economy (NOIE) prefers the figure of $1.3 billion for "Internet-based commerce" in 2001. To add a little variety, the Australian Bureau of Statistics put "Internet e-commerce sales" at $5.1 billion in Australia as of June last year.

With that kind of spread, it looks like you pays your money and you takes your pick. So why can't the prophecy-meisters get their crystal balls vibrating on the same wavelengths? Partly it is due to the greenfield effect. Infotech forecasters IDC and Gartner do superb work predicting hardware and operating system futures where they can lean on 30 years of experience and past trends. When Internet commerce burst on the scene in the early 1990s, however, the pundits lacked any historical reference points to extrapolate from.

Facing data deprivation, our brains have a trickly little habit. They prefer to invent phantoms rather than accept blank space. Sit in a dark room with your eyes open and your brain will try to convince you there are shapes and patterns swimming in the blackness. Something identical happened in the mid-1990s when market researchers were pressed for numeric predictions on how the Internet business frenzy would play out. Staring into the techno-equivalent of a dark room, they produced a set of phantoms called e-commerce forecasts.

In the wake of the dotcom fiasco, any link between these forecasts and reality proved entirely coincidental and did considerable damage to reputations. "I think every one of them [the forecasting specialists] got caught," says Accenture Asia Pacific managing partner, supply chain practice, John Gattorna. "Just about everyone who tried to forecast numbers must look back and say: ‘We got it wrong'."

Marc Phillips, CEO of online research company APT Strategies, has two words of advice for consumers of predictive reports about e-commerce: caveat emptor. APT Strategies conducts e-commerce surveys to produce real-time snapshots of the state of the play and Phillips knows the numbers game inside out. In the late 1990s, he was in a short-lived joint venture with US firm research firm Jupiter Communications (now Jupiter Media Metrix). The experience gave him a ringside view of the process of "religiously spitting out vertical sector forecasts for three to five years out".

Phillips says a lot of statistics are crunched in accordance with established methodologies to produce forecasts. The technique takes many factors into account and generates figures, which have their place. However, their apparent solidity is an illusion, he says, because they are often based on assumptions that flow from the thought processes of "one or two people" in any research firm.

Phillips suggests that forecasting technology business futures is a farce in which the apparent victims are willing accomplices. Companies looking for material to support a business case put pressure on the research companies to supply future outlooks. "First people scream at you to produce forecasts, then they scream at you if the forecasts are wrong." When the figures fail to come true, "it is hard to know who is more guilty", says Phillips, the forecasters for pretensions of infallibility or their customers for being too credulous.