Contrary to popular opinion, the battle between online merchants and brick-and-mortar merchants is not new. It started 27 years ago when NASD, the National Association of Securities Dealers, built Nasdaq, a very pre-Internet network connecting brokers selling stocks that weren't listed on the New York exchange and providing up-to-the-minute quotes of all stocks traded. Since then, the New York Stock Exchange (NYSE), the biggest and most prestigious stock market in the world, has watched with increasing envy as The Nasdaq Stock Market Inc. and the value of stocks traded there have grown. And grown.
Officials from NYSE, where traders still clutch handfuls of paper and scream to each other across a crowded room, have tried to persuade a new generation of technology giants, such as Cisco Systems Inc., Intel Corp. and Microsoft Corp., to sell shares on their floor. The tech stocks declined their offer, choosing instead the familiar terrain of IT-driven Nasdaq, where trades are made by code flicking silently through silicon systems.
Two months ago, in a symbolic concession to the power of technology as well as to the value of technology stocks, the NYSE revealed that it was exploring ways to incorporate an electronic trading system that would let it trade stocks that have until now been traded on the Nasdaq. Exactly how the Big Board will accomplish this goal is undecided: It may partner with one of several electronic communications networks, which trade stocks from all exchanges at all hours; it may build its own network; or it may work with Nasdaq itself. The last may be the most sensible choice because if the NYSE has learned anything from its months of explorations, it's that building a network as fast, powerful and secure as Nasdaq's is a formidable task.
In the past two years, Nasdaq has spent $80 million on a massive upgrade of its Trumbull, Conn.-based enterprise, which will soon connect 10,000 traders all over North America. At least $50 million of that went to computer equipment.
What Nasdaq got for its money may well be the fastest, most secure, most reliable and most finely tuned network in the world, excluding those with military purposes. And it better be. Because every workday, five days a week, about $22 billion worth of stock trades depend on it.
For traders, the madness starts early, as soon as the Nasdaq log-on appears on their screens. The first order of business is to execute the backlog of trade orders that has built up overnight.
"About 10 percent of the total volume of the day may happen in the first 10 to 15 minutes of the morning," says Gregor S. Bailor, Nasdaq's executive vice president and CIO. "That translates into around 1,200 transactions per second, a figure that slumps to a mere 500, 600, 700 transactions per hour during the middle of the day. "On most days, says Bailor, Nasdaq traders conduct more than a million-plus trades, requiring several million quotes.
"Yesterday was nearly a billion-share day," Bailor says. "On a billion-share day, you're talking gigabytes of information being created virtually by the hour." A few years ago, billion-share days were rare. Now they're not. In one 5-day period last fall, the billion-share mark was topped three times. No one imagines that the volume of trades will decrease. And no one, particularly no one at Nasdaq, would imagine trying to handle that volume the way it is handled at the NYSE.
"At the Big Board," says Bailor, "someone carries [a piece of paper] across the floor. We don't believe in that approach. It is more costly and it is more inefficient. The primary mechanism of the Nasdaq market is through our network system." Consequently, he says, they never bring down the Nasdaq market at times of high volume. Bailor argues that maintaining a physical stage also makes tracking trades and market action far more difficult. When information moves from the virtual world to the real one, he says, it falls off the virtual world's radar screens.
"We don't have the hurdles in trying to keep a fair and orderly network, because you have visibility across the network," he says. "You have the ability to deal with multiple people getting into and out of a stock without having a space issue on the floor. It makes it possible for us to manage any level of activity in a particular stock."Nasdaq's impressive capabilities rely on 324 top-line Tandem fault-tolerant mainframes, more than 50 Dell Computer Corp. and Compaq Computer Corp. servers, and a small forest of 18 Unisys Corp. mainframes. All this power is obsessively dedicated to providing two things: speed and reliability. This system cannot bog down. A half-second delay means the difference between a trader making or losing a buy. Neither can it break down. In Trumbull, dozens of thick, steel racks hold hundreds of thousands of pounds of batteries that make up the facility's doubly redundant uninterrupted power source. Nasdaq tests them relentlessly, as it does the three huge diesels that shine like new stock car engines and the huge chillers that provide the gusts of cold air needed to keep all the computer equipment from baking to death. It's top-end technology designed to be pushed to the limit, so when officials of the NYSE contemplate creating a network that would rival NASD's, they see a technological challenge of an almost unimaginable scale. And with a whole new set of problems.
Small Time, Big Trouble
Network technology may have eliminated distance, but it hasn't eliminated time, and the time it takes for data to move from, say Trumbull to Hartford, is still less than the time it takes for data to move from Trumbull to Boston. That difference, however minuscule, is big enough to tip the scales in favour of those close to 1 of the 22 hubs that Nasdaq maintains.
"The further you get from a hub," says John Hickey, Nasdaq CTO and executive vice president, "the opportunities for delay intensify. Without correction the market maker most removed from the central processor would see a quote slightly behind the market maker closer to it," he says. "That would give him an advantage because he would be able to enter orders or executive trades more quickly than someone who was more remote."To fix this, Nasdaq adds a tag to each message it processes. "A message that is released from the host is time-stamped with an embargo time," Bailor says.
"Messages go to hubs with the internal notation that says 'I can be delivered after such and such a time.' In cases where it's the normal situation, the messages can be held so all messages are released at their point of presence.
This ensures everybody is released at the same time. Our criteria was 500 milliseconds. Our performance is 250 milliseconds. It's almost imperceptible to the naked eye."If the planned T1-based system were brought up gradually, those clients on the upgraded stretches of the network would have at least a temporary advantage.
Consequently, Nasdaq must operate the old, 56KB system until the moment all the T1 links are complete and fully tested. Then, in a single moment that will leave no one advantaged or disadvantaged, Nasdaq will leap into a new world of bandwidth.
If there is one big reason Nasdaq operates on a private network rather than on the Internet, it is Bailor's conviction that private networks limit the risk of unwanted visitors. Nasdaq users must either dial into it or have a hard-wired connection. A user can be on in only one place at a time, and if someone other than the authorised user tries to sign on with that user's ID, she is not allowed onto the system. Punishment for unauthorised behaviour is swift and certain, as he knows personally. Hickey does at least one tour a week. He used to have his own account, which he used for demonstrations. He accidentally made some trades. Luckily, the market was kind to him. But he no longer is able to use the very system he's in charge of.
NASD executives say they don't know if the Internet can work as a trading vehicle; the openness that has fuelled its growth makes it insecure enough to fuel their doubts.
"You don't have protection on the volume," says Bailor. "You don't have protection on anything that's in the pipe."What the future may hold, however, he and others aren't saying.
Tests and More Tests
"MIS departments are inherently risk-averse," says Hickey. The problem is, in the information industry, the risk of falling behind and becoming prey to better-equipped competitors is as great as the risk involved in constantly adding new capabilities and systems. "You can't stand still because if you do you'll lose the race," Hickey says. "You need to continually add functionality."To keep ahead of the curve, Nasdaq has changed even more radically in the last decade than the information industry has itself. One change involved hosting the network. Before the upgrade, NASD operated its own network; now MCI does that job. "You can't be afraid to make changes to your system," says Hickey.
"You just have to have in place the process and procedures to make sure the changes you make are bulletproof."At Nasdaq, all changes from software fixes to advances in network technology are thus tested and tested and then tested again. Clients are never part of the software testing process. "We can't release beta versions and expect Nasdaq to work every day," Bailor says. "We test 19 browsers to make sure we run on all the things we're supposed to."And when programs do finally make it to user machines, they are tested extensively during off-hours before they actually operate in the real world.
That is what happened with Nasdaq's Y2K upgrades. "There is an industrywide Y2K test sponsored by the Securities Industry Association," says Bailor. "We come in on a designated weekend, starting on a Friday, night and have traders drive the system all day Saturday and all day Sunday."One Last ThingAnd just in case those multiple backup tests don't eliminate all the risk, Nasdaq has built an entire backup facility in Rockville, Md. Regulations didn't require it, Hickey says. Reality did.
"With our engineering firm, Kling Linquist," he says, "we conducted extensive research on best practices and concluded that to provide the service level required of our marketplace we would have to provision N-2 redundancy in our critical infrastructure components."The redundancy added 30 percent to the facility's cost, but Nasdaq officials believed they could not live without it. Although not as powerful or up-to-date, the Rockville plant can carry Nasdaq's entire load for an indefinite period. In fact, several weeks a year Rockville carries the Trumbull load to make sure the facility is fully functional. Switching between the two sites takes time, however, and that continues to bother Hickey. Four years ago, after a menacing squirrel shut down the Trumbull plant, Rockville went from backup to fully functional in half an hour. Hickey wants to get it down to 10 minutes.
Hickey and Bailor have much to look forward to, with a model of state-of-the-art networking at their disposal and billion-share days growing more common. Hickey says that after spending $80 million, he would like to work on controlling costs, and he is exploring ways to run his operation inexpensively and efficiently. One thing he isn't exploring is building a trading floor where traders can race about swapping handfuls of paper.
(Tony Seideman is a freelance technology writer in New York City. He can be reached at firstname.lastname@example.org.)Nasdaq's Other NetworksNasdaq's private network is its most important online service, but it isn't its only online service. NASD's intranet is used for interdepartmental communication. The intranet is used to make such material as human resources manuals, telephone directories and organisational charts available.
NASD's extranet provides information to stock issuers and traders. The association recently upgraded its Mutual Fund Quotation System so that firms can transmit information to Nasdaq via the Internet where it is processed and transmitted to newswires daily. An Internet site, www.nasdaq-amex.com, provides investors around the world with financial information on all member companies.
The site averages more than 20 million hits a day. -T. Seideman
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.