Critical.
Authoritative.
Strategic.
Subscribe to CIO Magazine »

Hadoop Is Ready for the Enterprise, IT Execs Say

Big companies are using Hadoop systems in big projects, despite concerns about issues such as security.

Despite some lingering user concerns about security and other issues, Hadoop is ready for enterprise use, according to IT executives at the Hadoop World conference in New York earlier this month.

Larry Feinsmith, managing director of IT at JPMorgan Chase, told a keynote audience that the financial services firm has been increasingly using the open-source storage and data analysis framework for almost three years.

JPMorgan Chase still relies heavily on relational database systems for transaction processing, but it uses Hadoop technology for a growing number of purposes, including fraud detection, IT risk management and self service, Feinsmith said.

With over 150 petabytes of data stored online, 30,000 databases and 3.5 billion log-ins to user accounts, data is the lifeblood of JPMorgan Chase, Feinsmith said.

Hadoop's ability to store vast volumes of unstructured data allows the company to collect and store Web logs, transaction data and social media data. "Hadoop allows us to store data that we never stored before," he said.

The data is aggregated into a common platform for use in a range of customer-focused data mining and data analytics tools, Feinsmith said.

Meanwhile, eBay is using Hadoop technology and the Hbase database, which supports real-time analysis of Hadoop data, to build a new search engine for its auction site.

Hugh Williams, vice president of experience, search and platforms at eBay, said the new engine, code-named Cassini, will replace technology the company has used since the early 2000s. The update is needed in part to handle surging volumes of data.

He noted that eBay has more than 97 million active buyers and sellers and over 200 million items for sale in 50,000 categories. The site handles close to 2 billion page views, 250 million search queries and tens of billions of database calls daily, he added.

The company has 9 petabytes of data stored on Hadoop and Teradata clusters, and the amount is growing quickly, he said.

Williams said about 100 eBay engineers are working on the Cassini project, making it one of the company's largest development efforts.

The new engine, slated to go live next year, is expected to respond to user queries with results that are context-based and more accurate than those provided by the current system, he said.

Feinsmith warned that IT shops interested in Hadoop should be aware of potential security issues . And he explained that aggregating and storing data from multiple sources can create a slew of problems related to access control and data management, while raising questions about data entitlement and data ownership.

Feinsmith also listed other potential Hadoop drawbacks that users should be aware of before embarking on big projects.

For instance, he said the Hadoop marketplace is "very confusing," featuring an oft-changing slate of vendors, products and standards. In addition, skilled Hadoop engineers are scarce .

And Williams noted that related technologies, such as Hbase, are still somewhat immature, which raises questions about system stability.

But Hadoop has plenty of potential. Feinsmith said that IT workers at JPMorgan Chase are debating whether relational database technologies will evolve to meet the bank's emerging big data needs, or if Hadoop-based systems will become adept at transaction processing.

This version of this story was originally published in Computerworld's print edition. It was adapted from an article that appeared earlier on Computerworld.com.

Read more about data center in Computerworld's Data Center Topic Center.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

More about: eBay, Morgan, Teradata, Topic
References show all

Comments

Post new comment

The content of this field is kept private and will not be shown publicly.
Users posting comments agree to the CIO comments policy.
Login or register to link comments to your user profile, or you may also post a comment without being logged in.
Related Coverage
Related Whitepapers
Latest Stories
Community Comments
Tags: CIO role, Configuration / maintenance, Data Center, ebay, hardware systems, IT Leadership, it management, JPMorgan Chase
Latest Blog Posts
Whitepapers
  • Why Hackers have Turned to Malicious JavaScript Attacks
    Website attacks have become a serious business proposition. In the past, hackers may have infected websites to gain notoriety or just to prove they could—but today, it’s all about the money. Reaching unsuspecting users through the web is easy and effective. Hackers now use sophisticated techniques—like injecting inline JavaScript—to spread malware through the web. Learn about the threat of malicious JavaScript attacks, and how they work. Understand how cybercriminals make money with these types of attacks and why IT managers should be vigilant.
    Learn more »
  • 10 Essential Steps to Web Security
    This short guide outlines 10 simple steps to best practice in web security. Follow them all to step up your organisation’s information security and stay ahead of your competitors. But remember that the target never stands still. Focus on the principles behind the steps – policy, vigilance, simplification, automation and transparency – to keep your information security bang up to date.
    Learn more »
  • Closing the print security gap - The market landscape for print security
    Today, many organisations continue to rely on printing to support business processes, particularly in the public sector, finance industry and legal profession. Whilst MFPs and printers have improved business productivity, they pose the same security risk as any networked device if left unprotected. With reported data breaches on the rise and growing industry and regulatory requirements around information security, businesses may suffer financial and reputational damage if they ignore the risks of unsecured printing. Read more.
    Learn more »
All whitepapers
rhs_login_lockGet exclusive access to Invitation only events CIO, reports & analysis.
Recent comments