Open Source 'Lingual' Helps SQL Devs Unlock Hadoop
- 22 February, 2013 20:15
- Comments
One commonly cited stumbling block to the broader adoption of Apache Hadoop in the enterprise is the difficulty and expense of finding and hiring developers who understand and can think in Hadoop MapReduce. But Big Data application framework specialist Concurrent wants to change that.
Concurrent is already the driving force behind Cascading, a stand-alone open source Java application framework designed by Concurrent founder and CTO Chris Wensel as an alternative API to MapReduce.
Cascading gives Java developers the capability to build Big Data applications on Hadoop using their existing skillset. Now Concurrent hopes to help SQL users get into the act with Lingual, an open source ANSI-standard SQL engine that runs on top of Cascading.
Lingual Lets Analysts, Developers Tap SQL Skills for Hadoop
Lingual, which will be publicly available under the Apache 2.0 license within the next few weeks, gives analysts and developers familiar with SQL, JDBC and traditional BI tools the caoability to create and run Big Data applications on Hadoop using their existing skillsets.
"Concurrent was established with the belief that there had to be a simpler path to mass Hadoop adoption," Wensel says. "And since day one, we have worked to create solutions that make it easier for developers to build powerful and robust Big Data applications quickly and easily. With the Lingual project, we are one huge step closer to realizing our mission."
To date, many Hadoop users have turned to Apache Hive (a data warehouse infrastructure built for Hadoop) and Apache Pig (a high-level platform for creating MapReduce programs) to achieve SQL-like capabilities.
"Pig and Hive have their own qualities and actually are quite good, but sometimes you just want SQL," Wensel says. "Lingual is great for people who don't know how to use Hadoop but know SQL. The best way to get value out of something in many cases is just to use SQL."
"We just want to make it easier for people to get data off Hadoop or to port their apps to Hadoop using skills they already know," he adds.
Use Cases for Open Source Lingual SQL Parser
Example use cases for Lingual include the following:
- Giving data analysts, scientists and developers the capability to "cut and paste" existing ANSI SQL code from traditional data warehouses and instantly access data locked on a Hadoop cluster
- Giving developers the capability to use a standard Java JDBC interface to create new Hadoop applications or use any of the Cascading APIs and languages, like Scalding and Cascalog
- Giving companies the capability to query and export data from Hadoop directly into traditional BI tools
"We are very excited about the prospect of using standard SQL to provide seamless access to the billions of events that we track daily," says Zack Shapiro, director of engineering at Kontagent, a Concurrent customer.
"Rather than filtering through events and exporting them to MySQL, our customer support staff and data scientists will finally be able to work with tools they already know to query the raw data directly within our Hadoop cluster through the use of Lingual and Cascading," Shapiro says.
Cascading has already been adopted in some of the biggest and most well-known Big Data companies, like eBay, Etsy and Twitter. Twitter uses Cascading to streamline its data processing, data filtering and workflow optimization for large volumes of unstructured and semi-structured data. It is also the driving force behind three popular open source language extensions: PyCascading (Python + Cascading), Scalding (Scala + Cascading) and Cascalog (Clojure + Cascading).
"eBay has picked that up and is running it as well," Wensel notes. "All of eBay's search is now running on Scalding."
Thor Olavsrud covers IT Security, Big Data, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline, Facebook, Google + and LinkedIn. Email Thor at tolavsrud@cio.com
Recommended
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.
- Bookmark this page
- Share this article
- Got more on this story? Email CIO
- Follow CIO on twitter
-
Why change management doesn’t work
-
Larry Page wants to see your medical records
-
Dual-Persona Smartphones Not a BYOD Panacea
-
After two-year hiatus, EFF accepts bitcoin donations again
-
CIOs struggle to deliver timely mobile business apps: survey
-
Securing the Road to Virtualization and Beyond
Traditional security controls for enterprise don’t necessarily translate into the new world of virtualisation and cloud environments. When mapping out a secure virtualisation roadmap, click to find out about pave a more secure, risk free path. -
Deploying Flash in the Enterprise
Flash is quickly emerging as the preferred way to overcome the nagging performance limitations of hard disk drives. However, because flash comes at a significant price premium, outright replacement of HDDs with flash only makes sense in situations in which capacity requirements are relatively small and performance requirements are high. Learn how deployment approaches-including hybrid storage arrays, server flash, and all-flash arrays-that combine the performance of flash with the capacity of HDDs can be cost effective for a broad range of performance requirements. -
New Demands for Real-time Threat Management
Many organisations are evaluating a new security model based upon IT risk management best practices. This is a good idea, but not enough for today’s dynamic and malevolent threat landscape. To keep up with IT changes and external threats, large organisations need to embrace two new security practices: real-time risk management for day-to-day security adjustments and real-time threat management to detect and remediate sophisticated, stealthy, and damaging security breaches (i.e., advanced persistent threats, or APTs). Learn more.
Get exclusive access to Invitation only events CIO, reports & analysis. 
