Open source deduplication software released for Linux
- 25 March, 2010 07:28
- Comments 2
A new open source project, dubbed Opendedup, has appeared with the goal of creating a deduplication-based file system for Linux called SDFS.
The project’s developer Sam Silverberg says today’s deduplication solutions only solve the problem of storing deduplicated data, not reading and writing inline data.
SDFS is designed to support the needs of virtual environments including the VMware, Xen, and KVM hypervisors.
The filesystem can deduplicate inline (at a line speed of 150Mbps or greater) or periodically based on needs and this can be changed on the fly. Support for file or folder level snapshots is also a feature.
With support for deduplication at 4K block sizes, virtual machines data can be deduplicated and stored locally, across multiple nodes or in the cloud. It supports some 3TB of storage per gigabyte of memory.
A design goal was a distributed architecture and SDFS is scalable to eight petabytes of capacity with 256 storage engines, which can each store up to 32TB of deduplicated data. Each volume can be up to 8 exabytes and the number of files is limited by underlying file system.
The requirements for Opendedup are a 64-bit Linux distribution (it’s tested and developed on Ubuntu), Fuse 2.8 or greater, 2 GB of memory and Java 7.
Silverberg designed Opendedup to run in user space and be object-based because it would be platform independent, have a faster development cycle, easier to scale and cluster and to provide flexibility for integrating with other user space services like Amazon S3.
There is also the opportunity to leverage file system technologies like replication and snapshotting.
The latest release of SDFS, version 0.8.8 adds better I/O performance, scheduling of filesystem tasks, and a fix for a data corruption issue when removing unused deduplicated chunks.
The maximum file size it currently limited to 250GB with 4K chunk size.
Opendedup’s architecture consists of a SDFS Volume (one deduplicated file engine and one Fuse-based file system); a dedup file engine (manages file-level activities); a Fuse-ased file system; and a dedup storage engine which is the server-side service that stores and retrieves chunks of deduplicated data.
SDFS is licenced under the GPLv2. Windows support and block level replication are on the Opendedup roadmap.
Opendedup is online at Opendedup.org and on the Google Code portal at: http://code.google.com/p/opendedup/.
Recommended
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.
- Bookmark this page
- Share this article
- Got more on this story? Email CIO
- Follow CIO on twitter
-
Why change management doesn’t work
-
Larry Page wants to see your medical records
-
Dual-Persona Smartphones Not a BYOD Panacea
-
After two-year hiatus, EFF accepts bitcoin donations again
-
CIOs struggle to deliver timely mobile business apps: survey
-
The Power of Cloud
Although cloud is widely recognized as a technology game changer, its potential for driving business innovation remains virtually untapped. To take advantage of cloud’s potential to transform internal operations, customer relationships and industry value chains, organisations need to determine how best to employ cloud-enabled business models that promote sustainable competitive advantage. Learn more about driving business model innovation. -
Endpoint Security and Virtualisation
Besides form factor, virtual systems are not really that different than physical systems. They both use the same operating systems and applications. They both present users with computing resources such as RAM and hard drives. Consequently, the ability to exploit vulnerabilities in a physical environment will present a significant threat to virtualised environments as well. This paper examines the different endpoint security methods for virtualised environments and presents how Endpoint Protection security provides optimal performance, protection and manageability. -
Implementing A Security Analytics Architecture
According to the 2012 Verizon Data Breach Investigations report, 99% of breaches led to data compromise within “days” or less, whereas 85% of breaches took “weeks” or more to discover. This presents a significant challenge to security teams as it grants attackers extended periods of time within a victim’s environment. More “free time” leads to more stolen data and more digital damage. Principally, this is because today’s security measures aren’t designed to counter today’s more advanced threats. Read on.
Get exclusive access to Invitation only events CIO, reports & analysis. 
Comments
Bob
1
Deduplication
This is all very interesting I'm sure. But it would be nice, when writing articles or blogs like this, if the writer would define the subject of the article for "the rest of us". What the hell is "deduplication"? Makes no sense to me.
What is it like? You copy a file and then you uncopy it? Or you copy a music CD and then you uncopy the music CD????
Dreadmaul
2
Be wary of the website. It has a trojan in it!