How Netflix has the cloud do the heavy lifting for video transcoding
- 04 December, 2012 15:24
For Netflix vice-president of digital supply chain Kevin McEntee, the US-based video streaming company's shift to using the cloud for transcoding its massive content library comes down to a modern take on the fable of the tortoise and the hare. Or, as he told the audience at the AWS re:Invent conference in Las Vegas last week, it's like a choice between moving a room full of people to another city by using expensive high-performance Ferraris or a fleet of somewhat more humble Toyota Priuses.
When it began its shift away from DVD rentals to Internet-based video streaming, Netflix initially employed a 'Ferrari' approach to dealing with the computationally intensive task of encoding movies and TV shows in a format that could be streamed to client devices.
Amazon CEO Jeff Bezos on innovation and entrepreneurship
Werner Vogels: The ‘four commandments’ for 21st century application architecture
Netflix CEO: Cloud currently akin to pre-compiler era
Amazon cloud chief: Two Availability Zones just the beginning
In 2006/2007 when Netflix began the move to streaming, it found that the video processing technology typically employed in Hollywood centred on ensuring minimal latency: It was optimised for scenarios such as a single video editor mastering a Blu-ray image of a movie: "[It was] optimised for the expensive time of that one operator; essentially the artist sitting there doing that mastering."
"Back in 2006/2007 we hired people out of this industry and we ended up building out a data centre that [was] very Ferrari-like," McEntee said, with custom, GPU-based encoding hardware; "boxes that had custom GPUs that were built specifically to dump video very fast." It was expensive and constrained by the fixed footprint of Netflix's data centre.
However the limitations of this approach became evident in late 2008, when Netflix set out to launch new video players for PCs and Macs, and jumped onto TVs by launching a player on the Xbox in November of that year.
"There was such an amazing lack of standardisation around video streaming those days, and there still is today, that we had to create new formats for those players," McEntee said.
"And at the same time [as] we were innovating in the player space and therefore causing the need for new formats, our content team in LA was licensing more and more content, so our content library during the course of that project also doubled in size. And so we set out re-encode using the hardware farm that we had built."
Unfortunately for Netflix, the hardware didn't deal well with the load, and the company encountered frequent hardware failures. Fans on the custom GPUs being used were too small and "boxes were melting", McEntee said.
"It was really a very frustrating experience and in fact that catalogue re-encode was late and we failed. Basically we launched these players and the catalogue was not complete."
It was reflecting on this experience that caused Netflix to make the move to the Amazon Web Services cloud for transcoding. "If you jump forward a year, we had made the jump to move our transcoding farm into AWS. And we had seen the opportunity in fall 2009 for launching a video player on the [Sony PlayStation 3] so this was our first 100 per cent AWS transcode. "
"The player developers again realised they had to rely on a new format; they had to transcode the entire library," McEntee said. The new format was not finalised until three or four weeks before the launch of the new player, but Netflix was "able to spin up enough instances in EC2 to transcode the entire library in about three weeks" and managed to meet the deadline.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.
- Amazon CEO Jeff Bezos on innovation and entrepreneurship
- Werner Vogels: The ‘four commandments’ for 21st century application architecture
- Netflix CEO: Cloud currently akin to pre-compiler era
- Amazon cloud chief: Two Availability Zones just the beginning
- Techworld Australia
- Computerworld Australia
Ruggedized scientific calculator perfect for extreme math
How to Switch From iPhone 5S to BlackBerry Z30 (and Why)
How to Switch From iPhone 5S to BlackBerry Z30 (and Why)
CIOs to Become In-House Brokers -- and That's a Good Thing
The future of computing
IDC MarketScape Excerpt: Worldwide Client Virtualization Software Assessment
The rise of BYOD is creating governance and regulatory nightmares while providing end users with unprecedented flexibility and agility. While IT is still intrigued by the possibility of a better desktop management model and the operational savings client virtualization software could deliver, it is the increased governance and the ability to deliver desktops, applications, and data to any device that are driving today's purchases.
Why you should be re-thinking your approach to data protection
Organisations of all shapes and sizes need a new approach to data protection that addresses the challenges of data growth, but IT budgets are not keeping pace with the escalating costs of supporting storage requirements. This whitepaper explores how securing and retrieving organisational data will need to be done more efficiently.
Guide: 10 Essential Questions about Mobile App Containers
Mobile apps are the ultimate corporate dilemma: They turbo-charge productivity—but also bring significant security risks. How do you navigate the challenge and emerge with both? With a comprehensive approach to security.Learn the key areas to focus on as you build a mobile app security strategy.