Security and Google Apps

How does Google deal with patching then?

One of the reasons patching exists and takes so long is that people have very heterogeneous enviroments. We are, according to Gartner, the fourth largest server manufacturer in the world. It is kind of amazing – because we don’t really sell servers. We’re only producing them for ourselves, but we’re fourth in the market after IBM, HP and Dell. We designed our own chips, we wrote our own operating system, we completely created our own infrastructure on a heavily, modified Linux stack. One of the advantages of this is that everything looks the same. It makes it really easy to manage the infrastructure so that when it is time to update we can do it in a rapid, uniform fashion.

Now if you are going to put all your eggs in one basket, you need to guard that basket really, really well. That’s what having a single, custom-built, hardened solution allows you to do. Because you now know everything about that system as opposed to having to know about 10 different systems.

How do keep that homogeny given the sheer numbers of servers involved?

It’s tough. It’s part of our core strengths – for example, another technology that we have written and used is something that sits on every server and asks: ‘Do I look like the Google Standard Gold Image?’ And if it doesn’t it tries to correct itself. And if it can’t, it sends an alert to somebody. It turns out to have not only management advantages, but also security advantages, because one of the first things hackers tend to do when they get access to a machine is put a rootkit on which changes how that machine is going to look. As long as you have a Google Gold Image – and that changes with time – everything else knows to phone home and look like that image; it’s a very manageable problem.

Any plans to take those servers and technology to the market?

I can’t comment on any future plans, but you can see a lot of lessons we have learned from managing this infrastructure in moving to things like Chrome OS and having a very different model to Chrome OS. It’s a definite maybe. We typically try not to comment about any future release stuff.

So what storage technologies do you use?

We really just use consumer-grade hard drives…and just lots of them. The same things you have in your PCs at home. It could become an operational nightmare but we turn that into an advantage in how we manage those disks and the life of the disk. I know in any given point of time where a disk is – they each have a serial number – who put it there, what was on it when it was decommissioned, when it went back to inventory, when it was deleted. I have not met too many organisations that have any idea where their disks are, let alone given them serial numbers. When a disk goes bad, most organisations have an outside vendor come in and replace it. Now when I talk to CIOs their eyes light up and they ask, ‘How did I miss this?’

Do you use tape technologies at all?

We do use tape for some things like Gmail. We do tape backups on top of the multiple copies we talked about online. Some products like Gmail take a snapshot of everything every so often and go offline. If there was a huge catastrophic failure, and we had to recover everybody’s data, we have that ability.

You can imagine the Hollywood-type of scene where a group of Google engineers collude to delete everybody’s Gmail unless they’re given $3 million. Well, we have backup tapes in an offsite location that these people don’t have access to.

It sounds as if you’ve ‘War Gamed’ just that scenario...

We play a lot of games here. Part of our disaster recovery plan is to assume the worst has happened. Last year’s scenario was Google got attacked by aliens and California’s off the map. What do we do? How do we run our infrastructure?

From a network security perspective, we build our own Google front ends which are custom-built firewalls that use the same concepts – a homogeneous environment where we can learn a lot of lessons.

I have been in the security field for about 20 years and in my mind the measure of a good security organisation is how it reacts to an incident. People don’t like to talk about – we never want to think about getting into a car accident – but the reality is security incidents happen for various reasons. They may not even be your fault, but how do you react to that? Having a 24/7 security team is part of that and having our major security operations in California and Zurich so we can work through time zones. When there is a security incident, we assign an incident coordinator whose job is to triage that incident. And I think a big misnomer about this is if there is a security incident that affects customer data, we believe and contractually commit that it is our responsibility to notify those customers. There’s an idea that if something happens to your data, you won’t know. For sure – we will tell you.

Can you give us an example?

Docs oversharing. It affected 0.01 per cent of documents and we notified the people affected with as much information as possible – who, what documents, when – and let them make their decisions.

It’s important to put it in perspective. We make headlines because we are Google but the reality is worse stuff is happening in the traditional environment every day. Is Cloud computing perfect security? No. It’s not. I’ll be the first one to say that. I was in an intelligence community where we proved we could find out information about a computer that was not connected to a network and was in a secure room using various technologies. But I think Cloud computing is as secure, if not more secure, than what most organisations are doing today.

What kind of verification does Google undertake in relation to security?

Penetration tests and various audit reports – SAS 70 is the new one, SSAE16 and the new international version of that, the ISAE 3402. And we’re glad to give those reports to customers so they can see what the auditor is looking at regarding confidentiality, integrity and availability of data.

I think also one of the great things of the Cloud is the ability to innovate. About a year ago we gathered all the top professionals in the company and we asked: If we were going to make one change that would have the biggest impact across the board, what would it be? And the reality is most customers still get compromised because of a password. It’s something that security professionals don’t like to admit; we know all this wonderful cryptography and all these great systems we pay millions of dollars for and it comes down to a password.

We knew that if we could fix that problem, it would address a lot. And what we came up with was a two-step verification system. It’s free for enterprise and for consumers and it’s a one-time token on top of my password. I have to enter a six digit number that changes on a regular basis so if somebody steals my password, it’s not enough. They need a number that is generated on my smartphone and that number will change. You can also have it sent to you as an SMS or as an automated number and we have now released that in 150 different countries and 40 languages. It’s easy to use and it really improves security at the customer level.

What’s the take up like at the enterprise level?

Some enterprises are using their own system, so we support SAML [Security Assertion Markup Language] for single sign-on also. It’s got good takeup, I’d like to see more.

Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

• Bookmark this page
• Share this article
  •  
  • facebook
  • slashdot
  • digg
  • Reddit
  • stumbleupon
  • linkedin
  • twitter
• Got more on this story? Email CIO
• Follow CIO on twitter

• Share
  Share this article1

  •  google+
  • facebookfacebook
  • slashdotslashdot
  • diggdigg
  • RedditReddit
  • stumbleuponstumbleupon
  • linkedinlinkedin
  • twittertwitter
• print
• email
• Bookmark