Reality ain’t what it used to be. Augmented reality, virtual reality and mixed reality tools are creating a lot of buzz — as well as a lot of potential headaches for IT departments in everything from networking and security to data retrieval.
Even understanding what each of the terms refers to can be confusing, as they aren’t always used consistently. The term “mixed reality” (MR) was coined by researchers Paul Milgram and Fumio Kishino in 1994 to refer to every sensory environment in between the purely physical and the purely virtual. But over time, it fell out of use in favor of the more specific “augmented reality” (AR) for digital information overlaid on the real, physical world and “virtual reality” (VR) for a completely digital environment.
Over the past few years, Microsoft has revived the term “mixed reality” to market its HoloLens product, and vendors now tend to use the term to refer to the projection of 3D virtual objects or holograms into physical space. With AR, the digital overlay usually appears to float on a plane in front of the observer, while with MR it’s possible to walk around a virtual object to see it from all sides.
One other difference: While today’s VR and holographic MR applications typically require specialized hardware such as glasses or headsets, AR is already being built into apps for ordinary smartphones and tablets, as evidenced by last year’s Pokémon Go game craze. Pokémon Go laid a social as well as technological foundation for the adoption of AR, says Tuong Nguyen, a principal research analyst with Gartner. “It made people familiar with the idea that if I hold up my phone, I expect to see some type of digital representation overlaid on the physical world. It caused a behavioral change even if you didn’t play the game.”
Whatever flavor of reality you prefer, the technologies hold a lot of promise for businesses. “We already have enterprises that are investing in AR,” says Mark Sage, executive director of the Augmented Reality for Enterprise Alliance, a nonprofit trade organization promoting the adoption of AR. Companies typically deploy the technology for specific use cases or to solve particular business problems, Sage says. “It’s all about improving performance and creating efficiencies.”
“When you think both AR and VR, think of them as user interface technologies,” says Nguyen. “There’s a handful of horizontal applications for AR, like task itemization and pick/pack lists. I put on my [AR] glasses, and it says, ‘Hey Tuong, you’ve got to go down to Aisle 13, Bay 4 and pick this off the shelf’ and maybe overlay a little map.” And both AR and VR could prove useful for design and collaboration, he says, enabling building engineers and construction workers to visualize how structures are going to look in place.
“The field service world is another one,” adds Michael Fauscette, chief research officer at B2B reviews platform G2 Crowd. “Say you service a solar farm and you have to make a repair. If you have AR capability, you could see the work order, see the process, even see an overlay of the repair — there are all sorts of opportunities to put that data into your field of vision that makes it so much faster and easier to do your work.”
Challenges for IT
With great opportunities come great responsibility, as Uncle Ben might have said if he’d worked in IT. “Like anything else, when you adopt a new technology, it puts a strain on your infrastructure,” warns Chad Holmes, principal and cyber CTO at professional services organization EY. “You have to uplift your infrastructure to support a new communication path or channel.”
Unfortunately, Holmes continues, many companies don’t have the back-office capabilities required to support AR, MR and VR. “The technology is there, but a lot of improvements need to be made to old legacy systems to make sure that these types of technologies can function how they’re supposed to,” he says. “For most, it’s a big investment.”
The other big concern cited by Holmes and others is the security issue raised by the addition of new remote devices sharing data wirelessly. “So much is new and untested when it comes to AR and VR,” Holmes says. “We don’t yet know all the ways in which these solutions can be vulnerable. You’ve got to think about what they’re going to be used for — what they’re actually transmitting and looking at.”
Nguyen agrees. “We have a head-mounted display, it’s a portable device, it probably will have a camera on it, and the camera might be on all the time,” he says, recalling that when phones first started getting cameras, IT departments were going crazy.
The constant flow of information also raises privacy issues. “I have this device on, so my employer can track and know that I took 25 minutes on a maintenance task that usually only takes 15 or that I wandered off someplace,” he says. “These are things IT departments have to consider.”
AR, VR and MR deployments may seem a long way off, but it won’t be long before most enterprises will need to be ready. While consumer AR apps like Pokémon Go may get all the buzz, Nguyen says, “What I’m seeing is that business-to-business AR is ahead of business-to-consumer AR, whereas on the VR side it’s flipped.” He estimates that enterprise use of AR technology is two or three years ahead of the consumer side. “I expect the maturing and the traction within the enterprise to continue to outpace the consumer side, at least in the next three or so years,” he adds.
Fauscette agrees. “Over the next two to three years, you’re going to see AR go mainstream around some specific tasks like field service, logistics, warehouse, repairs and design,” he says. “VR lags that by 25% or so.”
And with Apple’s June announcement of its ARKit developer framework for easily creating AR apps, it’s clear that the company envisions AR having a significant presence on millions of iPhone and iPad users after it releases iOS 11 this fall. Initially, most of those apps will be consumer-facing, but they’ll be entering the workplace on employees’ mobile devices whether IT is ready or not. And, of course, ARKit opens up even more opportunities for companies to create worker-facing AR apps.
Corporations such as Boeing and Bosch are already running pilot programs with AR, VR and MR, as are educational institutions such as Case Western Reserve University. We spoke to these three organizations about the benefits and the challenges of mixing realities, and sought advice for IT departments preparing for the coming influx of apps and devices based on these technologies.
Boeing: Assembling satellites with an AR assist
Aviation giant Boeing has been experimenting with augmented reality for nearly 30 years. In the late 1980s, researchers were using headsets that could show a 2D display of a wiring diagram in a complex wiring harness. A more recent effort began about six years ago, starting in Boeing’s satellite factory in El Segundo, Calif.
“We’ve mainly been doing pilots around AR to figure out how we would implement these things on a larger scale,” explains Boeing senior technical fellow Paul Davies. “That involves, ‘What is the concept of operations? Do workers hold a tablet or do we try a wearable?’ We’re also trying to figure out how to connect these systems to our back end. The big goal, at least for my group, is reducing the cycle time — the build time — and then improving the quality.”
Boeing has built its own AR tool based on a commercially available SDK, adding a custom interface. With it, the technicians who assemble Boeing products (satellites, in this case) can receive technical information about the product as they work. “In some cases it could be step-by-step guidance on how to do a job,” says Davies. “Or maybe they’re seeing a model of a part so they can understand which way to orient it on a satellite panel.”
Davies explains how helpful the latter would be when assembling waveguides in a satellite panel. “They look just like bent pieces of tubing,” he says, “and it can take a while to figure out which way it goes and how it mates. If we use AR to show that in 3D and it’s basically registered and overlaid on the panel, it becomes really quick and easy to understand how to install it.”
The 3D model comes from the CAD files created by the programs used to design the products. “We would just export it or convert it into the right format,” Davies says. “Sometimes we already have digital 2D tech instructions that are showing the 3D model, so we could bring those into the AR scene as well.”
For the company’s IT department, one of the biggest concerns is with putting the new devices on the network. “We have very stringent security requirements for what we allow on our network,” Davies says, “so when new devices come out from vendors — headsets or whatever — if it’s not running our enterprise load of Windows, it can’t go on the network. Even if it is, it’ll probably still have to go through a review.” To that end, the IT department is adding staff familiar with mobile device management and wearables.
Davies emphasizes that Boeing is still in the pilot phase. “We haven’t done the ROI, and we don’t claim to have saved any time or money or anything like that,” he says. “The purpose was for us to learn and figure out how we’re going to implement these things on a larger scale. We’ve done analytical studies and analysis of user groups, and we know what we think we can expect in terms of efficiency improvements.”
Case Western Reserve University: Studying virtual bodies
Sue Workman, CIO at Case Western Reserve University (CWRU), was not impressed with her initial experiences with virtual reality. “You put these glasses on, and the data comes to both of your eyes working separately rather than together. It was not very realistic, and it made you feel dizzy or even a little bit sick,” she recalls.
When Microsoft invited executives from Cleveland Clinic and CWRU to a demo of its HoloLens “mixed reality” devices in 2014, she expected more of the same. Instead, she came away excited for what HoloLens could do for her school’s curriculum.
“We’re building a health education campus with Cleveland Clinic,” she explains, a 485,000-square-foot building scheduled to open in 2019. “It will house our schools of medicine, nursing and dental medicine, including Cleveland Clinic’s Lerner College of Medicine.” The school didn’t want to include cadaver labs in the new building, so Workman had been looking for ways to teach anatomy without cadavers. The HoloLens MR technology projects 3D anatomic holograms into the room, where students can see them from all angles and interact with them, as shown in the video at the top of this article.
“We looked at several other technologies,” Workman says. “For instance, one option we examined was a table monitor that’s horizontal instead of vertical and displays 3D anatomy. But it’s still 3D on a 2D screen. [HoloLens] technology has been the one that’s really stood out to us and seemed likely to make the most sense for our anatomy faculty.”
So far, Workman says, supporting the HoloLens doesn’t look like it’s going to be a big challenge to the school’s IT resources. “The headset is a Windows 10 computer, and you use Unity to develop the applications,” she explains. You do need some storage capacity when you’re developing, until you can compress the data to actually send it to the HoloLens. “We’re currently using Microsoft Azure to serve our devices,” Workman says, “but it really doesn’t matter — you could use AWS.”
That’s not to say that deploying several dozen headsets (so far) has presented no challenges. “The infrastructure is where you have to concentrate,” Workman says. “The biggest impact has been with Wi-Fi, and we’re working through those issues now.”
What she’s finding is that the devices might connect to the first wireless access point they encounter and stay connected, even when transferring to another access point might be more efficient. “If there are thirty people in the room, all thirty might connect to the same WAP,” she says, “or there might be five WAPs in there, but only two of them are really serving.” But, she’s quick to point out, this isn’t a problem unique to or new with the HoloLens — “it also happens in libraries and residence halls with students’ cell phones.”
So far, security hasn’t been an issue because the tests haven’t used real patient data or anything identifiable. “Someday, in a clinical situation, there will absolutely be privacy and security concerns,” Workman acknowledges, adding that CWRU is working through security and compliance issues related to HoloLens now. But again, she says, it’s no different than dealing with those issues on cell phones. “You’ve got to have best practices in place, you‘ve got to have guidelines, you’ve got to have storage that’s secure and networks you can trust. It’s no different than with any other device.”
Bosch: Working toward a unified AR interface
Bosch Auto Parts, which produces everything from brakes to batteries, is both a user and a supplier of AR systems, according to Juergen Lumera, director of global technical information systems product management and Innovation at Bosch Automotive Service Solutions. Lumera works with both internal and external customers, helping their service and IT departments understand what they need to be aware of when implementing an AR product. “We do the majority of our applications — service information, training, and end-of-line production solutions — for automotive enterprises with twenty to thirty thousand users,” he says.
An example is the first-responder support developed for Daimler vehicles. When a vehicle is involved in an accident, rescue personnel may have to cut into the vehicle to extract passengers, and they need to know which parts of the car are dangerous to cut into. Bosch developed an app for smartphones and tablets that displays the relevant parts, superimposed on the device’s camera view of the vehicle, to highlight these locations.
But Bosch’s vision for AR extends beyond information displayed on a smartphone or tablet. “For us, AR will be the future interface,” Lumera says. “In the workshop, a technician will have only their AR glasses and will get all the information they need through that. You have a lot of diagnostic systems connected while you’re repairing a car — why would I want somebody to leave the car to find out what to do? You'll find equivalent problems in any industry, where a technician is forced to interact with multiple devices. Our goal is to totally eliminate that.”
To get there, Lumera warns, IT departments need to solve a lot of problems. Preparing the AR content for Bosch’s systems requires access to an enterprise system’s CAD data showing its products, but that data isn’t usually set up for easy retrieval. “When we do proofs of concept,” Lumera says, “the first thing we find is that the data is not structured so that you can easily extract what you need for a given situation. So IT departments need to make it simpler to fetch the right kind of CAD data.”
The next problem is how to securely transport the data and ensure that each user gets only the data they need. “I really need to ensure that the 3D data has enough information to represent variants, and that I only see the variants relevant to me,” Lumera continues.
He also points to security and privacy issues and the ways AR glasses differ from cell phones. An AR headset has a camera so it knows what data to display, but unlike with a phone, a user might accidentally look in any direction, not just at the product they’re working on.
As at Boeing and CWRU, network issues round out the list of challenges. “Ask an IT manager to get a HoloLens on the official network,” Lumera says. “I’m pretty sure that 95% have no idea how to do it. Whether Microsoft can technically do it is not the problem. It’s more about how I deal with the rights, how I deal with the camera, and what user profiles I get. This is not an easy task.”
Ready, set, go
What steps should enterprise IT take now to be ready for these technologies?
Recruit staff with the right expertise. “There are some real holes when companies start to think about both AR and VR,” says G2 Crowd’s Fauscette. “One of them is the skill set and talent you need. A lot of AR and VR apps are custom add-ins or extensions. The standard skill sets that most IT organizations have aren’t going to be the ones they need to do that.”
For developing such tools, he suggests thinking about recruiting staff from the gaming industry who understand the technology and the user experience. On the implementation side, take a cue from Boeing and look for workers who have mobile device management expertise.
Get help from the experts. One helpful resource is the Augmented Reality for Enterprise Alliance (AREA), launched in 2015 to help organizations make decisions about these new technologies. (Bosch and Boeing are both members.) The organization has recently developed a set of global enterprise AR hardware and software requirements in conjunction with UI Labs, Lockheed Martin, Caterpillar and Procter & Gamble.
Because of the customization necessary, Fauscette and Gartner analyst Nguyen emphasize the value of partnerships. “There’s still a lot of specialized work and integration that needs to be done,” says Nguyen. “You have to look for different partners who can help with that. Ideally, you should be able to leverage a lot of the resources you already have.”
Accenture is already building a practice around these technologies, says Fauscette, adding that other large systems integrators are probably doing the same or soon will. “The general questions of, ‘I need to build this, integrate it, and build the content behind it’ could easily be the kind of help you get from a systems integration partner,” he says.
Begin a pilot project. AREA’s Sage says enterprises should set up an incubator-style project or a “skunkworks” to work on testing and getting ready. Conducting small test programs can help companies ensure that source data is accessible and properly structured; identify trouble spots in networking, storage and security infrastructure; work through systems integration issues; and develop best practices around data privacy and device management.
Whether you line up partners, start talking to integrators, set up a skunkworks, recruit staff with the right expertise — or all of the above — it’s not too late to get started. Remember, it was only about three to five years ago that BYOD was regarded as a new trend. For IT departments, reality in any form comes at you fast.
Join the CIO Australia group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.