Augmented Reality in Manufacturing

Augmented Reality in Industry 4.0 Foundations

One of the great promises and at the same time one of the main focus areas in Industry 4.0 is the bridging of digital/cyber/virtual and physical worlds, hence the focus on cyber-physical systems.

Global spending in virtual and augmented reality is poised to double every year through 2025 (IDC)

Apart from the fact that this isn’t just a technology issue (nor is the Industry 4.0 vision as such), from the technological perspective one immediately thinks about the Internet of Things. However, virtual reality (VR) and augmented reality (AR) are certainly as important.

Virtual reality and augmented reality are used in several sectors and contexts, from consumer applications to manufacturers. Yet, it’s in manufacturing that augmented reality offers great value in myriad applications, in combination with several other technologies as per usual.

The use of VR and AR in manufacturing and other industries for which the term Industry 4.0 gets used is not fiction. It happens as we speak and is poised to accelerate as the benefits become increasingly clear, offerings, hardware and applications mature and move to the next level and manufacturers increase their digital transformation efforts on the strategic and staged path towards the realization of Industry 4.0 and the digital transformation of manufacturing.

Use Cases of Virtual and Augmented Reality in Industry 4.0

Virtual reality and augmented reality can play a role in the typical earlier stages where optimization and enhanced productivity (quantity, quality, speed, flexibility) are more important than later stages of innovation and genuine business transformation (which can of course be set out as Industry 4.0 goals at the start, more about that in ‘Finding the value in Industry 4.0‘).

Just think about how simulation models and the use of augmented reality can speed up the entire production chain, in combination with the right data, starting from the use of AR and VR in virtual design. Or about the use of augmented reality in maintenance. And then there is of course the possibility to put a virtual layer, based on the right data and information, on top of the ‘reality’ in all sorts of factory and industry environments, using devices such as AR/VR glasses/viewers. The latter is probably the best known illustration of how de facto virtual or cyber and physical meet.

By 2021, the majority of AR/VR spending will be for industrial maintenance

Use cases include machining and production, education and collaboration, factory planning, assembly, security, testing and digital prototyping, to name a few. As we’ll see showcasing and immersive (key in AR/VR) experiences on the customer side are important as well. So, marketers should also pay attention, certainly in the manufacturing of products where showcasing technological manufacturing expertise strengthens the perception of the technological wealth of both company and product. So, no, it is not a coincidence if many use cases of AR/VR in manufacturing which get a lot of attention are in, for example, the automotive industry (and certainly the luxury car brands).

In operations, you can certainly also imagine how, with the proper equipment and solutions (from the worker’s perspective as well of course, have you seen those first headsets?) service people, factory staff and logistics staff can better perform their tasks if they have the information they need in front of their eyes and two of their main work instruments, on top of their brain, free: the left hand and the right hand. The result: smoother processes and flows.

Augmented Reality in Industry 4.0: Follow the Value and Key Assets

AR and VR is far from omnipresent in all the mentioned and other use cases applications, even in high-tech industrial manufacturing.

In this stage virtual and augmented reality certainly aren’t mainstream and you most likely will find them in (product) design, in virtual training programs and in the simulation of important scenarios and tests regarding key assets in factories and beyond. In other words: where the stakes are high and the value/risks are equally high.

Process manufacturing training, assembly and safety are by the way among the main use cases of VR and AR. However, across all industry use cases it’s retail that takes the lead from a spending perspective with the ‘retail showcasing’ use case, as we’ll cover next.

Product design, virtual training and simulations/tests with a focus on important assets, scenarios and security aspects are key VR/AR applications in Industry 4.0.

As mentioned, showcasing also plays a role in manufacturing, among others in design and development and in customer-facing circumstances. In the end, someone needs to convince and sell. But this of course doesn’t mean that VR and AR are just sales tools or gizmos, not in the consumer industry and most certainly not in Industry 4.0. A token of the increasing role of augmented reality in Industrial IoT is the growing support for it in several Industrial IoT platforms.

Time to look at some facts and findings about the usage of virtual reality and augmented reality in manufacturing and beyond with research data, predictions and trends, before looking at benefits, solutions and a few practical applications and cases.

AR and VR Trends in Industry 4.0

The above mentioned main industry use cases of VR/AR for 2020 are part of some takeaways of IDC’s August 2019 forecasts on the worldwide spending on augmented and virtual reality.

In that forecast IDC predicts that global spend on AR and VR will double each year through 2021, which is pretty impressive. However, given the broad scope of VR and AR use cases across several industries this obviously doesn’t only fit in a context of manufacturing and Industry 4.0.

Process manufacturing and discrete manufacturing will take over consumer segment spending in VR/AR by 2021

In each of the regions IDC looked at, the consumer segment is poised to be the largest in 2019. However, in the US and Western Europe, discrete manufacturing and process manufacturing already rank second in third.

In the US, process manufacturing and discrete manufacturing are predicted to take over the consumer segment in the forecast period, along with government, retail, construction, transportation and professional services. In Western Europe, discrete manufacturing, retail and process manufacturing are predicted to start growing fast by the end of the forecast (until then the consumer segment remains largest).

If we look at the key use cases for VR/AR we also see evolutions with a strong role for industrial use cases. As mentioned previously, in 2019 the three major VR/AR use cases from an investment perspective are respectively:

• Retail showcasing, accounting for a total investment of $442 million.
• On-site assembly and safety, worth a total spend of $362 million.
• Process manufacturing training as the number three with $309 million.

By the end of the forecast, however, the majority of spending will go to industrial maintenance with $5.2 billion and public infrastructure maintenance with $3.6 billion. And that of course brings us close to one of the key aspects of the Industrial Internet of Things, Industry 4.0 and so forth: maintenance, preventive and predictive. In other research, which we covered in our article on Industry 4.0 trends, drivers and spending evolutions, preventive and predictive maintenance were mentioned as main priorities.

Again, the number of applications for VR/AR is very broad and manufacturing, transportation, logistics (Logistics 4.0) and other markets in today’s current 4.0 scope are far from the only ones. Moreover, in some regions AR and VR predominantly will keep seeing investments in, among others, the consumer segment and retail. Education is often mentioned as well. In the APeJ region it’s already the third most important ‘sector’ in 2019 according to the mentioned IDC research. The link with training in any given manufacturing context is quickly made.

Yet, it’s clear that the usage and types of applications with VR/AR in manufacturing and related industries increase with a current focus on training and safety, to name just two, and an increasing focus on discrete manufacturing, process manufacturing and maintenance in the US and Western Europe.

Make your business case for AR and VR

In a previous update (February, 2017) of its Worldwide Semiannual Augmented and Virtual Reality Spending Guide, IDC’s Tom Mainelli rightfully stated that “AR and VR headsets get most of the media attention right now, but the hardware is only as good as the software and services running on it”.

Truism. In an Industry 4.0 context we can add that the use cases which will thrive are those generating the highest value, enabling to avoid risks, issues and downtime the most, optimizing the end-to-end manufacturing process and workflows in the best possible way, making productivity, satisfaction and making experiences of field engineers, factory workers, customers and stakeholders the most satisfactory – and immersive in ways that make sense. And that goes beyond just the technological dimensions and requires an individual business case – as always.

The graphic from the announcement of that semi-annual update below shows the predictions of IDC per AR/VR industry from a spending perspective for 2019, showing the ‘place’ of process manufacturing and discrete manufacturing in the bigger picture and isn’t related to the forecasts mentioned for the next years.

Neuroscience and Augmented Reality in Manufacturing

Augmented Reality in Manufacturing, It’s All in the Neuroscience

Augmented reality (AR) refers to a class of tools and technologies that overlay virtual objects and information onto the user’s view of the physical world that become a part of the real-world. AR is taking the world by storm. From the explosion of interest in games like Pokemon Go in 2016, to the growing number of use cases we see today, with each passing day, new AR tools and capabilities are being introduced.

Although the term “enterprise-grade AR” is common, no objective and rigorous standards have been developed to define AR solutions along a continuum of excellence.

Although the term “enterprise-grade AR” is common, no objective and rigorous standards have been developed to define AR solutions along a continuum of excellence. This report represents a first step toward developing an objective, rigorous standard for defining enterprise-grade AR that is grounded in the neuroscience of learning and performance. Regardless of the application, it is the same brain that learns and drives performance. Thus, a deep understanding of “Your Brain on Augmented Reality” is the key to building the highest quality enterprise-grade AR solution.

This is an insightful quote from Albert Einstein that is supported by the neuroscience of learning and performance. But why? What is it about experience that is so rich that it is fundamental to learning, and why is information so much less effective?

The Neuroscience of Learning and Performance
“Learning is an experience. Everything else is just information” -Albert Einstein

The human brain is comprised of at least three distinct learning and performance systems (see Figure). As Einstein so eloquently stated, experience is at the heart of learning. Importantly, it is also at the foundation of enterprise-grade AR. AR provides augmented information co-located on the users’ current reality or experience.

The experiential system has evolved to represent the sensory aspects of an experience, whether visual, auditory, tactile or olfactory. Every experience is unique, adds rich context to learning and is immersive. The critical brain regions associated with experiential learning are the occipital lobes (sight), temporal lobes (sound), and parietal lobes (touch).

The cognitive system is the information system. It processes and stores knowledge and facts. Cognitive information comes in the form of text, graphics, sound, or video and is limited by the learner’s working memory and attention span. Critically, these are limited resources and often form the bottleneck that slows learning with more information coming in and available to the learner (the green arrows) than can be processed (the red arrow). This system encompasses the prefrontal cortex and hippocampus.

The cognitive system is the information system. It processes and stores knowledge and facts. Information comes in the form of text, graphics, sound, or video and is limited by the learner’s working memory and attention span.

The behavioral system in the brain has evolved to learn motor skills. The critical brain structure is the striatum. It is one thing to know what to do, but it is completely different (and mediated by different systems in the brain) to know how to do it. Processing in this system is optimized when behavior is interactive and is followed in real-time (literally within milliseconds) by corrective feedback. Behaviors that are rewarded lead to dopamine release that incrementally increases the likelihood of eliciting that behavior again in the same context. Behaviors that are punished do not lead to dopamine release thus incrementally decreasing the likelihood of eliciting that behavior again in the same context. Real-time feedback is critical because the brain’s response fades quickly (within a few 100 milliseconds), meaning that learning and unlearning will not occur.

A Neuroscience-Based Definition of Enterprise-Grade AR

Now that we understand the neuroscience of learning and performance, we can objectively and rigorously define the neuroscience-based enterprise-grade AR standard, and identify the AR capabilities needed to meet this standard.

An enterprise-grade AR tool must engage experiential, cognitive and behavioral learning and performance systems in the brain in synchrony.

• First, and foremost, an enterprise-grade AR tool must engage experiential, cognitive and behavioral learning and performance systems in the brain in synchrony. This can be achieved by engaging the experiential learning system with AR assets overlaid on the real-world that provide information to the cognitive learning system and real-time interactive feedback to the behavioral skills learning system. This builds a cognitive understanding while simultaneously developing a strong behavioral repertoire, all within a real-world setting.
• Second, the AR assets must be presented to the user in a way that minimizes the load on the cognitive system. Recall that the bottleneck in the cognitive system is working memory and attention that are both limited capacity resources. Regardless of whether the AR asset is text, graphics, sound, video, or holographic, it must be presented in a way that minimizes the cognitive load on the user by presenting the user with what they need, where they need it, when they need it. This is not a trivial problem to solve and the type of asset, as well as the timing and location of the asset will be determined by the use case, but the goal must always be to reduce the load on cognitive processing.
• Third, the AR solution must be interactive in the sense that AR assets must guide the users’ behavior, and react to the users’ behavior in real-time. AR assets that guide users’ behavior must be temporally linked to the users’ behavior. In other words, as a user completes a step, AR assets that guide the next step must be presented in real-time. In addition, these assets must react to the specific behavior elicited by the user. If a correct behavior is elicited by the user, then the next asset in the chain should be presented. On the other hand, if an incorrect behavior is elicited by the user, the AR solution must provide some form of real-time corrective feedback to the user. This is where a combination of AR and the Internet of Things (IoT) would be advantageous. Information from IoT will be shared with the AR solution in real-time that will drive the spatial and temporal presentation of AR assets. This real-time interactivity will effectively engage the behavioral learning system and quickly build a strong behavioral repertoire.

Any AR solution must be interactive in the sense that AR assets must guide the users’ behavior, and react to the users’ behavior in real-time.

Augmented Reality in Manufacturing – A Use Case

The Problem: A major threat to the manufacturing and industrial sector is the increasing workforce skills gap. Research from the Manufacturing Institute suggests that nearly 10 million manufacturing jobs will be needed in the next decade with millions of these going unfilled. To make matters worse, new workers will not have the expertise of seasoned veterans, leaving the manufacturing sector with a less productive, and potentially more error-prone, workforce.

Research from the Manufacturing Institute suggests that nearly 10 million manufacturing jobs will be needed in the next decade with millions of these going unfilled.

What is needed are methods for training the new workforce that quickly and efficiently instill the subject matter expertise and behavioral skills needed on the factory floor. For example, when training a new worker to operate and maintain a piece of equipment, the ideal approach is to train the vocabulary to describe and label each part while simultaneously training the behavioral repertoire needed to effectively operate and repair the equipment. This coordinated training approach reduces the cognitive load on the user while simultaneously providing the opportunity for limitless cognitive learning and behavioral practice. Taken together this speeds learning and retention, quickly builds behavioral mastery, and streamlines performance. Ultimately this will lead to fewer errors and reduced time to completion by minimizing unscheduled downtime.

The Solution: Consider the problem of equipment operation and maintenance. In this case, a worker might don a Microsoft Hololens or some other hands-free AR device. Let’s examine the capabilities of this solution to determine whether it meets the enterprise-grade standards outlined above.

The AR assets must be presented to the user in a way that minimizes the load on the cognitive system.

1. An enterprise-grade AR tool must engage the cognitive, behavioral and experiential learning systems in the brain in synchrony. With the Hololens (or any other hands-free AR device) cognitive information can be provided in the form of a virtual dashboard that includes numeric data and step-by-step instructions that guide the worker’s behavior. The AR solution is wearable and hands-free, thus allowing the worker full range of motion to complete behavioral tasks, and to build a behavioral repertoire in a naturalistic setting.
2. The AR assets must be presented to the user in a way that minimizes the load on the cognitive system. Recall that the critical factor here is to present the user with what they need, where they need it, when they need it. What asset is needed (whether text, graphics, sound, video, or hologram) will depend upon the specific use case. In a typical manufacturing setting text and graphics are the dominant AR assets. Where assets need to be presented to minimize cognitive load is with a co-located, contextualized overlay so that the worker never has to shift gaze or attention away from the task at hand. Every gaze or attention switch slows the process and increases the likelihood of error. Finally, when the asset is needed is just-in-time. As soon as the asset is needed it must be presented to the worker. Every second delay, or need to search long-term memory increases the cognitive load slowing the process and increasing the likelihood of an error.
3. The AR solution must be interactive in the sense that AR assets must guide the users’ behavior, and react to the users’ behavior in real-time. Let’s assume that the information in the virtual dashboard is providing the worker with step-by-step guidance while monitoring the correctness of the workers’ actions. Most likely an IoT solution is working in tandem with the AR system to determine what assets to present and to determine when and where they should be presented. Once a step is completed successfully, the next instruction is presented immediately because the IoT solution knows in real-time that the step was completed successfully. If a step is not completed successfully the worker is immediately presented with corrective feedback and a chance to complete the step successfully, again because the IoT solution knows in real-time that the step was not completed successfully.

This broad-based and coordinated brain activation that results from enterprise-grade AR solutions reduces the cognitive load while simultaneously providing the opportunity for limitless cognitive learning and behavioral practice.

With an enterprise-grade AR training tool, the training can be repeated as many times as the user would like, and the user can be placed in a broad range of situations. For example, the user can be tested by placing them under time pressure, or they can receive additional training on rare, but potentially dangerous, situations.

This broad-based and coordinated brain activation that results from enterprise-grade AR solutions reduces the cognitive load on the user while simultaneously providing the opportunity for limitless cognitive learning and behavioral practice. Taken together this speeds learning and retention quickly builds behavioral mastery and streamlines performance. Ultimately this will lead to fewer errors and reduced time to completion by minimizing unscheduled downtime.

Recommendations and Closing Remarks:

Whether you are an AR developer or user, it is important to understand which AR solution to use when. The best way to make this determination is to understand “your brain on augmented reality” so that you can select an enterprise-grade AR solution.

Here are some guidelines:

– If the users of your AR solution need to learn critical motor skills along with relevant subject matter, as in healthcare or manufacturing, then an enterprise-grade AR solution that uses a wearable device to keep the hands free to work in recommended. This AR tool should present AR assets, such as text, graphics, sound, video, or holograms, in a co-located fashion where the user needs them, and just-in-time when the user needs them. This will reduce the cognitive load on the user that is an ever-present bottleneck on cognitive processing. These assets must be interactive and temporally linked to the users’ behavior. Most likely this means that an industrial IoT solution is being combined with the AR solution to achieve real-time interactivity. If a correct behavior is elicited by the user, then the next asset in the chain should be presented. On the other hand, if an incorrect behavior is elicited by the user, the AR solution must provide some form of real-time corrective feedback to the user.

This AR tool should present AR assets, such as text, graphics, sound, video, or holograms, in a co-located fashion where the user needs them, and just-in-time when the user needs them.

– If you decide to choose a non-interactive AR solution, consider the use case as you decide between a tablet-based solution and a hands-free wearable solution. If your goal is behavior skills development, then choose a wearable, hands-free device. Allowing workers to generate natural motor movements, instead of holding the tablet with one hand while performing some task with the other, will go a long way toward building muscle memory.

– The specific use case will determine whether location-based assets such as GPS, beacon, or triangulation tools are needed. Similarly, the range of content types available should be determined from the use case. For example, holograms, generated voice and graphics may be critical when a 3-d mental representation of a complex piece of machinery, system in the body, or retail setting is needed to build a cognitive and experiential understanding. The more that the user needs continuous eye fixation on the task at hand, the more you want graphic overlays (perhaps in the form of 3-d holograms) and generated voices. Keep in mind what the brain needs, not what is flashy.

– In this report, we focus on training as the use case, but what if you want an AR tool to enhance performance of a repetitive task such as navigating an automobile or managing inventory? In these cases, the goal is to streamline the performance of a well-understood process, not to impart a detailed cognitive and behavioral understanding. When performance is the goal, the key is to reduce cognitive load by reducing working memory demand and the number of attention shifts. An AR tool that automatically detected the user’s location and directs their focus is ideal. In navigating an automobile, the ideal AR solution would overlay a virtual map on the windshield offering turn by turn instructions through non-distracting video and audio. In the case of inventory management, the ideal AR solution would automate the counting of items, facilitate registering the information, flag cases in which more inventory must be ordered, then direct the user to the next item. In both cases, this significantly reduces the cognitive load which reduces errors and speeds time to completion.

In Summary:

AR tools have enormous potential in manufacturing and many other sectors including automotive, medical, and retail, to name a few. Although the necessary AR features depend upon the use case the goal is always to reduce cognitive load by broadly engaging experiential, cognitive and behavioral learning centers in the brain. Enterprise-grade AR tools have the potential to engage the learners’ brain with co-located information, just-in-time experiences and behavioral training that provides the users’ brain with what they need, where they need it, and when they need it. This speeds time to productivity while reducing errors and costs.

AR tools have enormous potential in manufacturing and many other sectors including automotive, medical, and retail, to name a few.