Train welders using machine learning and augmented reality

Ever since the ancient Egyptians fused two pieces of gold together by hammering them together, welding technology has constantly advanced.

Iron Age blacksmiths used heat to forge and weld iron. The discovery of acetylene at the beginning of the Industrial Revolution added a versatile new fuel to welding. Two of his engineers invented metal arc welding at the end of the 19th century. More recently, the rise of robotic welding systems and advances in high-strength alloys have expanded welding applications.

Despite a long history of technological improvements, welding remains a difficult skill to learn. Requires a combination of technical knowledge and manual dexterity. And given the prevalence of this welding in many industries such as manufacturing, construction, aerospace, and automotive, the need for skilled welders remains strong. According to the American Welding Society, U.S. employers currently face a shortage of 375,000 welders.

At Carnegie Mellon University, researchers are tackling this problem by developing a new way to train welders that reapplies new techniques. Dina El-Zanfarey, assistant professor in the School of Design, and Darrah Byrne, associate professor in the School of Architecture, are collaborating with a team of researchers to develop an augmented reality (XR) welding helmet and torch system to assist welders. has been developed. Gain the specific knowledge you need to master difficult skills.

“Not only is this a truly great project, but it also incorporates key objectives of the MFI mission,” said Sandra DeVincent Wolf, MFI Executive Director. “This is groundbreaking research that advances manufacturing technology, contributes to workforce development, and engages community partners.”

Augmented reality combines virtual reality (VR), a computer-generated environment that simulates real or imagined experiences. Augmented Reality (AR). It combines computer-generated information with the user's real-world environment. In mixed reality (MR), the real world and digital objects coexist and interact in real time. Together, these features create an immersive experience that allows users to interact with information, environments, and digital content in real time.

Welder training requires the development of hand-eye coordination and a keen awareness of body position and movement in space. This embodied knowledge is acquired through hands-on interaction with tools and materials and can be difficult to reproduce in training scenarios.

Researchers at Carnegie Mellon University worked to better understand training challenges by organizing a series of co-design workshops. They worked with eight instructors and his four students at Industrial Arts Workshop (IAW), a nonprofit youth welding training program in Pittsburgh's Hazelwood neighborhood, to equip welding helmets and torches with Meta Quest. We have developed a system that integrates with Pro and machine learning. A model that enhances concrete learning of welding in three key ways.

Motion sensing integrated with visual XR guidance

The highly immersive and tangible nature of welding labs makes it extremely difficult for instructors to visually monitor the process and provide timely, safe, and audible feedback to students. Neither written instructions nor feedback can convey nuanced practical skills in real time.

The researchers overcame these obstacles by modifying a welding helmet with a Meta Quest headset that displayed a series of visual feedback mechanisms. This mechanism guides students during training sessions and provides a record of performance that instructors can evaluate during or after the session.

Two separate XR indicators inside the welding helmet indicate slight changes and adjustments the welding student needs to make to maintain the correct angle of the welding gun connected to the Quest Touch controller. Status icons near the top of the headset viewport allow users to see feedback without taking their focus away from the active weld. Status icons also give instructors and users a clearer overview of performance when viewing live replays.

Leveraging feedback from workshop instructors, the researchers enabled users to use the welding gun to set the start and end points of the weld line and set the desired scroll guideline position. We determined how to align the XR representation of the weld to the actual workpiece. To obey.

Sensing acoustic cues during welding practice

Carnegie Mellon University researchers learned from workshop instructors that experienced welders can evaluate welds by actively listening. So instead of visually evaluating welds after completion, the system can diagnose welds in real-time using auditory-based methods.

“For example, my instructor told me that the proper welding speed should sound like bacon sizzling, not popcorn,” Elzanfali explained.

Inert gas welding of metals involves forcing a metal wire through the tip of a welding gun, shielding the wire with an inert gas, and using the heat generated by the short-circuit current between the wire and the workpiece to fuse the two metals together. . Incorrect settings of this system will result in poor weld quality. For example, if the welder's amperage setting is too low, the weld bead will be too thin and the work plate will have uneven penetration.

Prior research and feedback from AIW instructors indicate that different settings change the welding sound, which provides potentially important training feedback. However, the extreme heat, light, and acoustic conditions of the welding space, along with the bulky welding helmets and other personal protective equipment required to protect against intense heat, sparks, ultraviolet light, and metal splatter, make it difficult for welders to use this equipment. Limits the ability to perceive auditory stimuli. .

By employing sound detection enabled with Tiny Machine Learning (TinyML) to recognize important factors such as settings and tip distance, researchers are able to eliminate errors such as those that occur when a gun tip collides. , we trained and deployed a model to provide visual feedback indicating errors detected by sound. Too far from the weld plate.

The researchers asked experienced welders to perform the same welding motion repeatedly, changing only one setting for each weld. They collected over 20 minutes of audio data, evenly distributed across category settings, for use in training and testing a TinyML classification model.

TinyML focuses on deploying and running machine learning models on resource-constrained devices such as microcontrollers. In this case, the microcontroller was connected to an extended helmet to provide feedback. The researchers trained his TinyML model to alert trainees to watch out for common mistakes, such as incorrect settings and gun tip distance.

Sound was also used to detect the beginning and end of welding. Researchers used recordings from five different devices simultaneously (two microcontrollers, two Samsung smartphones, and a USB microphone) to train a classification system that was able to detect welds with 97% accuracy. We collected 19 minutes of welding sounds. This classifier replaced the need to electromechanically detect interaction with a physical button on a welding gun, which was used to initiate weld tracking and make the system more portable.

meditation before welding

During the workshop, the researchers found that instructors taught students how to relax and develop concentration before starting welding as a way to encourage relaxation and develop concentration to offset the effects of a welding environment that can be overwhelmed by loud noises. I made sure to encourage people to practice mediation and breathing exercises. Sparks, heat, and the smell of burning.

To enhance mindfulness practices, researchers programmed the platform to begin each welding session by encouraging trainees to participate in breathing exercises. We also installed an anemometer near the mouth and nose inside the welding helmet to measure the wind speed of exhaled breath and track breathing patterns over time to help welding students adjust their breathing to improve work performance. We've developed system prompts to help you improve your experience.

The system's ability to sense motion, detect sound, and increase user focus through mediation and breathing exercises will help students transfer the skills learned in virtual training to real-world welding practice. The ability to provide real-time guidance has many benefits for both students and instructors who need to rely on information obtained after a weld is complete to evaluate performance. The holistic approach goes beyond welding training and could also benefit his training in craft and skills with XR systems.

“What's really exciting about our work is the system's ability to take an off-the-shelf XR and welding setup with a little modification to enable an on-site welding experience,” El-Zanfaly explained.

Their work has already been recognized with awards at the 2023 Association for Computing Machinery (ACM) Conference on Interactive Surfaces and Spaces and the 2024 ACM Conference on Tangible, Embedded, and Embodied Interactions.

Going forward, the team plans to pursue many opportunities to improve both the technical aspects and the embodied experience of the work. They plan to introduce this platform into his A/B lab research and evaluate how long-term use of this device over several weeks at IAW contributes to the formation of skills, habits, and beginner experience.