New technology lets humans and autonomous robots speak the same language
Researchers from the ORCA (Offshore Robotics for the Certification of Assets) Hub, a consortium of five universities led by Heriot-Watt and Edinburgh universities, have unveiled a new method of communication that allows machines and humans to speak the same language and understand each other’s actions in real time.
Named MIRIAM (Multimodal Intelligent inteRactIon for Autonomous systeMs), the new method of communication is a significant breakthrough that enables people to work with autonomous robots. It allows users to ask robots questions, understand their actions and receive mission status through a digital twin. MIRIAM uses natural language, allowing users to speak or text queries and receive clear explanations about a robot’s behavior in an intuitive way.
By improving how robots communicate, MIRIAM will help build the confidence of those who will be using them and overcome any adoption barriers in the workplace.
MIRIAM will be used by a new collaborative team that sees Heriot-Watt University, through the ORCA Hub, working and integrating its research with the expertise of Phusion, an engineering software firm and Merkle Aquila, a data science firm. They are the latest additions to global energy partner Total’s multimillion-pound research consortium that is working to develop autonomous capabilities with multiple robotic systems for work on offshore energy installations. MIRIAM will initially work with a tracked robot that will be deployed at Total’s Shetland Gas Plant.
“Total believes that robots have a huge amount to offer our industry. We are pioneering their use on oil and gas sites and this collaboration is helping us to make significant progress,” Kris Kydd, Head of Robotics at Total E&P UK, said. “This announcement is the first important step towards making the actions of autonomous robots clear and understandable, and this will allow them to gain the confidence of the wider workforce. Robots offer immediate advantages such as increased safety and efficiency. In the long-term, they offer us new ways of working and are limited only by our imagination.”
“When robots have an autonomous element, it means that they sense the environment and can make certain decisions by themselves,” Helen Hastie, professor of computer science at Heriot-Watt University and ORCA work package lead, said. “However, there is currently a communication barrier between robots and their human supervisors in relation to the reasoning behind system actions. This is particularly problematic in remote, highly challenging, and hazardous environments such as offshore, which can involve multiple vehicles and platforms.”
“This lack of transparency can lead to a reduction in situation awareness and therefore user-confidence in the robots, hindering the human-robotic team and resulting in unnecessary aborts or laborious manual manipulation of the robot,” Dr Hastie added. “It is therefore essential that a supervisor understands what the robot is doing and why.”
“MIRIAM’s technology is designed to provide invaluable transparency in this process. Our team of researchers have taken a user-centered design approach, which supports Total’s forward-thinking efforts to making operations safer offshore. We have also modified MIRIAM to the specifics of Total’s robotics program.”
Phusion is supporting the project with the implementation of a master data model that will enable the framework for a digital twin of Total’s facilities. This will allow the creation of autonomous routines, enabling Total’s robot to undertake maintenance checks and feedback results in a consistent manner to the digital twin – creating a single source of information for the operating asset.
MIRIAM will allow humans to communicate with this digital twin, and the data held in it, including the robot, via text or speech. This means supervisors can ask where the robot is going, what it plans to do next and why, and query data it has gathered during its missions.
“This ground-breaking project demonstrates the importance of high-quality, accessible engineering data as an enabler of digital transformation,” Steve Pearson, Chairman of Phusion IM, said. “Through our master data model developed for Total, the robots are able to develop capabilities through machine learning and deliver the ultimate benefits of efficiency and operational improvements alongside workforce safety.”
Merkle Aquila is the industrial data science, data engineering and agile delivery collaborative partner. The company has been working with the team to develop solutions using AI to extract maximum value from robotic image data and demonstrate the automation of manual human inspection using computer vision algorithms.
“We have been able to develop working solutions quickly using agile delivery methods, algorithms developed on the Microsoft cloud, tactical data collection and close team collaboration,” Sean Robertson, Total Client Director from Merkle Aquila, said. “We have demonstrated leading innovation in tackling these barriers to develop a robust and adaptable solution.”
The project will be used by the original OGRIP (Offshore Ground Robotics Industrial Pilot) robot developed by Total E&P UK, Austrian tech firm Taurob and the Aberdeen-based Oil and Gas Technology Centre (OGTC). OGRIP is designed to support exploration and production operations in increasingly harsh and challenging conditions, including extreme cold, arid climates and isolated locations.
