Intuitive Control of Robotic Systems Could Revolutionize Combat

Shifting the military from operating individual robots to directing groups of them from afar will transform the Department of Defense's autonomous robotic systems and bring warfighting into a new era.

To make that happen, MITRE is taking existing autonomous technologies that work for single vehicles and wrapping them into a common command and control framework for collaborative missions with different kinds of robots. It's part of a year-long effort called Heterogeneous Autonomous Vehicles with Intuitive Control (HAVIC).

"We're going to see cheaper robots employed on a broader scale and we need to figure out new ways to coordinate them," says MITRE's Carrie Rebhuhn, who leads the project.

The Pandemic Spurs Innovations in Testing

COVID-19 limited the team's ability to meet and test robots in person. They adapted by enabling remote support of robotic testing to continue their work on HAVIC during the pandemic.

Several representatives from MITRE's Army Program and our Charlottesville site observed a live demonstration in Purcellville, Virginia. The demo modeled an intelligence, surveillance, and reconnaissance (ISR) task supported by both aerial and ground robots.

Onsite observers first watched an ISR task conducted by the group of robots to demonstrate the power of heterogeneous pairing, including small unmanned systems and larger manned vehicles.

Additional guests observed the event on a Microsoft Teams call. Mary Hatfield, the lead developer for the HAVIC User Interface (UI), orchestrated the mission from the comfort of her home, while safety operators monitored robot behavior in the field. The UI was built on an Android tablet and showed the real-time positions and status of the autonomous robots and enabled tasking of the robots over the internet.

This demonstration featured a Jeep entering an unknown area and tasking one of its aerial "companions" to search ahead for threats using image-recognition software. The threat was identified by the aerial drone and was confirmed by the human user through the Android interface designed under HAVIC. Two other robotic systems (one ground, one aerial) then intercepted the threat and provided a battle damage assessment to the user in the Jeep.

Hatfield explains: "The team on the ground sent me live commands on how to maneuver the robots. When I received a command, I put it in through the tablet emulator and the robots responded. The team on the ground relayed what the robots were doing so I knew when they were misbehaving or doing what was expected."

Hatfield shared her computer screen remotely as she commanded all three robots through HAVIC's user interface.

Mission Command from Home

The team set up a new method to access and connect the robots that allowed one user to control several robotic systems simultaneously remotely across the internet.

"Enabling that access was really important to continue field testing so we could have much broader participation and development without the risks of physically getting together," Hatfield says.

Rebhuhn, who was part of the onsite demo, added that one of the most impressive things the team observed was the flexibility and robustness of the missions they configured. That allowed them to deviate from the rigid script that often characterizes robotic systems demonstrations.

"Within 10 to 15 minutes, we discussed a new mission and made the robots do what we wanted in a coordinated way," she says. "This was a pattern of execution that we had never tested. Our command method enabled us to execute without reengineering the system to do those new things."

Robots and the Future of Combat

Jeff Heissler, MITRE's director of Army Modernization and HAVIC senior lead, says robotic systems will revolutionize the nature of war and how soldiers fight.

"That demonstration was about showing the art of the possible," he says. "Getting multiple robots—and multiple types of robots—collaborating creates a greater effect than either humans or one robot trying to accomplish a task."

Ultimately, the HAVIC team aims to shift the cognitive burden to take advantage of onboard autonomy and automation so that the soldier can be a manager of robots rather than commanding a single robot.

"That might involve figuring out the optimal route for unmanned aerial vehicles or using drones to identify threats and alert ground robots to destroy those threats in advance," Heissler says. "We're showing how all robots can interact."

We're also working on building autonomous systems that don't require a constant communications link.

Heissler explains that military operations often require engagements in hostile environments where networks may be unreliable or do not exist.

"That network is giving the user situational awareness. However, if the adversary takes down that network, and you lose command and control operations, you want to develop an autonomous system that doesn't require constant communication because it has instructions it can maintain without intervention."

"Initially we focused on one user controlling multiple robots in the vicinity of the robots. I think COVID made us see that we don't need to be near the robots—with a network connection, you can control them from anywhere.

"It was an unexpected side benefit that we discovered. Now it becomes a question of enabling this for the military."

—by Aishia Caryn Freeman