News & Information   |   Photos   |   Videos   |   NREC Brochure   |   NREC Flyers  
© Carnegie Mellon University
News & Updates

Field testing the Crusher autonomous unmanned ground vehicle during the UPI program. While field testing is a valuable part of development, it cannot uncover all autonomy system vulnerabilities.

NREC Developing Stress Tests for Autonomy Software

Researchers from NREC are developing a toolkit for stress testing autonomy software.  Stress Tests for Autonomy Architectures (STAA) will push these systems beyond their limits, unearthing vulnerabilities that are impossible to find during traditional field tests.  A $1.8 million award from the Department of Defense’s Test Resource Management Center (TRMC) underwrites this work.

Developing robust autonomy software for robots is a difficult task.  This type of software is very complex – too complex for code reviews to catch every major problem that could affect system performance and safety. To test it, developers typically observe its performance in the field as part of an entire unmanned autonomous system.  These system-level tests involve months of careful planning and can be very expensive.      

It’s impossible to field test autonomous systems under all combinations of operating conditions, vehicle behavior, sensor inputs and environmental factors.  Since these factors usually can’t be controlled or replicated, it’s also impossible to perform regression testing to compare successive versions of the autonomy software.  Odd behavior and outright software failures might not be discovered until the system is actually deployed and used.    

STAA takes an entirely different approach to testing.  It’s based on Ballista,® a software tool for stress testing computer operating systems and other software systems.  Instead of testing how an operating system handles normal, “clean” inputs, Ballista tests it on abnormal, “dirty” inputs. It feeds anomalous and erroneous data – for example, inputs that are out of range or cause divide by zero errors – into an operating system (or other system with a similar software interface) until it breaks.  This provides detailed information about system vulnerabilities.

STAA will use the same tactic to test autonomy systems for robots.  It will feed abnormal inputs – for example, out-of-range sensor data and vehicle readings – into the autonomy software until it breaks. This reveals vulnerabilities that cannot be uncovered by other forms of testing.  It will also compare the performance of different versions of the autonomy software on the same inputs, enabling developers to do regression testing.   

With STAA, there’s no need to wait until the entire robotic system is fielded.    Stress testing can be done at any point during the development process. Problems can be discovered early on, when they are easier to fix.  This early testing will improve the autonomy software’s robustness, resilience, and performance.

One of STAA’s biggest benefits is that it creates a benchmark for testing the robustness of autonomy software.  This is especially valuable for acceptance testing of unmanned autonomous systems.   These systems currently undergo standardized acceptance tests that focus primarily on their physical characteristics (mobility, operating temperature range, and so forth).  STAA will allow their software to be tested with the same level of rigor.  Autonomy software can be evaluated on how well it performs under stress testing with STAA, including whether it exhibits unsafe behavior.  This will give acceptance testers a better idea of how the overall system will perform when it’s deployed.

Note: This project is funded by the Test Resource Management Center (TRMC) Test and Evaluation/Science & Technology (T&E/S&T) Program through the U.S. Army Program Executive Office for Simulation, Training and Instrumentation (PEO STRI) under Contract No. W900KK-11-C-0025.