Traditional test methodologies employ testing as part of the validation phase of a project. While testing plays an important role for validation and quality control, it can also serve as a developmental catalyst if the experiments are performed during earlier phases of a design. Development is accelerated by iterative testing, because it exposes false assumptions before changes to the design become cost-prohibitive.

The discontinuous step fields form a repeatable test fixture which simulates rubble.
In the tactical robot world, this creates a dilemma. While many companies house “robot playgrounds” for internal use, very few have the resources to devote to a comprehensive test bed, and few ad-hoc test areas are equipped with the instrumentation necessary to collect and analyze data rigorously. Independent robot test facilities can be found at some military bases, universities, and non-profit organizations, such as the Small Robotic Vehicle Test Bed (SRVTB) at Southwest Research Institute (SwRI), but to take advantage of these resources, a developer must divert time and personnel to off-site evaluations. In most cases, traveling to a specialized third-party test site is reasonable for a mature robot platform but not for research prototypes because a research - er cannot stop to test after every minor design change or software tweak, even if it would help them understand the robot better.

Another factor that makes early testing difficult is that most commercial robot builders prefer to keep glitches to themselves, even though these are a normal part of the development process. By nature, testing highlights deficiencies, and if a government sponsor or potential buyer is present, the developer is not truly free to fail in a way that is most beneficial to the design. Instead, good business practice dictates that the developer focus only on the most impressive and reliable features of the robot during a test and avoid taking risks in front of a client.

Risk aversion may be good business practice, but it does not make for a good design environment. Because of this conflict and the disruptions of early testing, developers may be tempted to delay testing until most of the major design decisions have been made. The downside to this is that the developer may not understand some of the more subtle design tradeoffs until it’s too late to make a change.

In cases where an early public test is unavoidable, developers sometimes lobby for easier testing on the grounds that the robot is not yet ready to be stressed. This conservative approach is understandable, but it usually reduces a test to a “demonstration of current capabilities.” When a test becomes a demonstration, it leaves observers with a false impression of a system’s reliability, and it deprives the designer of the opportunity to learn from pushing the system’s limits.

These are significant challenges for program managers in the world of tactical robotics, but now there is a new tool to help test robots sooner without hindering early research efforts. The Department of Homeland Security (DHS), in collaboration with the National Institute of Standards and Technology (NIST), has sponsored a suite of internationally recognized robot testing standards (ASTM E54.08.01), which are now available for use by robot developers. Instead of facility-specific testing, a robot developer can go to any of several equivalent test beds around the world, or the robot builder can even construct many of the apparatuses from home.

The procedures, methods, and apparatuses defined by ASTM E54.08.01 are products of many years of development and feedback from robot companies, first responders, military personnel, and the testing community, and they continue to evolve alongside the robots they evaluate. The standard incorporates a mix of physical obstacles and sensor evaluations that are analogous to situations robot operators will encounter in the real world. Obstacles such as stairs, inclined ramps, and simulated rubble highlight a robot’s mobility, while fixtures such as the “Directed Search” apparatus are used to evaluate cameras in both light and dark conditions. The test suite also includes tests for radio range, manipulator strength, manipulator dexterity, battery endurance, and more. There are even budding metrics for flying and swimming robots.

A first responder looks on as Telemax inspects a simulated train wreck at Disaster City.
DHS originally intended ASTM E54.08.01 for validation purposes, and they had some early successes. In 2007, the METROTECH National Capital Region Bomb Squad Working Group hosted a competition based on an early version of these standards. The data they collected was used to make a collective $1 million purchasing decision. Together, first responders selected one large and one small robot, and now each participating municipality is trained for interoperability with their neighbors, who all use the same robot models.

Another success story came in July 2011, when the Joint Improvised Explosive Device Defeat Organization (JIEDDO) worked with NIST to organize a competition for small throwable robots at NIST’s test facility in Gaithersburg, MD. Both the METROTECH competition and the JIEDDO throw-bot competition helped to put “best in class” robots into the hands of tactical robot operators. Although the situation for a domestic response team is very different from what a soldier faces in Afghanistan or Iraq, in both cases, the purchasing decisions were based on numerical comparisons of robot performance in the specific test methods that were most relevant to the end user.

The success of the new test methods does not apply only to people who make purchasing decisions. The tests have also proven useful for robot companies looking for an edge. For example, many developers collect invaluable input from operational users at NIST’s annual Response Robot Exercise, where first responders are given hands-on experience with the latest and greatest robot capabilities at the Disaster City training facility in College Station, TX.

The ICOR Caliber pulls a sled weighted with 175 lbs during a towing capacity test at Disaster City.
Andreas Ciossek, a representative of the German robot maker Telerob, uses the ASTM test methods to improve the user interface on his rescue-robot, Telemax. He says, “For me, the test methods deliver, from a development perspective, very valuable results when I am watching other people doing the same tests with our robot. When you are developing such a system you try to make everything as easy to use as possible, or, in other words, ‘intuitive.’ But when you are coming, for example, to Disaster City and see many different people using your system, you will most probably see the same number or even more intuitive ways of doing things and some of them you have never thought of.”

Ciossek also uses the independently collected scientific data from these tests as a sales tool to demonstrate the virtues of his robot. He says, “First of all, the test methods are a great thing to show the capabilities of the systems, and because the customer can be sure the results were gained in a reasonable way, they are very helpful when it comes to selling a system.”

So how can other robot developers take advantage of these test methods? There are several ways. The easiest way is to sign up for NIST’s annual Response Robot Exercise at Disaster City. For year-round access to the test methods, NIST has also established two permanent test beds in North America: one in Gaithersburg, MD, and one at Southwest Research Institute in San Antonio, TX. (There are also test facilities in Germany and Japan.) For aspiring young roboticists, the annual Robocup Rescue competition also uses modified versions of the ASTM apparatuses to pit the world’s best student-built robots against realistic rescue scenarios.

While these test opportunities are a step toward easy access to universal robot testing, this still does not solve the problem of how developers can test a new robot without disrupting the development cycle. It is true that any developer can use the standard’s specifications to replicate an apparatus for internal use (and the test community highly encourages this), but to build all of the fixtures would require a significant amount of space, time, and carpentry work. To reduce the burden on developers and to encourage early and frequent use of these new tools, Southwest Research Institute is working with several government agencies to develop a mobile version of the ASTM test bed that breaks down into a series of shipping containers. The idea behind the “test-bed-in-a-box” is that any particular test method can be removed from a standard forty-foot shipping container, set up quickly by two or three people without using tools, and be easily replaced back in the container after the experiments are complete. This allows developers to use the same facilities and methods that DHS and JIEDDO have used, but in the privacy of their own parking lot. And rather than maintaining a year-round dedicated space for the facility, the developer can return the shipping containers to Southwest Research after they are finished. Under this model of “taking the test to the user,” it will soon be possible to deliver the same test methods and facilities that the U.S. government uses for validation to almost any parking lot in the world. Prototypes of the “test-bed-in-a-box” are expected to be completed by mid-summer of 2012.

This article was written by Andrew Moore, Senior Research Engineer, Department of Electronics Systems and Robotics, Applied Physics Division 14, Southwest Research Institute (San Antonio, TX). For more information, Click Here .