The adoption of advanced test methods increases the efficiency and capability of virtual V&V testing, making full Digital Twin development increasingly feasible. Standardized test methodologies and languages, such as SysML, open the door for high levels of automation in virtual test. Test methods are utilized at various stages of the design process, often in increasing complexity, to evaluate system design and performance:
• Virtual – Virtual test takes place completely in the cyber realm. The plant model and controller model communicate with simulated hardware systems. Effective virtual test is highly portable and lends itself well to collaborative efforts.
• SIL – Software-in-the-loop testing exercises production code, interfacing with a physics-based model of the plant. SIL testing can be completed entirely on low-cost client PCs, without any need to develop, maintain or occupy costly hardware assets.
• PIL – Processor-in-the-looptesting moves the test execution onto the targeted project architecture. Production code that has been compiled for the processor can reveal issues not found in SIL testing.
• HIL – Hardware in-the-loop testing brings real system hardware into the test setup. The controller and system are evaluated with interfaces to physical sensors and actuators. As the HIL system matures, components of subsystems can eventually be switched between real and simulated, depending on test objectives.
• Final – Final testing evaluates a system in a state nearly identical to the finished product, save for some modifications for data monitoring and collection. Operational costs of final testing are high, as are any changes to the design required when issues are found at this stage. Ideally, final testing is conducted largely for purposes of certification for safety critical systems, eg a test flight program required for type certification. When the final test stage is reached, system dynamics and behavior are highly predictable and understood based on the previous levels of testing.
Programs often utilize both open and commercial tools to develop (and make use of existing) test assets. With model assets developed in standards-based environments, reusability and portability are inherent benefits. As physical mechanical and electrical design and development activities are taking place, so do their virtual and cyber-physical counterparts. Early in systems development, requirements are captured and may be modeled using standards-based modeling technology such as SysML. SysML allows for a system or subsystems to be characterized by the elements of which they are composed, the relationships and interactions between those elements, and the behaviors the system exhibits. This allows functional requirements to be captured, verified using automated simulation, and formally verified for consistency. Another very powerful capability offered by formal modeling tools, including SysML, is the ability to automatically derive and execute functional test cases, i.e. test vectors, from the SysML model. A V-Model of an automated T&E systems development lifecycle:
The combination of traditional and advanced T&E methodologies can be thought of as a set of filters rejecting different types of design and implementation issues and flaws. Virtual test is used to ensure fundamental design flaws are rejected early in the development project. SIL test is brought to bear once production code becomes available and provides a fine filter ensuring application system and operating system bugs are rejected ahead of integration with hardware. The PIL test capability provides a test platform for finding and rejecting Hardware-Software Integration (HSI) issues as early as possible. The HIL test provides a platform for exercising integrated subsystems as well as the complete integrated system in a “fail-safe testing” environment. The complete set of development T&E capability as a set of filters to catch and reject design and implementation flaws and issues:
Virtual, SIL, and PIL capability also offer a cost-effective way to dramatically increase design and implementation scrutiny with far higher resolution and higher detail testing with much faster test cycle times. Leveraging these various, complex systems test capabilities can appear cost-prohibitive but are often viewed as the most cost-effective way to achieve certification for many categories of complex systems, particularly where safety-criticality, mission-criticality, and/or cyber-criticality are included within the requirements of the complex system.
In addition, the ADEPT Framework allows for a heavily automated T&E workflow, from project development and test definition all the way through live and post-run results analysis. A reusable project framework is developed in the ADEPT-DE development environment, based upon the system requirements, design, and test plans previously defined by the program. The project framework brings in model assets, in various popular formats, and assigns the framework to a real-time server for run-time. Project I/O is configured and connected to models using data dictionaries. Communications network definitions are created and maintained in ADEPT-DB, effectively managing thousands of signals required of complex systems, in a variety of protocols including ARINC-429, CAN bus, UDP, RS-422, RS-436, etc.
Test cases are executed, controlled, and automated using the developed frameworks in ADEPT-VI. While allowing for more traditional manual operation of test cases, advanced programs utilize the automation capabilities of this tool to operate at high efficiency at run-time while high value hardware assets are in use. Manual operation is replaced with scripted Python automation. Automation APIs allow for simple interfacing to external applications. Data monitoring and logging is configured for live or post-run analysis.
Live data visualization and analysis of test data is provided by ADEPT-GD. This graphical interface displays data collected from the real-time server on the client PC. Test results can be automatically collected using Python automation and deposited into test reports. Post-run, the high-level analysis can be performed utilizing the wide breadth of open Python libraries.
Best-in-industry organizations have been able to manage the cost of their traditional and advanced T&E capabilities by designing and implementing a harmonized capability solution for test automation, test analysis, and test data & results reporting across the T&E capability types.
