Learning from other Domains to Advance AI Evaluation and Testing: Testing in Aircraft Design and Manufacturing

In aviation design and manufacturing, testing can generally be described as serving the following purposes: (i) facilitating development of a product design; (ii) verifying its functionality and robustness; (iii) demonstrating compliance of the design with regulatory standards in order to obtain approval from a civil aviation authority such as the U.S. Federal Aviation Administration; and (iv) verifying that the as-built product conforms to its design, complies with applicable regulations, and is in a condition for safe operation. Testing is, therefore, closely linked to showing compliance with regulatory requirements, which forms the predicate for obtaining approvals from aviation authorities that allow aircraft to be operated. To be sure, some design features do not require testing to show compliance, and compliance showings may also be made based on analysis, historical findings of compliance, inspection, or system architecture. But, testing remains a centerpiece of compliance, and the regulations to which compliance must be shown fundamentally define the nature of aircraft products and the processes for creating them. For these reasons, testing with respect to aviation products must be considered in the context of the overarching regulatory framework for standards and how compliance to standards must be shown.