12 B. B. Schroeder et al. QMU formulation also included a fourth element, credibility, defined as the answer to the question ‘how much confidence do we have in our risk assessment?’ [4]. Historically at Sandia National Laboratories (Sandia), QMU was largely applied to experimental data-based problems, but it appears likely that an integration of computational simulation (CompSim) results and experimental data will be the paradigm of the future. While processes for conducting QMU have developed over time (e.g., [5, 6]), there are still no formal processes for evaluating the credibility of a QMU analysis. QMU entails comparing a performance measure to a performance requirement to determine the likelihood of functioning as intended, considering all relevant uncertainties. Implementing a QMU analysis requires building a team with the relevant expertise; identifying performance measures and requirements; assimilating relevant data; running an analysis; and communicating the results. Considering these steps of a QMU analysis, a corresponding QMU credibility assessment should address many of the inherent aspects of the analysis such as relevance of the performance measure and requirement, data quality, and analysis limitations. 2.1.2 Why Measure Credibility? There is currently a gap in guidance within Sandia National Laboratories (Sandia) for assessing the credibility of QMU analyses. New guidance for QMU was recently released as internal documents within Sandia in two sections: (1) an overview of high-level QMU concepts and processes and (2) descriptions of statistical tools that can be used to derive QMU results, with a focus on QMU for experimental data. This new guidance pushed for broadening the analyses supporting QMU results beyond extrapolative statistical models and advocated for a more holistic picture of risk, including information garnered from both experimental and CompSim campaigns. Although this new guidance improves the informational basis of QMU analyses, it does not provide a consistent framework for communicating the credibility of analysis results. Credibility assessment guidance for QMU is needed because: • Decision-makers are increasingly asking for credibility assessments when being provided analysis results. Decisionmakers are learning that they must understand the level of confidence they should invest in the results to better utilize the analysis that they commissioned. • Failing to provide guidance for communicating credibility may lead to overconfidence in results. A question that should be posed to QMU analysts is, “What is the credibility of your results?” Without asking this question, the decision-maker may believe results are more reliable than is warranted and make ill-informed decisions. • A unified QMU credibility framework would result in greater consistency in information presentation. When credibility results are analyst-specific and/or analysis-specific, decision-makers will interpret results differently depending on who conducted the analysis. • Streamlined documentation of important auxiliary information (e.g., metadata, methods) is integral to understanding and reproducing QMU results. Summary QMU results (for example, margin over uncertainty ratios) always rely on auxiliary supporting information about the QMU process and supporting experimental data. Without a consistent credibility assessment framework, decision-makers must rely on source credibility, or their belief in the source of the information. Although not specific to the reception of QMU results, psychological research has explored the role of source credibility in other information distribution areas. Across the psychology literature, source credibility is typically attributed to a person providing a message. Key aspects of source credibility include the source’s trustworthiness and expertise [7]; to a lesser degree, composure, dynamism, sociability [8] and even accents of voices [9]. Chaiken and Maheswaran found source credibility can affect decisions in two ways: (1) by serving as a peripheral cue for simple acceptance or rejection of an argument, and (2) by biasing the strength of the decision-maker’s argument processing [10]. While biasing the belief in results based on the source is potentially problematic in itself, Heesacker et al. found that as source credibility increases, persuasion also increases [11]. They attribute this phenomenon to more credible sources eliciting greater thinking about the message (improved information presentation, not informational content). Across psychological research a theme persists: human judgment is persuaded and biased by a variety of minute factors. As humans participate in high stakes decision making, it is important to understand how small changes in presentation of the message (or data) can unintentionally bias the decision-maker. To mitigate such bias, credibility frameworks may help through providing consistency, transparency, and structure.
RkJQdWJsaXNoZXIy MTMzNzEzMQ==