Lost in translation: the validity of a systemic accident analysis method embedded in an incident reporting software tool
- Authors: Goode, Natassia , Salmon, Paul , Taylor, Natalie , Lenné, Michael , Finch, Caroline
- Date: 2016
- Type: Text , Journal article
- Relation: Theoretical Issues in Ergonomics Science Vol. 17, no. 5-6 (2016), p. 483-506
- Full Text:
- Reviewed:
- Description: ABSTRACT Despite the proposed advantages of systems accident analysis (SAA) methods for understanding incident causation, they have not been widely adopted by practitioners. The aim of this study was to evaluate the criterion-referenced validity of an SAA method embedded within an incident reporting software tool. Thirteen practitioners used the tool to collect and analyse incident data within their organisation. The incident data were then analysed by researchers experienced in using the SAA method. Overall, there were low levels of agreement between participants and researchers regarding the identification and classification of factors and relationships. The findings indicate the systems thinking principles underpinning the SAA method may have been ‘lost in translation’, in that participants often identified only one or two factors and showed a poor understanding of how to identify relationships between factors. The methodological developments required to ensure that practitioners can validly apply the SAA method are discussed.
Developing a contributing factor classification scheme for Rasmussen's AcciMap : Reliability and validity evaluation
- Authors: Goode, Natassia , Salmon, Paul , Taylor, Natalie , Lenné, Michael , Finch, Caroline
- Date: 2017
- Type: Text , Journal article
- Relation: Applied Ergonomics Vol. 64, no. (2017), p. 14-26
- Relation: http://purl.org/au-research/grants/nhmrc/565900
- Full Text: false
- Reviewed:
- Description: One factor potentially limiting the uptake of Rasmussen's (1997) Accimap method by practitioners is the lack of a contributing factor classification scheme to guide accident analyses. This article evaluates the intra- and inter-rater reliability and criterion-referenced validity of a classification scheme developed to support the use of Accimap by led outdoor activity (LOA) practitioners. The classification scheme has two levels: the system level describes the actors, artefacts and activity context in terms of 14 codes; the descriptor level breaks the system level codes down into 107 specific contributing factors. The study involved 11 LOA practitioners using the scheme on two separate occasions to code a pre-determined list of contributing factors identified from four incident reports. Criterion-referenced validity was assessed by comparing the codes selected by LOA practitioners to those selected by the method creators. Mean intra-rater reliability scores at the system (M = 83.6%) and descriptor (M = 74%) levels were acceptable. Mean inter-rater reliability scores were not consistently acceptable for both coding attempts at the system level (MT1 = 68.8%; M T2 = 73.9%), and were poor at the descriptor level (MT1 = 58.5%; M T2 = 64.1%). Mean criterion referenced validity scores at the system level were acceptable (MT1 = 73.9%; M T2 = 75.3%). However, they were not consistently acceptable at the descriptor level (MT1 = 67.6%; M T2 = 70.8%). Overall, the results indicate that the classification scheme does not currently satisfy reliability and validity requirements, and that further work is required. The implications for the design and development of contributing factors classification schemes are discussed. © 2017 Elsevier Ltd