Developing Advanced Crew Resource Management (ACRM) Training: A Training Manual(77)

曝光台 注意防骗 网曝天猫店富美金盛家居专营店坑蒙拐骗欺诈消费者

EnsuringEvaluator Reliability Prior to LOFT/LOE Assessment
ACRM promotes the use of videotapes of crew performance in conjunction with the IRR process to establish evaluator reliability. Videotapes of real crew performance, as opposed to tapes of crews acting a part, allow a group of instructors to observe and rate the type of performance they are likely to see in LOFT/LOE sessions. The IRR process is used to provide those crews with the results of their ratings along with measurements of their agreement, congruency, consistency, and sensitivity.
An airline should strive for a specified level of reliability prior to conducting the LOFT/LOE portion of crew training. One way to ensure that the instructor/evaluators reach that level is to pick a realistic set of benchmarks. If an airline does not have substantial experience in the standardization of assessment, it is reasonable to establish a more moderate set of benchmarks. For example, Williams, Holt, & Boehm-Davis (1997) propose a range between .7 and .8 for an Agreement Index benchmark. For airline with little or no experience in this area it would prudent to select a goal of .6 to .7 across all items being rated and work diligently to establish that level. From experience with a range of airlines, it is difficult to establish a .8 to .9 level of agreement, especially when working with a complex LOE and real crew behaviors. Over time, an airline can raise its benchmarks as the assessment team develops strategies to maintain good reliability. The proposed benchmarks for the IRR components (Williams et al., 1997) are:
Agreement .70 to .80 Congruency .50 to .80 Consistency .65 to .80 Sensitivity .40 to .60
. Other strategies for meeting an IRR benchmark include removing some items that are causing rating problems.
An airline should also consider several other strategies in meeting a level of reliability prior to the start of LOFT/LOE assessment. If a particular item on the gradesheet is causing problems, and if, after trying to refine the items, that problem persists, consider the importance of that item and whether it could be removed. Sometimes, the refinement or removal of just a few items allows an airline to meet its Agreement Index benchmark. An airline may also have to look at its assessment team and determine whether all the members can be trained to the benchmark within the time required. Additional training may be an option for some team members and, in other cases, some individuals may decide they do not want to be part of the assessment team. Benchmarks give airlines a concrete way of knowing whether they have established the reliability goals.
Providing Instructor/Evaluatorswith Ongoing Feedback
The training department should plan on providing instructors/evaluators with ongoing rating feedback at appropriate times. By maintaining current instructor data, the training department and instructor/evaluators can determine the frequency and exact form of that feedback. In general, if an assessment team continues to meet its benchmarks and there are no indications of rater bias, the feedback can be on a monthly or quarterly basis depending on instructor preference and airline resources to provide the feedback. If the assessment team cannot maintain the benchmarks, then more frequent feedback is necessary.
　
中国航空网 www.aero.cn
航空翻译 www.aviation.cn
本文链接地址：Developing Advanced Crew Resource Management (ACRM) Training: A Training Manual(77)