To view this page ensure that Adobe Flash Player version 9.0.124 or greater is installed.
Providing Instructor/Evaluators with Immediate Feedback
Instructors/evaluators should be provided with accurate and immediate rating feedback from the start of their assessment training through the standardization sessions. The first rating sessions may take place in a larger group using taped pilot performance to present the crew behavior that must be rated. The group results may be presented via a spreadsheet and charts showing both individual and group data along with appropriate benchmarks or standards the group is trying to meet. This feedback is an essential part of IRR training and should take place as soon as possible after the ratings are made.
As assessment training progresses, it is advisable to work in smaller groups with fleet-specific performance as the object of the ratings. One reason for the smaller group is that it allows the individuals to start working as a team. This is something that can be done with 5 to 15 instructor/evaluators, but is much more difficult to achieve with substantially larger groups. Small group feedback should be provided on the agreement for each item being rated along with the average agreement for all items. In addition, small group feedback should include the systematic differences, average congruency, and average consistency.
When working with small groups and assessment teams, individual feedback is also important. A smaller group permits greater attention to be paid to individual feedback. Individual feedback includes systematic individual differences, individual congruency with the group’s distribution, individual consistency with other raters, and sensitivity to small performance
. ACRM assessment requires ongoing training and standardization.
differences. Discussion should focus on the individual and group problems for each of these measures.
Using this feedback approach over several rating sessions and providing feedback before going on to the next session can reduce the divergence found in many raters, and the measures allow the instructors to concentrate on the quantitative components of this process. After individual and group feedback is provided and explained, instructor/evaluators should be encouraged to develop new rating rules and strategies. This cycle of practice, feedback, and discussion allows the participants to improve their reliability, and should continue until corporate IRR benchmarks have been met. After that, the methods are used periodically in the standardization sessions.
One type of feedback that can be helpful when some of the instructor/evaluators depart from group norms is to provide individuals or a small group with Congruency graphs. In the following two examples, the first graph shows a rater who closely mirrors the group averages. With a Congruency Index of .97, this instructor has rated CRM the way the rest of the group is rating with about 80% of the ratings as "3" and about 15% as "4."
中国航空网 www.aero.cn
航空翻译 www.aviation.cn
本文链接地址：Developing Advanced Crew Resource Management (ACRM) Training: A Training Manual(51)
