DOE G 151.1-1 V7-3, Exercise Controller and Evaluator Manual

Exercises DOE G 151.1-1, Volume VII 8-21-97 3-1 3. EXERCISE CONTROLLER AND EVALUATOR MANUAL 3.1 Introduction The Exercise Controller and Evaluator Manual is designed to prepare DOE exercise controllers and evaluators to effectively perform their assigned functions during Emergency Management System exercises. It is organized to provide pertinent information concerning the exercise development, control, and evaluation processes. Specific controller and evaluator responsibilities before, during, and after an exercise are presented in detail. This Manual performs the following functions. ! Augments Volume VII, Chapter 1. ! Details the roles of controllers and evaluators in exercises. ! Explains the materials used by controllers and evaluators. ! Provides techniques on how to effectively perform the controller and evaluator functions. The Manual is structured so that sections can be read independently of, or in conjunction with, one another. Section 3.1 provides an overview. Section 3.2 details controller functions, while Section 3.3 is dedicated to evaluator functions. Controllers and evaluators may only need to review the section that applies to their assigned function(s) during the exercise. However, individuals assigned to both controller and evaluator roles should read the Manual in its entirety. Section 3.4 provides information relative to the exercise evaluation and critique process that is applicable to both controllers and evaluators. 3.1.1 Exercise Organization The exercise organization consists of the Exercise Director, responders, controllers, evaluators, and observers. Each performs specific assignments and roles. The following summarizes an exercise organization. Exercise Director, as the senior exercise official, has primary authority and overall responsibility for the design, development, control, and evaluation of the exercise. Controllers provide direction and control of the exercise. They monitor the sequence of events as they unfold, and are responsible for exercise safety within their span of control. Individual controllers may initiate certain actions in order to ensure the continuity of events described in the exercise scenario. It is their responsibility to ensure that CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-2 responders do not respond in a manner that might jeopardize safety and that responders remain focused on exercise play that demonstrates the exercise objectives. The control organization will vary in number depending on the exercise scope and may include the following controller positions. ! Senior controller is responsible for coordination and oversight of all other controllers. ! Lead controllers may be used to coordinate the activities of several controllers for larger or more complicated exercises that involve a number of response locations and emergency functions. Controller teams may be organized by location, function, or a combination of both depending on the needs of the exercise. However, controller team leaders should have previous experience as an exercise controller before they are selected to lead a team. ! Control cell is a simulation center located away from the responders. It is staffed by experienced controllers (and/or actors) who simulate or role-play non- participating organizations by providing input to responders, via telephone, on behalf of any non-participating individuals, companies, agencies, or Emergency Response Organization (ERO) members that would normally be involved in responding to an emergency. Role-players in a control cell are subject to evaluation of their performance just like any other exercise controller.

! Timeline coordinator, for complex exercises, is responsible for ensuring the exercise timeline remains on schedule—a key factor for proper attainment of exercise objectives. Should exercise play cause deviation from, or a delay in, the timeline, it becomes necessary to use previously prepared contingency materials. The timeline coordinator, typically co-located with the Exercise Director, is responsible for specific tasks or actions from the control cell. The timeline coordinator receives timeline status reports from lead controllers and provides this information to the senior controller and Exercise Director. ! Actors/role-players are controllers who simulate members of non-participating organizations and role play key individuals, such as injured personnel. They may come in face-to-face contact with the responders, functioning semi-independently as media reporters, next-of-kin, or injured personnel. They may be members of a control cell with telephone communication being the only interaction with responders. Evaluators document and evaluate responder performance and the adequacy of facilities and equipment against established emergency plans and exercise evaluation criteria. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-3 Individual evaluators are unbiased, objective, technical or functional experts tasked to evaluate responder performance, facilities, and equipment against emergency plans, implementation procedures, and checklists. The evaluator organization will vary in number depending on the exercise scope and may include the following evaluator positions. ! Senior evaluator is responsible for the coordination of all evaluation functions including preparation of the exercise evaluation report that identifies findings and specifies corrective actions. !! Lead evaluators may be used to coordinate the activities of several evaluators for larger or more complicated exercises that involve a number of response locations and emergency functions. Evaluator teams may be organized by location, function, or a combination of both, depending on the needs of the exercise. Evaluation team leaders are responsible for the coordination of a team of evaluators assigned to particular locations and/or similar response functions. As with the control organizations, evaluation team leaders should be selected on the basis of previous experience and demonstrated ability to successfully perform as an evaluator. Responders, often referred to as “players,” usually comprise the majority of people involved in the exercise. It is their responsibility to take whatever actions are necessary to mitigate the simulated emergency and thus demonstrate the ability to ensure the safety of facility personnel, the public, and the environment, in accordance with established emergency plans. Observers may be present to observe the exercise for either official or educational purposes. The attendance of observers at an exercise, their locations, and rules of conduct should be determined by the Exercise Director. Observers should not interact with responders, contribute information or opinions, or interfere with the exercise in any other way. Observers should direct all questions or comments related to the exercise to the controller for their area or escort, if appropriate. Although they may have prior knowledge of the scenario, observers should be held responsible for withholding that information from the responders.

Video teams and still photographers, considered observers for exercise purposes, may be used to document the exercise. These teams may film, record, and photograph response activities, as long as they do not interfere with exercise play. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-4 3.1.2 Controller and Evaluator Selection Controller and evaluator functions generally should not be combined. Each role has specific responsibilities that require total concentration to be performed effectively. If circumstances require that an individual be assigned to both roles, that individual must have a thorough understanding of controller/evaluator requirements and responsibilities. It is highly recommended that individuals who fill controller positions have extensive emergency preparedness experience and have previously participated in a variety of tabletops, drills, and exercises so that they know what behaviors and actions to expect from responders. Each controller should serve in that capacity for several exercises to provide continuity and consistency. Personnel who serve as controllers should be current on updates and upgrades to the emergency plans and implementation procedures that will be demonstrated at the locations they will control during exercises. Evaluators are selected based on knowledge of, and experience with, the functions they are to evaluate. It is important that they are technically competent to judge the actions of the responders. Wherever possible, their experience should be equal to or greater than that of the responders in their assigned area. The conduct of effective emergency exercises depends on the selection and assignment of top-quality controllers and evaluators. Although these individuals may be drawn from non-participating areas of a response organization, care should be taken to ensure that use of these personnel to support the exercise does not compromise the effectiveness of the response organization. 3.1.3 Role of Controllers Controllers are primarily responsible for ensuring the continuity of the scenario and maintaining safety and security. Controllers play a crucial role throughout the exercise process. Their first and most important function is to maintain exercise safety. They maintain the sequence of events, control the flow of message injects, and are responsible for the overall conduct of the exercise. Controllers are in a unique position to view exercise play, understand the dynamics of an action or activity as it unfolds, and comment on what they observe. Controllers provide scenario information to responders as it is earned and may be tasked to inject approved contingency messages to keep the exercise on track with the scenario and the exercise timeline. Controllers should do the following. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-5 ! Prior to the exercise, review appropriate Emergency Plans, procedures, and documents. ! Prior to the exercise, review appropriate exercise package materials, including the objectives, scenario, messages, Safety and Security Plans, and controller instructions. ! Attend required training and briefing sessions. ! Conduct the exercise by providing applicable scenario information to responders. ! Allow freedom of responder decisions and actions (i.e., free play) to demonstrate exercise objectives and response capabilities. ! Inject approved contingency messages or provide instructions, as needed, to keep the exercise on track with the scenario.

! Preclude responder decisions and control actions that may compromise the safety or security of personnel or the facility. ! Refrain from prompting, in any fashion, the decisions or actions of responders. ! Prevent observers and evaluators from having interaction with responders. ! Be prepared to suspend exercise activities in the immediate area and to use pre- arranged protocols to terminate an exercise. 3.1.4 Role of Evaluators An evaluator's function during the exercise is to observe and document exercise activities and conditions. The evaluation assessment is performed after the exercise is terminated. Evaluators document and evaluate the performance of the responders and the facilities, equipment, and resource documents (e.g., drawings, reference materials, maps) used by the responders. Evaluators are assigned specific locations or responder functions to evaluate. Responder performance must be evaluated against plans and procedures using criteria established prior to the exercise. Evaluators document the responders' performance and attend the critique facilitated by the controllers immediately following the exercise during which the responders discuss their performance. The evaluation, documentation, and critique discussion(s) provide important evidence that substantiate CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-6 exercise findings. The evaluators' report summarizes the overall results of the exercise and provides a comprehensive assessment of the emergency response performance. Evaluators should do the following. ! Review appropriate emergency response plans, procedures, and documents prior to the exercise. ! Prior to the exercise, review appropriate exercise package materials including the objectives, scenario, messages, Safety and Security Plans, and evaluator instructions. ! Attend required training and briefing sessions. ! Observe the performance of the responders during the exercise and document their actions using evaluation modules or checklists. ! Observe the performance of the control organization in controlling and directing the exercise. ! Refrain from interfacing with responders to prevent interrupting or prompting their decisions or actions. ! Evaluate responder performance and the adequacy of procedures, facilities, and equipment based on specific evaluation criteria. ! Document errors and problem areas in the scenario or conduct of the exercise. ! Present their evaluations and recommendations in a formal critique. 3.1.5 Controller and Evaluator Training A formal training program for controllers and evaluators enhances the capability of the Emergency Management Program to maintain a level of consistency in how exercises are managed and response capabilities are evaluated. The training program should include both initial and exercise-specific training to provide controllers and evaluators with the information and direction necessary to perform their duties with confidence. Both generic and exercise-specific training is detailed in Chapter 1 of this Volume. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-7 3.1.6 Keeping Exercise Material Confidential Exercise information should be closely guarded and not discussed with potential responders. Scenario materials should be kept under lock and key at all times, unless being used. All copies of the exercise package should be numbered and assigned while under review to ensure accountability during the review/development period. To ensure that exercise confidentiality is maintained, controllers and evaluators should do the following.

! Be careful of what they say and to whom because it may be overheard. ! Take care in positioning themselves while observing an exercise activity to ensure they do not give away specific information by their actions. ! Ensure responders cannot read their scenario, timeline, notes, inject messages or other sensitive materials before or during an exercise. 3.1.7 Exercise Evaluation by External Organizations Exercises are subject to evaluation by “external” Departmental organizations (i.e., an organizational entity beyond that of the immediate facility/site conducting the exercise), including the cognizant Operations Office and the Headquarters Office of Emergency Management. These evaluations include an evaluation of the manner in which the exercise is controlled and evaluated and an evaluation of performance of the controller and evaluator organizations. 3.2 Controller Activities Controller responsibilities include pre-exercise setup, exercise conduct, and post-exercise activities. These responsibilities are summarized in Figure 3.1. The following section reviews methods of controlling an exercise, as well as specific responsibilities of controllers before, during, and after an exercise. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-8 Timeline Management Managing Safety Controller Functions During the Exercise Process Initial Training Refresher Training Security Access Setup Simulations Communications Check Pre-Exercise Activities Initiating Training Responder Briefings Earned Information Contingency Messages Unplanned Suspensions Restart Exercise Termination Exercise Activities Participant Rosters Self-Critique Sheets Responder Critiques Evaluator Critiques Exercise Evaluation Report Post-Exercise Activities Figure 3.1. Sample Controller Responsibilities 3.2.1 Pre-Exercise Activities Controllers must review the exercise and scenario materials, attend exercise-specific training, perform communication checks, set up simulations, and conduct pre-exercise safety and security checks before an exercise. Controller Package. Controllers should be issued their materials for review prior to the exercise-specific training. The controllers, being experienced personnel, may be tasked to help the scenario developers finalize scenario details. The controller package may consist of part or all of the contents of the exercise package. A cover page should remind the controllers of the confidentiality of the scenario materials. Because some exercise packages are very large documents, controllers may need to reorganize the material so that the information critical to their specific assignment is readily accessible. Controllers should bring their packages to the exercise-specific training sessions and be prepared to discuss any concerns or questions they have about this information. The controller package should include, as a minimum, the following information. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-9 ! Schedule of control activities. ! Control organization and assignments. ! Procedures for reporting within the control organization. ! Suspension and termination procedures. ! Scenario material. - Objectives. - Scenario narrative. - Timeline and Master Scenario Event List (MSEL). - Position-specific messages or injects, including associated data. ! Controller position-specific safety and security instructions.

Exercise-Specific Training. Exercise-specific training is also known as controller and evaluator training, pre-exercise training, scenario training, or the pre-exercise briefing. Regardless of what it is called, the overall objective is to prepare the controllers to safely and effectively control the exercise without compromising the scenario or prompting responder actions. Controllers who do not attend the training should not serve as controllers during the exercise. Day of Exercise Preparations. On the day of the exercise, controllers report to the staging area with their notes, scenario messages, data sheets, controller log forms for recording activities, and any other materials assigned. Controllers should have reviewed their instructions (example in Figure 3.2), and highlighted specific responsibilities and messages that they are responsible for delivering. Setting Up Simulations. The controllers may be required to assist in setting up the simulations that will be used in their control areas. This may include the preparation of smoke generators, positioning of special equipment or vehicles, simulated spills, and injured role players. It may also include the pre-staging of simulations for use later in the exercise or for contingency purposes. All simulations should be checked before reporting “ready.” It should be remembered that realism is second only to safety. Pre-Start Safety Checks. Prior to the beginning of the exercise, pre-start safety checks are conducted. These include the checking of simulations, posting “EXERCISE IN PROGRESS” signs, weapons safety checks, and a final communication check with the lead controller. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-10 The Exercise Director will not start the exercise until notified that all controllers have performed their communications check, verifying that they are at their assigned location and that their safety checks have been completed. The lead controller will give the control organization a time check (synchronizing watches). Large exercises may be started at a predetermined time while smaller exercises may be started by voice command over the communications network. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-11 GENERIC CONTROLLER INSTRUCTIONS (SAMPLE) 1. Review the exercise objectives and controller package for your area of responsibility. 2. Using the Master Scenario Events List, highlight the specific messages for which you are responsible. 3. Be located in the appropriate emergency response facility at least 30 minutes prior to the start of the exercise. If you are not assigned to a specific facility, be in place to meet the responders at least 15 minutes prior to their activation. 4. Obtain or locate necessary communications equipment and test it to ensure satisfactory communication between controllers and the lead controller and/or the timeline coordinator. 5. Wear controller identification, such as the required badge, arm band, or vest. 6. Synchronize your watch with the lead controller to ensure that the exercise timeline and the controller logs are consistent. 7. As instructed, distribute an exercise participant package to specific responders. This may include exercise limitations, meteorology, instructions, and the exercise telephone directory. 8. Do not enter into personal conversations with any exercise responder. 9. Deliver the messages you have been assigned at the time indicated. Caution: If the

information depends on some action to be taken by the responder, do not deliver the message until the responder has earned the information by successfully accomplishing the required action. 10. When you deliver a message, notify the lead controller with the message number and the time delivered. 11. Begin and end all exercise communication over the radio or telephone with the phrase “This is an exercise message.” This precaution is taken so that anyone overhearing the conversation will not inadvertently mistake the exercise play for an actual emergency event. 12. If you are to deliver specific data, deliver it as directed on the message instructions. (Examples: Do not deliver vital signs of an accident victim until the first responder attempts the appropriate actions for obtaining these; do not volunteer radiation readings until the technician has turned on and read the detection instrument.) 13. Record all activities and the time in your controller log. Do not write opinions; rather, write about specific actions. Page 1 of 2 Figure 3.2. Sample Generic Controller Instructions. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-12 GENERIC CONTROLLER INSTRUCTIONS (continued) 14. If responders do not perform as expected and a contingency message is not provided, notify your lead controller immediately and ask for direction. No unplanned simulations should be allowed without the lead controller's approval. This differs from free play, which is action taken by a responder that is appropriate in solving the problem in a unique way. 15. Do not prompt a responder as to what a specific response should be unless a contingency message directs you to do so. Clarify information as long as it does not provide coaching. 16. Ensure that all observers stay out of the exercise activity. If you need assistance, notify your lead controller or security. 17. Do not provide information to the responders regarding scenario event progress or resolution of problems encountered by others. Responders are expected to obtain information through their own resources. 18. You will be notified by your lead controller when the exercise has been terminated. The exercise will be terminated when the Exercise Director, in conjunction with the lead controller, determines that all exercise objectives have been met, or enough time has elapsed for the objectives to have been demonstrated. 19. Pick up copies of responder logs and pertinent documentation prior to the post- exercise debriefing and critique. This information should be given to the lead controller. Coordinate this task with the evaluator in your area. 20. At exercise termination, summarize your notes and prepare for the local area critique. Have the summary ready to turn over to your lead controller. The facility lead controller shall provide this documentation to the area/site lead controller. THANK YOU VERY MUCH FOR YOUR HELP Page 2 of 2 [Source: Exercise Porcupine; Performance Test at Los Alamos National Laboratory 1994] Figure 3.2. Sample Generic Controller Instructions (continued). CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-13 3.2.2 Exercise Activities The control organization plays the crucial role in monitoring the sequence of events, injecting messages, and ensuring the overall safe conduct of the exercise. Monitoring For Safety A controller's primary function is to ensure the safe execution of an exercise. The safety of everyone involved in the exercise, as well as the facility, public, and the environment, is the highest priority.

Controllers are responsible for knowing the limitations and precautions for both safety and security for the exercise and for understanding and using this information to ensure that all participants comply accordingly. The precautions and limitations are provided to the controllers in the exercise Safety Plan and Security Plan and also in exercise-specific training. Such information may include details on physical security (such as facility access control), safety (such as the location of fire doors), information security (such as the location of classifiers to ensure classified information is not divulged), and other privileged instructions. All participants in an exercise are responsible for acting in accordance with appropriate safety plans and are bound by DOE requirements, as well as local laws and restrictions. It is the responsibility of the controller to stop any action that would violate any law or safety protocol. General safety rules applicable to all participants may include the following. ! Comply with all Federal, state, and local legal restrictions. ! Obey all traffic laws when responding to the exercise. ! Wear all personal protective equipment required for the job. ! Do not place yourself or anyone else in an unsafe position. ! Obey the directions of the controller at the scene. ! Preface and end all radio, telephone, and other communications with “This is an exercise.” ! Ensure any weapons being used are empty and on SAFE. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-14 ! Contact a controller in the event of an actual emergency. Methods of Control. Exercise control, in terms of the autonomy and authority of individual controllers, varies depending on the complexity of the simulated events, the number of participating organizations, and the number and experience level of the controllers. Individual controllers may be delegated extensive, limited, or possibly no authority to issue message injects and contingency message injects without the Exercise Director or senior controller direction. The method of control for each exercise should be identified, documented, and the controllers trained as to the level of authority they have for message inject release and subsequent reporting. Presenting Scenario Information. Controllers either initiate simulations or provide the description of the initial conditions to the responders. Controllers should introduce themselves and evaluators in the area and identify the exercise safety and ground rules. Simulations should be identified to the responders. For example, responders might be told that the real meteorological conditions will NOT be used and what the predetermined (“canned”) wind speed and direction are. Generally, it is the controllers' responsibility to set the stage for the event in their assigned area. The controllers depend on directions from the lead controller. Scenario information (including physical evidence and visual cues) should be presented in a realistic manner to the responders as it would be found, measured, or otherwise indicated. After a valid field measurement is taken, the controller should inject the scenario value. For consistency and documentation, controllers should use data and instrument readings provided in their controller packages when giving technical information to responders. Visual cues such as victim moulage, liquids, solids, smoke, and other stage props should be used to make the event appear to the responders as if it were actually occurring. If emergency procedures require the use of protective equipment and clothing, the participants should use the actual protective equipment and clothing during the event.

Controllers should not intercede in exercise play unless it is warranted by safety considerations. Controllers should not prompt by providing information early, providing more information than the responder has earned, or phrasing sentences in a way that would cause the responders to perform an action. ! Realism Versus Simulation Making the exercise as real as safety will permit is one of a controller's prime considerations. The more realistic an exercise is, the less information needs to be provided to responders. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-15 The value of an exercise is to demonstrate response capabilities under simulated emergency conditions. Although it is impossible to predict or measure precisely what response would be under actual emergency conditions, a realistic exercise can provide a good overall picture of the response capabilities of each participating organization. The realistic presentation of information during exercises can create the sense of pressure and stress inherent in actual emergency situations. Controllers should provide information to responders in a form and manner consistent with what would occur in an actual emergency, and present scenario information earned by responders as a result of their actions. Responders should carry out every activity and response action exactly as they would in an actual emergency, such as using emergency equipment and checking instrument and meter readings. However, responders should only walk through or discuss the response actions that would be necessary to restore or realign equipment using panel switches, to avoid changing critical process or plant equipment alignments and parameters. ! Free Play It is the controllers' responsibility to monitor free play, note activities, and intercede when free play exceeds the limits established for the exercise or when safety is jeopardized. Free play is a welcome part of an exercise because it allows responders to provide unique solutions to problems and to respond in ways not foreseen by the exercise planners. In order for free play to be successful, it must fall within prescribed parameters. If the responder actions compromise safety or exceed established limitations, the controller should note the intended action but prevent that action from actually occurring. ! Tracking the Scenario and Responder Actions One of the controllers' most important and difficult tasks is maintaining the timeline for scenario events and tracking responder actions. Controllers should record the time of all significant events associated with their part of the exercise, to include the following. – Time of message delivery. – Contingency message delivery. – Responder key decisions and mitigation actions. – Free play. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-16 – Conversations with responders (or other controllers). Controllers should note the effects of the messages on play and any other unexpected activities, and they should report any discrepancy in scenario progress immediately to the lead controller. Later, controllers from different areas can meet to develop a consolidated timeline of exercise play and discuss how the exercise progressed overall. Maintaining Exercise Pace and Focus. The lead controller, with the assistance of the timeline coordinator, should manage the exercise and ensure that the sequence of events identified in the exercise timeline occurs as close to schedule as possible. Individual controllers should follow the overall exercise timeline in order to keep their respective parts progressing in accordance with the scenario.

! Use of the MSEL The MSEL is one of the primary tools that controllers use to track the progress of the scenario. Evaluators will find that the MSEL is a useful tool for determining whether and when expected responses occur. An MSEL lists all exercise messages and key events in a table that specifies the time the message is expected to be delivered, who delivers it to whom, a message number, and a short description of the message. Some MSELs also contain the responder-expected actions and associated exercise objectives to assist the controllers and evaluators in performing their functions. ! Use of Scenario Messages Controllers use prepared scenario messages (also known as controller injects, cue cards, and data input) included in their exercise package in conjunction with the MSEL. These messages include information on the placement of props, initial conditions, set-up of the area, and placement of observers. Messages that contain information on activities that are dependent on the completion of other activities should also include information on what to do if that initiating activity is not completed. If this information is not readily available, the controller should contact the lead controller. Exercise messages are developed to do the following. – Create situations. – Provide specific instructions and data. – Correspond with activities required by emergency plans, procedures, checklists, etc. – Notify participants of safety or compliance violations. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-17 – Keep the scenario on track. During the exercise, controllers inject messages through one of three methods: voice, hard copy, or face-to-face contact. Voice. Controllers inject oral messages to control progress of the exercise scenario. These messages include descriptive information that simulates an event or condition, or they may initiate a specific activity that will keep the scenario on track. These messages are given by the controller on location, or over the phone or radio by a controller at a different location such as a control cell. At specified times, controllers will contact the appropriate responder(s) and read the prepared event text verbatim. Hard copy. Hard copy messages are designed to simulate electronic messages, memoranda, Material Safety Data Sheets, strip charts, news bulletins, etc. At designated times, controllers will deliver these messages to the appropriate responder. Messages of this type are provided on data cards or sheets of paper with appropriate time-related plant and radiological parameters, real-time data generated by a simulator or computer, or audiovisual presentations of data such as moulage, charts, pictures. Parameters can be posted on appropriate control room panels, on cameras for viewing in several locations, or posted at, or generated by, computer terminals and printers. When hard copy messages are provided, no additional comments are made by the controller. Face-to-face contact. Occasionally, selected controllers may play the role of a senior official, a decision maker, or an outside agency representative. While role playing, the controller interacts face-to-face with participants and provides information or responses to questions in a fashion appropriate for the role he/she is playing.

Messages help direct the progress of the exercise and clarify situations that cannot easily be described. Messages should normally include only information the participants could gain with their own senses (i.e., sight, hearing, smell, touch, and taste) in their location. Emergency situations like fires, tornados, injury of personnel, alarms sounding on control boards, suspected intruders, and radiation monitor alarms can be described in exercise messages. Messages may appear in several different formats based on how they will be used during the exercise. Message formats may include data sheets, charts, pictures, and hard copies of voice messages. Information contained in the message should include the recipient, the time of delivery, the expected responder action, and any CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-18 additional information or directions for the controller. The controller should generally not give the entire hard copy of the message to the responder. Only the information portion of the message should be delivered to the responder. Occasionally, the controller may give a message directly to a responder; for example, such as a note from simulated hostage takers. Messages should not be used to describe situations that participants can and should recognize from facility data. A message such as “temperature is increasing” is usually inappropriate because it prompts responders with information that would not be provided in that form in a real emergency. The controller should understand and be aware of the action that a responder should take in response to a message. Thus, a message with an “actions expected” section may be provided to controllers (not passed to participants) so that they will be aware of what the responders should do. Generally, controllers should provide the information to the responders as they progress through the scenario and “earn” information. To earn information, the responders must act in a manner that would provide them the information in a real event. For example, if a message states that the oil bull's-eye on a pump is empty, then the controller should provide that information when the participant looks at the bull's-eye. If participants follow appropriate steps for obtaining the information, the controller may then provide it. Controllers should only provide earned information and nothing else that would instruct the responders in how to proceed. For simulated events such as fires, controllers should provide the “scene setting” information as responders earn it. For example, in approaching a closed fire door, the controller should tell a team member “the door is hot” only after the team member has actually touched the door. Likewise, the team member would be told “the room is filled with smoke” only after the door has been opened. Information is provided in a sequence replicating the approach to an actual fire. ! Contingency Messages Controllers use contingency messages to force an action by a responder or response organization to keep the exercise on track. Controllers should issue contingency messages in accordance with the pre-established exercise protocol for their use. If responder actions require a contingency message, the controller should contact the responsible lead controller before injecting the message. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-19

Contingency messages provide supplemental symptoms or necessary information specifying existing conditions that will elicit the appropriate decision or response. Contingency messages should begin with an explicit directive such as the following. – “Declare a general emergency for the following reasons....” – “Contact the state at this time to recommend the following action....” – “To keep the exercise on track, order a site assembly at this time.” In some cases, a contingency message may be issued solely to keep the exercise scenario coordinated. An example contingency message would be one preventing operations personnel from beginning a radiological release until a pre-established time so that time-dependent, hard-copy radiological data remain credible for the plant status and conditions being simulated. A negative exercise finding may result if exercise play does not occur as expected, and responder(s) must be provided a contingency message to induce activities that should have occurred without controller intervention. Suspension or Termination of an Exercise The controller's role in suspending, restarting, or terminating an exercise is to ensure that responders clearly understand when these actions have been implemented. Controllers also provide specific instructions to responders covering any requirements or activities they must undertake. In the case of restart, responders may have to "redo" activities because they are critical for responder performance at other locations. The suspension or termination of an exercise is managed through the control organization. Responders are instructed to contact a controller when an unsafe condition exists or when an actual emergency is identified. The exercise package identifies how the control organization will be notified and the procedures for exercise suspension, restart, and/or termination. Figure 3.3 is an example of suspension and termination instructions. ! Suspension Provisions for suspension or premature termination of the exercise for safety reasons are provided in the controller package. If an unidentified or questionable situation arises that may affect the participants, a controller may suspend play and immediately notify the lead controller. Play may be restarted if the situation is resolved. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-20 In the event of a real emergency, it is the controller's responsibility to suspend the exercise in the immediate area for which he/she is responsible and to contact the lead controller. An actual emergency always takes precedence over an exercise. If necessary, the Exercise Director may terminate the exercise so that resources can be devoted to the real emergency. SAMPLE SUSPENSION AND TERMINATION INSTRUCTIONS The exercise is scheduled to begin at 8:00 a.m. Mountain Standard Time (MST) or 10:00 a.m. Eastern Standard Time (EST). No responders should be pre-positioned, and response should be in accordance with established policies and procedures. The exercise is scheduled to run 6 hours with termination at 2:00 p.m. MST/4:00 p.m. EST. Each emergency response facility participating in the exercise should conduct a critique of their involvement immediately following the exercise. All controllers and evaluators are expected to take notes of items identified by the exercise responders. If controllers or evaluators are asked for their impressions of how things went, specific issues or problems should not be discussed.

The exercise may be terminated by the Exercise Director when exercise objectives have either been demonstrated or given an adequate opportunity to be demonstrated. Following consultation with the lead controller and lead exercise evaluator, the Exercise Director will make the announcement concerning exercise termination. If an actual emergency occurs, the exercise may be suspended or terminated at the discretion of the Exercise Director. DOE, Federal, State, Tribal, local, or DOE contractor authorities, depending on the nature of the incident, may recommend termination to the Exercise Director. If the exercise is suspended, the controllers will instruct the responders to safely stop in place. The conditions for restart of the exercise will be determined by the exercise director in consultation with the lead control personnel from the participating organizations. The controllers will be instructed on what conditions are to be set for restart. Figure 3.3. Sample Suspension and Termination Instructions. ! Termination Information concerning the procedures and protocol for terminating an exercise should be included in the controller package and reviewed at the pre-exercise briefing. Termination of the exercise at any time is under the authority of the CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-21 Exercise Director. Upon notification from the Exercise Director or a lead controller, the controllers should immediately announce the termination of the exercise, record the time, and ensure that responder exercise activity ceases. In general, an exercise will be terminated when one of the following conditions is met. – Exercise objectives have been met and the pace of play indicates that major events have been drawn to a logical conclusion. – Enough time has elapsed to allow the objectives to be demonstrated. – An actual emergency occurs. Sometimes, facility locations and offsite organizations have specific objectives (e.g., recovery and reentry) that are not applicable to other exercise locations. If that is the case, the lead controller should determine the extent of play necessary to permit those locations/agencies to meet their objectives. In such cases, termination of an exercise may be staggered for different groups of responders. Premature termination of an exercise presents a problem among response organizations in that it may preclude them from meeting their objectives. To prevent premature termination, the Exercise Director should obtain concurrence from lead controllers that objectives have been demonstrated or sufficient opportunity has been provided for the objectives to be demonstrated prior to terminating the exercise. 3.2.3 Post-Exercise Activities The two primary post-exercise duties for controllers are facilitation of responder “hotwash” critiques immediately following the exercise and participation in the exercise evaluation process. Since the controllers ensured the pace and focus of the exercise, they have unique understanding of who performed response actions when and why and, therefore, typically lead the hotwash critiques. Controllers typically provide input to the critique and evaluation process because of their in-depth familiarity with the exercise activities and responder actions. 3.3 Evaluator Activities Preparation is key to effective evaluation. Persons assigned as evaluators cannot just show up on the day of the exercise. The evaluation may significantly change the way

CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-22 the facility/site responds to an emergency and should be based on the specific Emergency Management Program and specific scenario. 3.3.1 Pre-Exercise Activities Evaluators should be trained and thoroughly prepared for their assigned duties for each exercise. Understanding the scope, exercise objectives, and evaluation criteria, and being familiar with emergency plans and implementation procedures will help ensure that the evaluators can concentrate on observing the actions of the responders. Evaluators have the following three primary responsibilities prior to the exercise. ! Obtain and review all emergency plans, procedures, and checklists for the activities to be evaluated. ! Understand the exercise objectives and know how they relate to the evaluation criteria. ! Attend the pre-exercise training. Additional information about the site or facility being evaluated should be provided to those evaluators who work at another facility or site. This information should include the ERO structure, procedures, notifications, communication systems, Incident Command System, facility walk-downs, and maps. Emergency Plans and Implementation Procedures Evaluators should understand the Emergency Plans and implementation procedures being used by the responders they are evaluating. The evaluation includes an assessment of responder implementation of the plans and procedures, as well as an assessment of the adequacy of these plans and procedures. Emergency Plans and procedures frequently are updated. Failure to review them may result in evaluator errors and skewed evaluation results. The person assigned as lead evaluator should ensure that the evaluators receive the plans and procedures far enough in advance to allow for a thorough review and that evaluators assigned to the exercise know the importance of the plan and procedure review. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-23 Exercise Objectives, Criteria, and Checklist Relationship Evaluators must understand the relationships between the exercise scope, objectives, evaluation criteria and the evaluator checklists prior to the exercise. This relationship provides the evaluator with insight on what has to be done, which items are critical, and how findings should be classified. Exercises test response capabilities according to a set of measurable exercise objectives. Each objective is associated with a standard of performance, or criteria, that must be met to demonstrate that objective. Checklists are derived from these criteria to assist evaluators in measuring performance. The relationships between these elements and how they fit together in the evaluation process are described in the following paragraphs. Generic exercise evaluation criteria is provided in Volume VI, Chapter 3, and should be used to develop exercise-specific evaluation criteria as part of the exercise development process. Exercise-specific criteria are then used by evaluators to determine if an exercise objective has been adequately demonstrated. Figures 3.4 and 3.5 exemplify how evaluation objectives, criteria, and checklist items are tied together in evaluation materials. Though checklists vary from site to site, they should contain the following items. ! Method for recording a chronology or timeline of observed events. ! Standards related to a specific exercise objective. ! References for that standard. ! Criteria associated with the standard. ! Activity-specific criteria (list of activities to look for). ! Method for documenting and commenting on these specific activities.

Evaluation checklists for documenting specific activities may vary from yes-or-no answers for activity-specific criteria to numerical evaluations and/or evaluator comments. Standardized forms may simplify the process of documenting observations and analyzing the combined results. A simple form with brief instructions and space to list identified concerns works well, as does a checklist that rates various activities. Evaluation forms should be as straightforward as possible and should have space to identify the evaluator, location, activity observed, responders observed, and the time and date. Evaluator Package. Evaluators are issued their materials for review prior to the exercise-specific training. The evaluator package is a subset of items provided in the exercise package, as well as specific evaluator instructions and other items. Because the exercise package can be very large, evaluators should reorganize the information so the information critical to their assignment is readily accessible. They should bring the CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-24 package to exercise-specific training and be prepared to discuss any questions about information in their packages. The evaluator package should include the following information. ! Cover letter discussing scenario confidentiality and providing a point of contact. ! Chart or listing of the evaluation organization. ! Plans, procedures, and checklists used by the organization or people evaluated. ! Detailed instructions for the evaluator, including a schedule of events. Objective: Given the decision to activate the EOC, achieve operational status within 1 hour in accordance with emergency procedures. Criteria: A.3 Members of the ERO effectively demonstrate their roles, functions, and use of emergency equipment/facilities as outlined in the approved Emergency Plan and procedures. A.7 Control of operations, monitoring, and repair teams is clearly vested in either a single emergency facility or clearly defined among multiple emergency facilities. References: <Site> Emergency Plan, <Site> EOC Implementing Procedure Checklist: Criterion A.3: 1. The following positions were staffed within 1 hour: ____ Emergency Director ____ Federal Communicator ____ Plume Modeler ____ . . . (per procedures) 2. The Emergency Director: ____ Completed a turnover from the shift supervisor and assumed overall direction and control within 15 minutes of arrival at the EOC ____ . . . (as per procedure) Figure 3.4. Sample Evaluation Form. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-25 F15 INCIDENT COMMAND GROUP EVALUATION CRITERIA OBJECTIVE: Assemble, equip, brief, dispatch, and track offsite monitoring teams. REFERENCE: LANL Emergency Management Plan ERPIP 520 HAZMAT Group Supervisor Memorandum of Understanding between the County of Los Alamos and the Department of Energy, 3/22/93. 1. Los Alamos County or the New Mexico Department of Public Safety MOUs are implemented by request of the County Police Department Shift Commander, County Fire Department Shift Commander, County Fire Department Incident Commander, LANL Duty Emergency Manager, or the NM Department of Public Safety. 2. Monitoring teams are equipped then assembled in the staging area or other area determined by the HAZMAT Group Supervisor. 3. Monitoring teams are briefed and then dispatched. Are their locations plotted on the Incident Commander's status board?

4. Monitoring data and information are transmitted to the HAZMAT Group Supervisor. 5. HAZMAT Group Supervisor evaluates and assesses the information and then sends it to the Incident Control Group and the Emergency Technical Support Center. MET:______ NOT MET:_______ NOT OBSERVED:_______ JUSTIFICATION: ___________________________________________________________________________ ___________________________________________________________________________ [Source: Exercise Porcupine; Performance Test at Los Alamos National Laboratory, 1994] Figure 3.5. Sample Evaluator Checklist. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-26 ! Exercise scope. ! Exercise objectives and evaluation criteria. ! Evaluation checklist(s) and materials. ! Scenario material. ! Site map(s). Instructions For Evaluators. Evaluators are provided with general and specific instructions for supporting pre-exercise setup, exercise conduct, and exercise evaluation (see Figure 3.6). Information provided in evaluator instructions may include the following. ! Location and layout of the facility or function to be evaluated. ! Expected time of responder arrival. ! Logistical information. ! Communication contacts and equipment. ! Applicable plans and implementation procedures. ! Potential problem areas. ! Approved simulations. Each evaluator should review and become familiar with the specific information provided in his/her package. Exercise-Specific Training. Exercise-specific training is conducted prior to the exercise and assists in preparation of the evaluators to perform their functions. It may be combined with the controller training and includes a detailed briefing on exercise activities and the scenario. This training is the time for evaluators to ask questions and to ensure they completely understand their roles and responsibilities. Evaluator questions should be addressed and information clarified so that evaluators feel confident they can effectively perform their assignments. 3.3.2 Conduct of the Exercise During the exercise, an evaluator's primary responsibility is to document observations of responder activities. This includes maintaining a chronology of events and using the checklists to ensure that accomplishment of key actions has been recorded. Evaluators should report to their appropriate staging areas with any notes, evaluation criteria forms, evaluator identification, safety equipment (hard hats, safety glasses), and other evaluation materials. In addition, evaluators need to ensure communication arrangements are adequate and verify that any equipment they will be using is in working order. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-27 SAMPLE EVALUATOR INSTRUCTIONS EOC EVALUATOR ___________<Name> 1. Participate in the evaluator briefing on May 10, 199X, at 11:00 a.m. in the <location>. 2. Report to the visitor center not later than 6:30 a.m. on the morning of the exercise for transportation to the EOC. Bring your evaluator package with you. 3. You will receive your lunch and “evaluator” arm band at the EOC. 4. Check in with the lead evaluator by telephone (X-1234) when you arrive at the EOC. 5. Coordinate with the EOC controller for conduct of the responder critique after the exercise. 6. Report to the <location> at 5:00 p.m. for the evaluator debriefing. Bring your evaluation sheets, checklists, and notes from the critiques. Dinner will be provided. The meeting will not go past 10:00 p.m.

7. Attend the DOE management debriefing at 8:00 a.m. the morning after the exercise. The debriefing will be conducted at the Federal Building, Room 123. Figure 3.6. Sample Evaluator Instructions. During the exercise, evaluators should address any questions or needs for clarification of information to the controllers. They should not interact with responders to preclude prompting of and/or interference with responder performance. Realism versus Simulation. The value of an exercise is its ability to demonstrate response capabilities under simulated conditions. Even though responders know that the events are simulated and that an actual emergency is not occurring, realistic presentation of information can create an atmosphere that parallels that of a real emergency. The exercise should be planned so that events and required responses are as realistic as possible, thereby mimicking the sense of stress inherent in any actual emergency situation. Whenever possible, every activity and response action should be carried out exactly as it would be if the events were real. Controllers should provide information to responders in a form and manner consistent with what would occur during an emergency and as a result of specific events or actions taken by the responders. Responders must earn information CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-28 (e.g., attempt to check instruments, perform meter readings, or take vital signs) before they are given that information. However, they should walk through, rather than actually perform, the response actions to restore or realign equipment using panel switches to avoid changing critical process or facility equipment alignments and parameters. The evaluators should evaluate the methods controllers use in the dissemination of scenario information and the way the responders act on this information. Free Play. During an exercise, responders may interject mitigative actions that are not included in, but can be accommodated by, the scenario. In some cases, the scenario timeline will be modified to accommodate this free play. The controller team is responsible for controlling free play. Evaluators should note any free play activities in the exercise and the actions taken by the responders. Free play may indicate a better understanding of emergency management and response activities by the responders than the exercise planners. Evaluators need to be aware of what is going on between controllers and responders during free play so they can document the actions. Actual equipment and procedural problems during the conduct of an exercise interject a form of free play. Responders' solutions to actual equipment or procedural problems on a real-time basis during the exercise afford a valuable opportunity to evaluate the conduct and training of the responders. Controllers should allow responders to solve such problems unless safety is compromised or exercise limitations are exceeded. During the exercise, the evaluators' primary duty is documenting responder performance. After the exercise, that data will be used to determine whether the exercise objectives were demonstrated. Scenario Confidentiality. Scenario information should be closely guarded to ensure its confidentiality. If responders are aware of the scenario beforehand, it will skew any assessment of the emergency response capabilities. The following are some guidelines for evaluators.

! Evaluators should be careful of what they say and to whom because it may be overheard. ! Evaluators should be careful when positioning themselves to observe an activity to ensure they do not give away information by their actions. ! Evaluators should ensure that no one can see their scenario material or comments. They should never lay their scenarios, notes, or messages in a location where responders can read them. CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-29 Documenting the Exercise Evaluators observe and document responder activities during the exercise. It is essential that evaluators keep accurate records and notes because these will form the basis for evaluation of performance. The value of exercise evaluation is the ability to provide constructive feedback (positive or negative) to improve and enhance the effectiveness of an organizations's response to emergencies. Accurate and detailed documentation is critical to facilitate a full record of all the events in an exercise and in understanding responder actions. Evaluators document the exercise by maintaining a chronology of important events, decisions, and actions in their area. Evaluators should document key activities and those which require a timely response for later evaluation. A list of these important events is included as part of the exercise package - in the timeline or MSEL. Highlighting or noting those events which occur in an evaluator's assigned locations is an effective way to track responder activity. Evaluators should review their chronologies and notes immediately following termination to ensure an accurate reconstruction of events and activities for discussion at critique and evaluation sessions. Evaluation materials, as well as critique notes and forms, become part of the exercise documentation. Checklists and evaluation forms should be completed as thoroughly and accurately as possible. Evaluation Basics. Experienced evaluators use the following techniques for effective observation and evaluation. ! Use checklists to confirm that exercise objectives are met. ! Take detailed notes concerning significant activities observed, including the time of occurrence. ! Divide responsibilities, when more than one evaluator is assigned a facility/area, to ensure detailed observation of responder activities. ! Stay in close proximity to responder decision-makers. ! Focus on critical activities (e.g., dose assessment decisions, protective action decisions, command and control issues, and other similar activities). Although numerous events may occur simultaneously, evaluators do not need to record all of the action. Knowing which events are important eliminates superfluous information and CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-30 provides the kind of data most useful for exercise evaluation. Important events which evaluators should record include the following. ! Initiating scenario events (including when responders first detect abnormal conditions). ! Emergency facility activation and staffing completion. ! Actions of responders to the scenario. ! Key decisions made by managers and the time the decision was made. ! Deviations from plans and implementation procedures. ! Times when mitigating actions were completed. Placement and Monitoring. Evaluators should be located so that they can observe responder actions and hear conversations without interfering with those activities. Certain conditions may warrant more than one evaluator being located in a setting or area.

What to Look For. Individuals preparing the exercise evaluation report will analyze the results provided by all evaluators to achieve an integrated evaluation of response capabilities. Their analysis will focus on the measures taken to mitigate the simulated emergency, the timing of key events, decisions made, and actions taken. Potential problem areas to watch for include the following. ! Lack of timeliness in mitigative actions. ! Ineffective communication among responders and organizations. ! Inadequate direction and coordination of field activities. ! Inability to monitor and assess scenario events. ! Ineffective command and control at the scene or response facility. ! Control problems that hinder conduct of the exercise. ! Responder deviations from plans and implementation procedures. ! Unclear plans or procedures that hinder responder efforts. ! Facility or equipment shortcomings that hinder responder efforts. Evaluator Do's and Don'ts. Evaluators should know that scenario data and conditions cannot be changed without obtaining the permission of the Exercise Director or other designated persons. Evaluators should not interfere with a responder's action. Responders should be free to make their own decisions and should act on those decisions without interference. Listed below are reminder do's and don'ts for evaluators. Do's: CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-31 ! Be familiar with other controllers and evaluators. ! Remember that there may be two time frames, a scenario time and a real time. Scenario time may compress events so that several days are played in a few hours. ! Note any communications passed between time zones. ! Identify the participants by title and function. ! Be easily identifiable. Wear the prescribed identifier (e.g., arm band, shirt, or name tag). ! Position yourself to maximize your effectiveness. ! Locate the telephone or radio (for field teams) you will use and know how to use it. ! Be sure you understand the scenario. Know precisely what level of simulation is required and acceptable. ! Work with the other evaluators. Make sure they are reasonably aware of your actions and those of the responders. ! Make notes on responder's strengths and weaknesses related to the activities, as well as areas for improvement. Use critique sheets. ! Attend the post exercise critique session to provide your comments (if appropriate) and recommendations to the Exercise Director. ! Complete evaluation forms as soon as possible following termination of the exercise, while details are still clear in your mind. Identify your observations detailing areas of strengths, deficiencies, weaknesses, or improvement items. Don'ts: ! Don't leave your post at key times. ! Don't ever prompt a responder! ! Don't get in the way. ! Don't answer questions from responders; refer them to the controller. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-32 Termination of the Exercise. Upon notification from the Exercise Director or the senior controller, controllers will announce the termination of the exercise. Evaluators should note the time and circumstances associated with the termination. 3.3.3 Post-Exercise Activities

The primary post-exercise duties for evaluators are documentation of responder hotwash critiques immediately following the exercise, participation in the exercise organization critiques, and the exercise evaluation and report-writing processes. Evaluators will work with other members of the exercise organization during the evaluation process to “complete the picture” of responder actions and assess whether objectives were met, and what improvements or corrective actions are needed. These evaluation activities are covered in Section 3.4. Specific instructions, guidelines, and schedules for evaluators will be in their evaluator packages. 3.4 Joint Evaluation Activities Controllers and evaluators participate in the exercise evaluation process but to different degrees. While the bulk of controller activity occurs during exercise conduct, evaluators perform the bulk of their functions after the exercise is finished. Both groups make significant contributions to the evaluation process. 3.4.1 Evaluation Input In addition to evaluator observations and documentation, the following sources of information may be used to evaluate the exercise. ! Self-critique forms. ! Exercise critique comments. ! Exercise evaluation materials completed by controllers. ! Findings contained in the post-exercise reports submitted by participating agencies. 3.4.2 Critique of the Exercise A series of formal critiques is conducted after the exercise to provide participants (responders, controllers, and evaluators) the opportunity to identify and discuss findings (both positive and negative). Responder "Hotwash"/Critique. These critiques occur immediately after the exercise and are facilitated by the controller and evaluator team at each location. The purpose of these critiques is to provide a forum for constructive feedback on the exercise by the CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-33 responders. The identification of both positive and negative criticism provides a starting point for improving emergency response capabilities. This is a unique opportunity for responders to discuss their responses and their own perspectives on the activities and events. Controllers may discuss significant observations. Evaluators should remain silent and document the observations and feedback from the responders. Evaluation Critique. This critique session generally occurs the day following the exercise and includes participation by all controllers and evaluators. This critique should provide the forum for discussion and correlation of individual observations, the formulation of exercise findings, determination of objectives demonstrated, and determination of overall exercise performance. Recommendations for corrective and improvement actions should be addressed. Senior Management Critique. A senior management critique should be attended by key participants, including manager-level responders, the exercise director, the lead controller(s) and the lead evaluator(s). The overall exercise performance, significant observations, findings, and preliminary corrective and improvement actions should be addressed. Responder Hotwash/Critique. Controllers manage this critique. Generally, the controllers facilitate the exercise critique while the evaluators take notes. It is important that the responders' input be included so an effective and complete evaluation is accomplished and the significance of responder actions is understood. It may be necessary to hold several critiques to include all of the participants.

The critique should be performed while exercise activities are still fresh in the minds of the responders, controllers, and evaluators. Responders may identify any weaknesses, shortfalls, or improvement items. They evaluate their plans, procedures, and task checklists for specific response organization positions, equipment and supplies, facility layout, and performance. For smaller exercises, the facility director (exercise responder) often conducts the critique, but it may also be under the direction of the facility lead controller. Controllers should answer questions on the timeline and scenario. This session can also be used to clarify and verify any information on which there were questions. Responders usually have a basic understanding and evaluation of their job performance during the exercise. Controllers and evaluators should not provide the responders with details of any deficiencies or findings during this critique. Controller input should be limited to feedback concerning the actual event scenario, as opposed to the outcome of exercise. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-34 Responders should be reminded that all controller/evaluator observations are preliminary and may be revised based on information from other evaluators. If an evaluator or controller did not observe specific aspects of an organization's performance, the exercise responders may be asked to comment. Since it is critical that the evaluators not prompt or coach responders during the exercise, the evaluator should raise all questions of this nature through the controller after the exercise activities have been completed. These aspects should be indicated in the evaluation as being provided by responders. A responder self-critique form can be used for documenting responder information about the exercise. These are normally passed out by the controller just before the critique. They should be collected after the critiques along with all attendance or participation rosters. Controllers should emphasize to responders that the self-critique forms provide the opportunity to candidly comment on emergency response activities and effectiveness of the exercise. Evaluation Critique. Formal critique sessions are usually several hours in length and address, at a minimum, the following elements. ! Reconstruction and review of scenario events and shortcomings in the scenario or exercise conduct. ! A comparison of anticipated versus actual responder activities. ! An assessment of performance based on objectives, criteria, plans, and procedures. ! An assessment of the adequacy of plans and procedures. ! An assessment of the adequacy of facilities, equipment, and communications. The first part of the critique is devoted to reconstruction of scenario events and response activities. Timelines should be reproduced for major evaluations, such as the troubleshooting and restoration of a needed piece of vital equipment. At this time, evaluators will organize and consolidate their documented observations. The controllers will provide input to the evaluators' documentation. After this initial documentation is complete, the lead evaluator for the exercise will facilitate a review of the events (using the timeline) to document the interactions between response organizations. This is generally time-consuming, but it provides the information CANCELE D Exercises DOE G 151.1-1, Volume VII 8-21-97 3-35

required to check the communication process among all response organizations. This results in a consolidated exercise timeline of events that actually occurred. After this process is completed, the individual evaluators should have sufficient information available for determining whether the responders demonstrated the exercise objectives. Although the evaluation is primarily concerned with the exercise objectives, collateral observations of events that were performed which require mention (positive and negative) are documented. Each evaluator should develop a rationale for the evaluation from his/her respective point of view. This assists the evaluation report writer in assessing conflicting information that may occur from different sources. When the exercise demonstration is substantially at variance with what was expected, the evaluator should describe this in enough detail to provide a sense of what occurred. The critique should end with a discussion of the preliminary results of responder performance for each exercise objective. Evaluator notes and materials should be collected at the conclusion of this session. 3.4.3 Exercise Report An exercise report is prepared by the evaluator team to document evaluation of overall exercise performance. This report is the responsibility of the lead evaluator. Information from the formal evaluation and critique process provides the material necessary to generate the exercise report. Additional exercise reports may also be prepared by any external Departmental organizations that evaluated the exercise. The exercise report covers the schedule, scenario, participants' activities, observations, and recommendations for corrective actions. The report considers the observations and evaluations made by the evaluators, controllers, responders (self-critiques), and other participating organizations. The exercise report may contain the following. ! A narrative summary with introductory and general statements noting exercise scope, purpose, objectives, participants, and an overall performance rating of the exercise. ! Detailed findings for each objective, including positive and negative comments regarding the effectiveness of emergency planning and preparedness elements. ! Recommendations for correcting negative findings. CANCELE D DOE G 151.1-1, Volume VII Exercises 8-21-97 3-36 Once the report has been drafted, the evaluators should review the report for accuracy. Evaluator findings will in turn be reviewed to ensure responders were measured against the evaluated organization's plans and procedures. Because perceptions differ, the report writers may find it necessary to adjust or “level” various findings to achieve standardization and consistency within the report. The leveling of any findings should be approved by the lead evaluator. CANCELE D