DOE G 414.1-1A, Management Assessment and Independent Assessment Guide

NOT MEASUREMENT SENSITIVE DOE G 414.1-1A 5-31-01 MANAGEMENT ASSESSMENT AND INDEPENDENT ASSESSMENT GUIDE FOR USE WITH 10 CFR, Part 830, Subpart A, and DOE O 414.1A, Quality Assurance; DOE P 450.4, Safety Management System Policy; and DOE P 450.5, Line ES&H Oversight Policy [This Guide describes suggested, nonmandatory approaches for meeting requirements. Guides are not requirements documents and should not be construed as requirements in any audit or appraisal for compliance with the parent Policy, Order, Notice, or Manual.] U.S. Department of Energy Washington, D.C. 20585 DISTRIBUTION: INITIATED BY: All Departmental Elements Office of Environment, Safety and Health CANCELE D CANCELE D DOE G 414.1-1A iii 5-31-01 CONTENTS FOREWORD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii ACKNOWLEDGMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2. APPLICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.1 Assessment Program Expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.2 Assessment Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3.3 Graded Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4. GENERAL INFORMATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.1 Purpose of Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.2 Types of Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.3 Organizational Activity Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.3.1 Process Level Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.3.2 System Level Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.3.3 Program Level Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 4.4 Assessing for Compliance, Effectiveness, and Performance . . . . . . . . . . . . . . . . . . . . . 8 4.4.1 Compliance Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4.4.2 Effectiveness Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4.4.3 Performance-Based Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

5. GUIDELINES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5.1 Assessment Personnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5.2 Assessment Program Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.2.1 Assessment Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5.2.2 Management Assessment Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.2.3 Independent Assessment Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.2.4 Planning Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.3 Assessment Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.4 Assessment Agendas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.5 Performance Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.6 Assessment Planning Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.7 Independent Assessment Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.7.1 Preassessment Meetings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.7.2 The Entrance Meeting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 CANCELE D CANCELE D iv DOE G 414.1-1A 5-31-01 CONTENTS (continued) 5.7.3 Performing Independent Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.7.4 Independent Assessment Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.7.5 The Exit Meeting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.7.6 Assessment Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.7.7 Releasing and Responding to Assessment Reports . . . . . . . . . . . . . . . . . . . 21 5.7.8 Corrective Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.7.9 Follow-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.8 Management Assessment Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.8.1 Defining the System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5.8.2 Assessment Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5.8.3 Performing Management Assessments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.8.4 Assessment Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.8.5 Follow-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 5.8.6 Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 APPENDIX A—CONSENSUS STANDARDS AND REFERENCES . . . . . . . . . . . . . . . . . . . . . A-1

APPENDIX B—ASSESSMENT FUNCTIONAL AREAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1 APPENDIX C—TOOLS FOR ASSESSMENT PLANNING AND CONDUCT . . . . . . . . . . . . . C-1 APPENDIX D—INDEPENDENT ASSESSMENT PLANNING . . . . . . . . . . . . . . . . . . . . . . . . . D-1 CANCELE D CANCELE D DOE G 414.1-1A v (and vi) 5-31-01 FOREWORD This Guide is approved for use by the Department of Energy (DOE) Office of Environment, Safety and Health. It is intended and available for use by all DOE/National Nuclear Security Administration components and contractors. Suggestions for improving this Guide are welcome and should be sent to Gustave E. Danielson, Jr. EH-53/270CC U.S. Department of Energy 19901 Germantown Road Germantown, MD 20874-1290 Phone: (301) 903-2954 Fax: (301) 903-6172 e:mail: bud.danielson@eh.doe.gov DOE Guides provide supplemental information for fulfilling requirements contained in rules, Orders, Notices, and regulatory standards. Guides are also used to identify Government and non- Government standards and methods DOE finds acceptable for implementing the Department’s requirements. Guides are not substitutes for requirements nor do they replace technical standards, which are used to describe established practices and procedures for implementing requirements. CANCELE D CANCELE D DOE G 414.1-1A vii (and viii) 5-31-01 BACKGROUND Since the1994 revision of this Guide, assessment practices have evolved. This revision reflects current assessment practices, international standards, and changes in Department of Energy (DOE) expectations, including the following: • feedback and improvement requirements of DOE P 450.4, Safety Management System Policy; • new requirements in DOE O 414.1A, Quality Assurance, for corrective action plans in response to independent oversight assessments; • assessment requirements of 10 CFR 835; • requirements in the DOE implementation plan for Defense Nuclear Safety Board Recommendation 98-1; • new and revised international management system standards [e.g., International Organization for Standardization (ISO) 9001 and ISO 14001] and the integrated environmental and quality audit standard (ISO CD.3 19011); and • DOE P 450.5, Line Environment, Safety and Health Oversight. The Assistant Secretary for Environment, Safety and Health (ES&H) established a team to prepare the initial version of this Guide (published August 1996). The team identified all assessment requirements in the DOE ES&H Orders that were effective as of November 1994 and determined that 10 CFR 830, Subpart A, and DOE Order 5700.6C (superseded by DOE O 414.1) contained adequate assessment requirements. The Assistant Secretary for ES&H approved a significant reduction in prescriptive assessment requirements. The team recommended a guide for assessment be issued to convey helpful information from canceled Orders and current trends in assessment methodology. The team’s work yielded the following positive outcomes. • More than 400 requirements, rooted in more than 40 Orders, were replaced by the 2 found in 10 CFR 830, Subpart A, and DOE Order 5700.6C (now DOE O 414.1). • Factors that contribute to “stovepipe” assessment programs were deleted. • Flexibility to plan and implement assessment programs that provide value to an organization was increased. • Ability to respond to customer needs was greatly enhanced. • Redundant and excessive assessments were no longer mandated or implied.

• Ability to focus assessment resources on activities that pose the greatest risks and stand to benefit the most from improvement opportunities was enhanced. CANCELE D CANCELE D DOE G 414.1-1A ix (and x) 5-31-01 ACKNOWLEDGMENTS The Department wishes to acknowledge and thank the following team members and additional contributors. Without their expertise and dedication, the effort could not have succeeded. Process Improvement Team Members: Gustave E. Danielson, Jr. DOE, Office of Nuclear and Facility Safety Policy Keith Rademacher, Team Leader, Foster Wheeler Environmental Corp.—Hanford Margy Beckmeyer Westinghouse Savannah River Company David Brown DOE, Richland Operations Office Ken Gidlow Dyn McDermott Petroleum Operations Company Strategic Petroleum Reserve John Gran Westinghouse TRU Solutions LLC Waste Isolation Pilot Project Judith Malsbury Princeton Plasma Physics Laboratory Chuck Moseley BWXT Y-12, L.L.C. Y-12 National Security Complex Denise Viator Oak Ridge Institute for Science and Education Additional Contributors: Paul Chimah DOE, Albuquerque Operations Office Amy Ecclesine University of California Los Alamos National Laboratory Gene Langston Science Applications International Corporation John Palmer University of California Lawrence Livermore National Laboratory Donald White Waste Management Federal Services CANCELE D CANCELE D DOE G 414.1-1A 1 5-31-01 1. INTRODUCTION The Department of Energy (DOE) and its contractors are required to perform management and independent assessments [ Title 10, Code of Federal Regulations (CFR), Part 830, Subpart A, “Quality Assurance Requirements,” and DOE O 414.1A, Quality Assurance]. DOE P 450.4, Safety Management System Policy, and the Federal Acquisition Regulation, 48 CFR 970.5223-1, also require assessments as an element of the feedback and improvement safety function. Also, DOE P 450.5, Line Environment, Safety and Health Oversight, relies upon contractor (self-) assessment programs to reduce DOE oversight. This Guide gives information on establishing processes and performing effective assessments in support of these Policies, regulations, and Orders. (Appendix A contains a list of consensus standards and other references that support and supplement the guidance in this document.) Assessments integrated with the management system add value to products and services by providing feedback and linking the management and conduct of work to meaningful improvement actions. To enable management to take such actions, an assessment program should embody the following principles. • Managers are involved in the assessment process to ensure results contribute to improved performance of the programs, systems, and work processes. • Managers receive timely, objective feedback from assessments on the effectiveness of policies, requirements, standards, processes, and procedures, including evaluations of whether the organization complies with them. • Assessment process coordination and integration is maximized and disruption of the assessed work is minimized. • Organizational culture is one of continuous quality improvement, and assessments are accepted as contributors to the improvement culture. • Management’s goal to protect people and the environment from harm is supported by assessment results. • Quality problems (including safety issues) are identified for resolution by management. • Management takes timely actions to resolve quality problems.

This guide has been developed to assist DOE and contractor line management and personnel, assessors, and others involved in the assessment process in understanding the philosophy, requirements, expectations, and benefits of a comprehensive assessment program. CANCELE D CANCELE D 2 DOE G 414.1-1A 5-31-01 2. APPLICATION All DOE products/services (and the programs, systems, and processes that deliver them) can be assessed over their entire life cycles using this Guide. Environment, safety, and health (ES&H), radiological safety, and safeguards and security activities are also considered products/services and are subject to assessment (see appendix B). Technical standards and methods that DOE finds acceptable for meeting the requirements of 10 CFR 830, Subpart A, and DOE O 414.1A, DOE P 450.4, and DOE P 450.5 are referenced in this Guide. This Guide provides expanded detail on the assessment criteria discussion in DOE G 414.1-2, Quality Assurance Management System Guide. The Quality Assurance Management System Guide also describes the relationship of quality assurance (QA) criteria and Safety Management System (SMS) requirements. The Integrated Safety Management System Guide, DOE G 450.4-1B (vol. 2, appendixes D and G), describes the role of assessment in the feedback and improvement safety management function (similar discussions are included for applying the other eight QA criteria to SMS implementation). The Department’s line managers fulfill their safety responsibilities in part through line management ES&H oversight and have unfettered access to information and facilities in accordance with safety and security requirements. The contractor’s line managers fulfill their safety responsibilities in part through the implementation of their self-assessment programs. Contractors are responsible for establishing robust, rigorous, and credible ES&H assessment programs integrated with their SMSs (DOE P 450.5). Assessment programs conducted in accordance with this Guide and appropriately adopted standards will satisfy the requirements of 10 CFR 830, Subpart A, and DOE O 414.1A, DOE P 450.4, and DOE P 450.5. (Alternative methods may be acceptable to DOE if the methods are demonstrated to achieve an adequate level of safety and quality.) This Guide also provides a basis for determining the adequacy of QA programs and integrated SMS descriptions (DOE P 450.4 and DOE P 450.5) prepared in response to requirements in the previously noted Policies, Orders, and regulations. CANCELE D CANCELE D DOE G 414.1-1A 3 5-31-01 3. BACKGROUND 3.1 Assessment Program Expectations The development of an effective assessment and safety management program must focus on achieving DOE expectations, including the following. • A documented assessment program, defining the systems that will be used to plan, perform, and follow up on assessments, is in place. • Responsibilities for both performing and responding to assessments are defined. • Management at all levels is responsive to identified issues, regardless of how they are identified. • Actions are taken promptly to correct identified problems and prevent recurrence. • Information can be independently verified. • Feedback is solicited from a variety of sources (e.g. management, workers, independent evaluations, customers). • Measurable organization goals and objectives have been identified and progress toward those goals and objectives can be demonstrated. 3.2 Assessment Benefits

The success of an organization depends upon the extent to which its products and services satisfy customer requirements and expectations. Each member of an organization is responsible for customer satisfaction. The quality program described in 10 CFR 830, Subpart A, and DOE O 414.1A (also referred to as “the rule” and “the Order” in this Guide) provides a results-oriented management system that focuses on the customer’s requirements and expectations and embraces continuous improvement. The assessment component of this management system builds confidence that organizations can meet customer expectations. Assessments also provide objective evidence of those areas where improvement is needed to achieve organizational goals and objectives. Effective internal assessments prepare an organization for third-party, assessments performed by external governmental and nongovernmental bodies. Third-party assessors typically assess for performance to and/or conformance with national and international standards (i.e., conformity assessment). Third-party assessors are neither direct suppliers nor direct customers of the organization. The organization contracts for the conformity assessment service. Voluntary third-party conformity assessment bodies include quality/environmental management system registrars, laboratory accreditors, and product certifiers. By contracting with one of these assessment bodies, an organization can have its products; laboratory; and/or QA, health and safety, CANCELE D CANCELE D 4 DOE G 414.1-1A 5-31-01 and/or environmental programs registered/certified as compliant with various national or international standards. DOE contractors can participate in the Voluntary Protection Program and attain recognition for excellence in safety and health management. In addition, various national and state quality awards (e.g., the Presidential Award for Quality and the Malcolm Baldrige National Quality Award) use comprehensive assessments that focus on integrated management systems and customer service. All of these assessments provide evaluations of the management systems and implementation. Regulatory or oversight bodies such as the Nuclear Regulatory Commission, the Environmental Protection Agency, or the Occupational Safety and Health Administration measure compliance with regulatory requirements, standards, and related commitments (involuntary, third-party assessments). The benefit of an involuntary third-party assessment is the confirmation of compliance with regulatory requirements or the identification of noncompliance. Regardless of their main purpose, third-party assessors have at least one common interest: determining whether an organization has established and implemented an effective assessment process. In the regulatory arena, an effective assessment process, coupled with prompt improvements and corrective actions by management, may be considered a mitigating factor in determining civil and criminal enforcement penalties and in turn may yield reduced monetary fines and criminal sentences. In the voluntary arena, an effective assessment process can reduce the time and frequency of third-party assessments. This translates to lower conformity assessment service costs for the organization. 3.3 Graded Approach This Guide and the technical standards referenced herein should be applied using a graded approach. Items, services, or programs that contribute the greatest risk to quality, safety, and mission are assessed with the greatest rigor and frequency.

CANCELE D CANCELE D DOE G 414.1-1A 5 5-31-01 “Criterion 9—Assessment/Management Assessment. Ensure managers assess their management processes and identify and correct problems that hinder the organization from achieving its objectives.” [10 CFR 830.122(i)] 4. GENERAL INFORMATION 4.1 Purpose of Assessment Establishing and implementing an effective assessment program is an integral part of every management system. Assessment is an important feedback mechanism that provides management with information to evaluate and improve any aspect of an organization, for example: • organizational progress in reaching strategic goals and objectives; • adequacy and implementation of management programs for mission achievement; • performance capability of SMSs; • products and service quality; and • regulatory and contractual compliance. Simply stated, an assessment is an opportunity to • identify the gaps between where you are and where you want to be, • identify the reasons for the gaps, • identify the actions that will be taken to close the gaps (corrective actions), • close the gaps between where you are and where you want to be, and • verify that corrective actions have been effective and lasting. 4.2 Types of Assessment The Department’s QA rule and Order establish distinct requirements for two types of assessment: (1) management assessments and (2) independent assessments. DOE P 450.5 refers to contractor self-assessment programs that include line and independent evaluations. In this context, self- assessment is simply the assessments that a contractor conducts on its own ES&H performance. Management and independent assessments performed in accordance with this Guide will satisfy the requirements of DOE P 450.5; however, contractors must clearly describe how their self-assessment programs satisfy the requirement for independent assessment and/or the requirement for management assessment. Assessments are tools for improvement. Management and independent assessments may be performed on the same functions or organizations; however, each has a specific focus, defined by the QA rule and Order and described below. CANCELE D CANCELE D 6 DOE G 414.1-1A 5-31-01 “Criterion 10—Assessment/Independent Assessment. (1) Plan and conduct independent assessments to measure item and service quality, to measure the adequacy of work performance, and to promote improvement. (2) Establish sufficient authority, and freedom from line management, for the group performing independent assessments. (3) Ensure persons who perform independent assessments are technically qualified and knowledgeable in the areas to be assessed.” [10 CFR 830.122(j)] Managers must perform management assessments to comply with the rule and Order. Management Assessments look at the total picture: how well the management system meets the customer’s requirements; the expectations for safely performing work; and the organizational mission, goals, and objectives. The emphasis of management assessment is on management issues that affect performance and related processes such as strategic planning, personnel qualification and training, staffing and skills mix, communication, and cost control; organizational interfaces; and mission objectives. The purpose of this type of assessment is to identify the management aspects of performance and make improvements. Management assessment is an introspective self-analysis to determine whether the management infrastructure is properly focused on achieving desired results. Typically, management assessments are performed at a greater frequency than independent assessments and cover a broader spectrum.

Independent assessments evaluate the performance of work processes with regard to requirements, compliance, and expectations for safely performing the work and achieving the goals of the organization. The focus of independent assessments should be the items and services produced and their associated processes. The purpose is to improve product/service performance and process effectiveness. Independence is defined as not having direct responsibility for the work being assessed. Independent assessments typically are performed by personnel from organizations or work units outside the one being assessed. Thus, management receives an objective view of the assessed activity. Independent assessments are typically performed less frequently than management assessments but go into greater depth. Management is responsible for developing and implementing a coherent plan that balances management and independent assessments and other forms of feedback and improvement to satisfy the requirements of the rule and Order. 4.3 Organizational Activity Levels To shape a comprehensive assessment program that optimizes the application of each assessment type, it may be helpful to visualize the organization as having three interlinked levels of activity (figure 1). For this discussion, these levels will be referred to as “process,” “system,” and “program.” A process is a collection of steps or actions that yield some intermediate outcome; a CANCELE D CANCELE D DOE G 414.1-1A 7 5-31-01 Figure 1. Assessment triangle. system is made up of two or more processes that may operate independently or interdependently and may yield a complete product or service. A program is the most complex level and consists of multiple, interdependent systems that often require many interfaces to provide the desired product or service. Management and independent assessments can be applied at all three levels but will examine different aspects of them. 4.3.1 Process Level Assessments Process level assessments involve examination of work controls and verification that they are being implemented effectively. This level of assessment is critical for ensuring that the worker, the public, and the environment are protected from harm. Process level assessments should also assess the effectiveness of the processes from a quality and customer satisfaction perspective. 4.3.2 System Level Assessments System level assessments focus on whether appropriate leadership and support systems are provided to enable the implementation of work processes. These assessments are performed to ensure human and material resources are being properly used to achieve an organization’s mission and objectives. This level of assessment may range from informal daily oversight of performance to formal periodic evaluations using established protocols. 4.3.3 Program Level Assessments Program level assessments are used to determine whether overall organizational programs are properly established and implemented. They are appropriate for evaluating complex organizations from several perspectives; consequently, program assessments usually examine the integration of the many systems designed to achieve organizational goals and customer expectations (with an emphasis on ES&H factors). The following paragraphs demonstrate how the different types and levels of assessment could be applied to an organization’s work control methods. Some of the elements within the work control system are planning, identifying hazards, identifying controls, scheduling, performing, verifying/testing, and documenting the work performed.

1. At the process level, assessments would be performed by independent assessors to verify compliance with procedures and to ensure the work-control documents (e.g., procedures, CANCELE D CANCELE D 8 DOE G 414.1-1A 5-31-01 instructions, radiation surveys, permits, and safety checklists) accurately reflect the task and associated hazards. 2. At the system level, assessments would be performed to determine whether all the necessary elements and interfaces are addressed to ensure the system is capable of consistently meeting requirements and customer expectations. A management assessment of the work control system might determine the cost and resource allocation issues that impact the system. 3. At the program level, a maintenance management program, which relies on the work control system, would use results from the process and system level assessments to determine the effectiveness of the entire maintenance program. This program assessment could be performed as either a management assessment or an independent assessment. A management assessment might focus on comparing the strategic goals for maintenance with actual performance to determine whether the rewards and recognition plan targeted to improve maintenance has had the desired effect. The independent assessment might compare the program results with contractual and regulatory commitments or customer requirements. From the senior management perspective, the primary focus of management assessments should be at the program level and concerned with strategic issues. Data on systems performance can be rolled up from midlevel and first-line management assessments and independent assessments. Examples of comprehensive assessment models at the program level include the Malcolm Baldrige National Quality Award, Operational Readiness Reviews, and the Voluntary Protection Program. Independent assessment emphasis is placed on system performance in support of programs and to determine its ability to deliver product and services that meet customer expectations. Independent assessments may also be used to confirm management assessment results where organizational vulnerability is high (e.g., potential regulatory penalty, ES&H significant hazard). From the first-line management perspective, the primary focus of assessments should be the capability of systems and the processes that support them. To ensure that these systems contribute to program goals, managers must evaluate system performance based on these goals. Ultimately management, with support from assessments, is responsible for planning the balance and application of independent and management assessments to ensure they improve and add value to the organization. 4.4 Assessing for Compliance, Effectiveness, and Performance There are three different methods commonly used for accomplishing assessments. These are usually known as compliance assessment, effectiveness assessment, and performance-based assessment. While each method has distinct characteristics, a good assessment will use elements of all three. CANCELE D CANCELE D DOE G 414.1-1A 9 5-31-01 4.4.1 Compliance Assessment Compliance assessment focuses on verifying compliance with requirements through the implementation of procedures. Compliance assessment begins with a determination of the contractual and regulatory requirements binding the assessed organization. Assessors then verify that requirements flow down to implementing documents such as procedures, whose implementation is in turn verified.

Assessing for compliance alone may not adequately identify higher level systemic or programmatic problems or determine the effectiveness of the program. In many situations an organization has written procedures that appear to implement the requirements; however, in practice the intent of the requirements is not fully achieved because of variables such as the way those procedures are executed. 4.4.2 Effectiveness Assessment Effectiveness assessment begins like compliance assessment, looking for implementation of requirements in procedures and compliance with the procedures in the workplace. This is followed by a determination whether pure compliance has led to effective implementation of the intent of the top-level requirements. The assessor is expected to determine whether a noncompliance or series of noncompliances with procedures could actually result in a failure to satisfy top-level requirements. The assessor must return to the top-level requirements to determine the program effectiveness. 4.4.3 Performance-Based Assessment Performance-based assessment takes a different approach by focusing first on the adequacy of the process that produced a product or service and then the product itself. If problems are found in the product or work processes, then the assessor evaluates the methods and procedures used to implement the applicable requirements. This is done to find the failure that led to the problems. In performance-based assessment, great emphasis is placed on getting the full story on a problem before coming to a conclusion. If an assessor sees a problem with the execution of a welding process, he or she should determine the extent of the problem. Is it limited to one welder? Is it limited to one process? Can the problem be traced to the qualification program for the welder or to the qualification program for the welding process? Or is there a problem with the weld material itself, indicating an engineering or procurement problem? While the assessor must be familiar with requirements and procedures, in performance-based assessment the assessor’s experience and knowledge play an integral part in determining whether requirements are satisfied. Therefore, participants in performance-based assessments must be technically competent in the areas they are assessing. For example, if an assessor is evaluating a welding process, the assessor relies heavily on his or her knowledge of welding codes, welding processes, and metallurgy, rather than just verifying simple procedure compliance. Performance-based assessment usually provides the most useful information to management; however, it requires a much higher level of competence on the part of the assessment team. In practice, assessments are likely to include elements of all three methods. CANCELE D CANCELE D 10 DOE G 414.1-1A 5-31-01 5. GUIDELINES Organizations should establish procedures for planning and performing management and independent assessments. These should address training and qualification of personnel, planning the assessment processes, performance protocols and tools, reporting, distributing reports, and developing and implementing corrective actions and other follow-up activities. The following guidelines are presented to assist organizations in developing their procedures and protocols. Management assessments share many procedural and protocol commonalities with independent assessments. Because of this, the organization must ensure assessment procedures are well defined and integrated while maintaining the separate focuses of management and independent assessments.

5.1 Assessment Personnel Assessment personnel facilitate continuous process improvement by identifying ways programs, systems, and processes can be improved and by providing information to management and process owners. The assessor should be able to collect performance data through interviews, document reviews, observation, and inspection. It is very important that the assessor also be able to communicate effectively, both orally and in writing, and demonstrate effective interpersonal skills. Both management and independent assessments should be accomplished by qualified individuals who are knowledgeable of the program, system, or process being assessed and have had training to ensure an understanding of the assessment processes and the desired outcomes. Individuals performing independent assessments should not have performed, supervised, or been directly responsible for performing the activities or in the areas being assessed. Independence is determined based on the individual not having bias, rather than on organizational affiliation. The independent assessor must have both the personal and organizational freedom to communicate with the management of the assessed organizations. Organizations should establish a formal training and qualification program for independent assessors, including both assessment team leaders and team members, that reflects both regulatory and customer requirements. Organizations may adopt third-party personnel qualification programs such as the American Society for Quality’s “Quality Auditor Certification” (http://www.asq.org/cert/types/index.html) or the Registrar Accreditation Board’s certification program (http://www.rabnet.com/index.shtml). The International Organization for Standardization and the American Society of Mechanical Engineers provide additional guidance for training and qualification of assessors (ISO-9001 or ISO-14001 and NQA-1, respectively). At a minimum, however, training programs should be based on some recognized, relevant standard. Effective assessments may be accomplished through use of an assessment team with combined skills and experiences. Training for assessors should address the policies and procedures of the assessing organization. To enhance assessment performance and capability, new assessment personnel should participate in on-the-job training with qualified, experienced assessors before being CANCELE D CANCELE D DOE G 414.1-1A 11 5-31-01 Figure 2. Examples of assessment program documents. considered fully trained or receiving a required qualification. Further guidance on assessor training and qualification is provided in DOE Orders and Guides and the standards in appendix A. 5.2 Assessment Program Planning 5.2.1 Assessment Programs Assessment programs should be developed to the level of rigor and detail required to ensure adequate review of programs, systems, and processes. An assessment program is a guide for the overall process and ensures assessments are conducted in a cost-effective, efficient manner. Items (figure 2) considered essential for a comprehensive assessment program include the following. • Assessment scheduling, planning approach, and logic (including how independent and management assessments are balanced). • Methodology for determining/developing performance criteria. • Recognition and use of third-party assessment results (accreditation, certification, registration, and regulatory). • Assessment ethics and behaviors.

• Qualification and training of assessment team personnel. • Protocols for conduct, including interfaces and meetings. • Format/review of assessment plans and agendas. • Reporting methods/procedures for concerns, findings, observations, and improvement opportunities, including distribution, and mechanisms for addressing imminent danger issues. • Procedure/process for verifying and following up on concerns. • Assessment records management program, including identification of records that will be retained, retention periods, and protection. Assessments should be conducted at a frequency commensurate with the hazards, status, and importance of the program, system, or work process and should be focused on worker health and safety, public health and safety, environmental protection, community concern, strategic planning, organizational resources, compliance and liability, and business efficiency and productivity. CANCELE D CANCELE D 12 DOE G 414.1-1A 5-31-01 Complexity, reliability, risk, and economic considerations should also be considered when planning and scheduling assessments. The application of a graded approach using a risk-based decision- making process will ensure that resources are applied in a manner that provides the greatest benefit to the assessed organizations and their customers. 5.2.2 Management Assessment Planning Management assessments should be planned in a systematic manner by the individual managers to address all areas under their responsibility and to focus on those areas presenting the greatest risk for failure or potential for improvement or that have not been covered by an independent assessment. Senior management should retain overall responsibility for the planning and performance of management assessments. Results of the planning process should be documented in an assessment plan. Those responsible for planning individual assessments should consider the following: • descriptive title or name for the assessment area; • brief description of the area or activity to be evaluated; • identification of the assessment team leader and team members; • schedule for the start and completion of the assessment, including issuance of the assessment report; and • other information related to the actual assessment evaluation (e.g., performance objectives, management systems, resource availability, efficiency measures, effectiveness measures). Management assessment planning should include the mechanics of performing the assessment such as the expectations for time involved, assessment tools that may be used, reporting requirements, and how areas for improvement will be identified, tracked, and closed. Management assessments should focus on the identification and resolution of both systemic and cultural management issues and problems that may contribute to customer expectations not being met. Assessments should include evaluating conditions such as the state of employee knowledge, motivation, and morale; communication between employees and management; and the adequacy of human and material resources. 5.2.3 Independent Assessment Planning The purpose of independent assessment planning is to ensure assessments efficiently address the objectives of the assessment program with the least amount of disruption to the facility/organization being assessed. The level of planning will vary significantly depending on scope, breadth, and complexity of the system or process being assessed. The planning process should provide for input from the assessed organizations and consideration of their customers and stakeholders. Specific budget requirements and required resources and support should be identified as early in the planning process as possible. Administrative issues such as the need for any review documents

CANCELE D CANCELE D DOE G 414.1-1A 13 5-31-01 before the actual assessment visit and considerations for travel and clearances should be addressed. It is important to remember that each assessment needs to characterize a program, system, or process during a limited time. Effective planning will ensure this occurs. There are many scheduling and planning approaches to satisfying these requirements. Appendix C describes tools to aid in assessment planning, and appendix D provides an example of planning and scheduling independent assessments using an integrated, risk-based approach. 5.2.4 Planning Updates Assessment planning should be periodically reviewed and modified as new information on the facility or organization is obtained. An assessment that finds good performance should be used as justification to reduce the frequency and depth of future assessments. Areas of poor performance should receive increased attention, especially if there are indications that management is unable to correct identified problems, because recurring and cumulative deficiencies, even in a low hazard operation, may decrease the likelihood of its achieving its mission. 5.3 Assessment Integration Multiple layers of assessment of a single organization or facility do not add value and keep the organization from accomplishing its mission. Therefore, routine communication and trust among the various levels of assessment bodies is essential in coordinating plans. This task involves identifying overlapping and redundant assessments to reduce their negative impact on work performance. Once identified, assessments can be coordinated and consolidated in several ways. • Change assessment scope to prevent two assessment organizations from performing the same type of assessment on the same subject. • Combine separate assessment teams to evaluate the subject in a single visit. • Cancel an assessment based on an agreement to share the results of an assessment. Where significant redundancies exist, as when many contractors perform supplier quality assessments of analytical laboratory services, recognition and use of accredited, third-party assessment results and shared information from other contractors should be considered. (Note: The DOE Supplier Qualification Information Group is one source of shared information.) Each of these methods requires staff time to develop, implement, and manage; however, they can result in substantial savings in resources while enhancing the ability of the assessing organization to cover a greater number of activities. 5.4 Assessment Agendas The use and detail of an assessment agenda will vary depending on what is being assessed and whether the assessment is a management assessment or an independent assessment. Agendas are used to scope and plan individual assessments and should include input not only from the assessed CANCELE D CANCELE D 14 DOE G 414.1-1A 5-31-01 organizations but from their customers. A documented assessment agenda not only allows communication of expectations to the assessed organization, but also allows the assessment team to focus its activities more effectively. The scope of the assessment should be defined in terms related to the assessed organization’s mission and goals so the focus and value of the assessment will be clearly understood. The extent of detail included in the assessment agenda should be commensurate with the protocols of both the assessed and assessing organizations. The assessment agenda should include the following items.

• Team members and their qualifications, organizations, and interfaces. • Description of the assessment scope and performance criteria. • Dates of the assessment. • Schedule of assessment meetings—preassessment, daily, and postassessment. • List of documents to be provided the assessment team when it arrives. • Request for office space, phone lines, and other administrative support as required. • Request for site safety briefing. • Request for points of contact for each functional area. The organization being assessed should be contacted and provided an assessment agenda as soon as possible before the assessment (unless regulations or contracts specify other contact protocols). Management and internal, independent assessments in smaller organizations may not involve as much information as that listed above nor the degree of formality required for assessments performed by external organizations. 5.5 Performance Criteria Assessments seek to ensure that performance expectations defined by management and process owners are being met. Assessors should clearly understand the programs, systems, or processes being assessed, including their goals and associated objectives and requirements for efficient, effective performance of operations. Performance requirements can be found in the following source documents. • Source Documents —Federal and State regulatory requirements —Appropriate codes and standards —Contract requirements —DOE Orders, Manuals, and Notices —Implementation plans —Implementation procedures —Facility safety documents —Policy and mission statements CANCELE D CANCELE D DOE G 414.1-1A 15 5-31-01 —DOE-approved Work Smart Standards —Standards/Requirements Identification Documents (S/RIDs) —Plans and programs Much information on performance and additional performance requirements may be available to assessors in documents and reports like the following. • Performance Information —Reports from outside regulators —Facility operations reports —Performance reviews —Previous assessment reports —Internal inspections, reviews, and reports —Corrective action plans and status reports —Concerns and occurrence reports —Performance indicators —Price-Anderson Amendments Act Nonconformance Tracking System reports (at URL http://tis.eh.doe.gov/enforce/nts/nts.html-ssi) Requirements contained in these documents are selected based upon impact on the assessed organization’s mission and the relationship to the scope of the assessment. From selected requirements, objective statements (performance measures) are developed for determining whether or not a program, system, or process is working efficiently and effectively. From these measures, the specific performance criteria (based on written programs, DOE Orders, rules, etc.) are developed and tools selected for conducting the appraisal. In developing performance criteria, assessment personnel should not reinterpret or redefine requirements specified in the source documents. 5.6 Assessment Planning Tools

Assessment planning tools such as checklists are an essential element of an effective assessment. They vary in format, content, and level of detail, but all have one thing in common: they help focus the assessor on the mission and objectives of the program, system, or process being assessed. Application of planning tools before an assessment ensures more effective use of time and ensures the focus of the appraisal is identified and maintained during the course of the assessment. Assessment planning tools are often used to relate the performance criteria to the established assessment scope and may include lists of interview questions, major elements of programs, or detailed process work steps. Similar to a road map, each tool is used to remind the assessor of where he/she is going and the items likely to be encountered along the way. Planning tools are extremely useful when the assessment basis is complex or the requirements come from multiple sources. Typical planning tools include matrices, flowcharts, cause-effect diagrams, tree diagrams, checklists, and information systems. (See appendix C for examples and further discussion of these tools.) CANCELE D CANCELE D 16 DOE G 414.1-1A 5-31-01 5.7 Independent Assessment Process Sections 5.7 and 5.8 discuss independent assessment and management assessment, respectively, in detail. As discussed previously, there are many commonalities, so to avoid unnecessary duplication, Section 5.7 is very detailed, pointing out areas where the two types of assessment differ, and Section 5.8 is briefer, concentrating on those elements unique to management assessments. 5.7.1 Preassessment Meetings The effectiveness and efficiency of independent assessments can be improved greatly if representatives of the assessment team meet informally with key members of the assessed organization. This should be done at least a week before the assessment fieldwork actually begins. Preassessment meetings are particularly useful when the assessment team is completely external to the organization being assessed. These meeting are an opportunity to clarify the assessment team’s agenda and to work out logistical problems. The assessment team should identify and schedule the individuals they will need to interview so that arrangements can be made to ensure their availability. Documents that require effort to retrieve can be identified in advance, and the need for assessment team work space can be resolved. 5.7.2 The Entrance Meeting An entrance meeting should be conducted immediately before the assessment fieldwork begins. This is the appropriate place to “set the stage” for the performance of a positive and productive independent assessment. (Such meetings may not be necessary for management assessments.) This meeting is held between personnel from the assessing organization and the managers of the organization being assessed. It usually takes place at the assessed organization’s location/facility. The purpose of this meeting is to allow the assessment team to meet the assessed organization’s managers and to answer any questions they may have about the assessment. This meeting is also used to establish how concerns involving imminent danger or regulatory noncompliance will be communicated. The protocols to be followed during the assessment should be clarified during the meeting, which usually includes a discussion of the following: • purpose and scope of the assessment, including authority for conducting the assessment;

• assessed organization’s mission, program, systems, and processes; • scheduled length of the assessment; • source documents and performance information that form the basis for the performance criteria to be used; • knowledgeable individuals from the assessed organization as points of contact for each assessor; • any restrictions on the collection and/or disposal of assessment notes/records by the assessors; • logistics, including work area, working hours, lunch hours, etc.; • time and location of periodic status meetings; and • time and place of the postassessment meeting. CANCELE D CANCELE D DOE G 414.1-1A 17 5-31-01 5.7.3 Performing Independent Assessments The assessment should be conducted in accordance with established protocols developed by the assessing organization. Any agendas or specific protocols established during the preassessment meeting are used to ensure that the assessment is conducted effectively and safely. Assessors should keep their points of contact informed of their activities to preclude surprises during the postassessment conference. This may include requests for additional assistance or the communication of concerns that require immediate action on the part of the assessed organization. Timely communication, verbal and written, will allow the assessed organization to verify the accuracy of observations and provide relevant facts and background on the issues. One way to accomplish this communication is to meet periodically (daily or every second day) with the organization being assessed to convey questions, concerns, and status. Daily team meetings provide assessment team leaders with information on the completion status of the assessment plan and issues requiring additional action (e.g., clearances, access, requests for personnel or material, and impasse resolution) and may be useful to ensure continuity and overall focus. These meetings are also the setting for informing other team members of issues that may be of interest in their assigned scope or for integrating data gathered by the various assessors. The meetings should be brief so that they do not reduce the team members’ time with the processes they are to assess and the people they are to interview. It is important that sufficient information is gathered during the assessment to determine whether an activity meets the performance criteria established. The assessor should be able to clearly state the criterion impacted by the activity and whether identified results also impact the mission/goals of the organization. To accomplish this, the assessor may deviate from the assessment agenda to determine the extent and significance of an issue. Deviations that affect the assessor’s ability to complete the assessment agenda must be made known to the team leader and the organization being assessed. 5.7.4 Independent Assessment Techniques Effective assessments use a combination of tools and techniques to maximize the productivity of the assessment team and resources. Such assessment techniques include document reviews, interviews, observation, inspection, and performance testing. Use of the planning tools discussed in appendix C also allows for more complex analysis and systematic coverage of the areas being assessed. In using these techniques, the assessor should not forget that the objective is to verify accomplishment of an organization’s mission. To save time, the assessor should gather only data and information relevant to overall program performance and the achievement of program objectives.

It is generally not acceptable to identify suspicions about the adequacy or inadequacy of a program, system, or process. Investigations should be sufficiently thorough and information gathered with sufficient diligence that accurate, detailed conclusions and issues can be provided to assist the organizations that will receive the final report. In using any of these techniques, assessors should maintain good records of the assessment results. These may include personal notes or other information to support the assessment and may be included in the checklist information. These records are useful in writing the report and any CANCELE D CANCELE D 18 DOE G 414.1-1A 5-31-01 associated findings and recommendations and will become invaluable if questions arise during the report review process. All classified notes should be disposed of properly in accordance with established and agreed-upon procedures. A discussion of each of the techniques follows. 5.7.4.1 Document Review. Document review is used extensively during an assessment to substantiate the information obtained during interviews and observation. During the course of an assessment, questions may arise concerning what is heard and seen. The review of documents (including logs, procedures, work orders, and other data) provides a method for answering these questions and validating the assessment results. The drawback of document review is that the accuracy of the records cannot be ascertained by review alone. This technique should be combined with interviews, observation, inspection, and/or performance testing to complete the picture of performance. Records and documents should be selected carefully to ensure they adequately characterize the program, system, or process being assessed. 5.7.4.2 Interviews. Interviews provide a means to verify the results of observation, document review, inspection, and performance testing. In addition, interviews allow the responsible person to explain and clarify those results. The interview helps to eliminate misunderstandings about program implementation and provides a venue where apparent conflicts or recent changes can be discussed and the organization and program expectations can be described. Tools developed during assessment planning are used to prepare for the interview. Assessors should also prepare questions in advance to keep the interview focused. 5.7.4.3 Observation. Observation, the viewing of actual work activities, is often considered the most effective technique for determining whether performance is adequate. Assessors should understand the effect their presence has on the person being observed and convey an attitude that is helpful, constructive, positive, and unbiased. The primary goal during observation is to obtain the most complete picture possible of the performance, which should then be put into perspective relative to the overall program, system, or process. Before drawing final conclusions, the assessor should verify the results through at least one other technique. 5.7.4.4 Inspection. Inspections are performed to verify the adequacy and condition of physical facilities, systems, equipment, and components. Usually inspections are used to obtain additional information concerning other items evaluated during the assessment, such as equipment labeling, configuration control, the status of system lineups, adequacy of construction, or material storage. Inspections may also be performed to gain information and data for interviews and/or work observation. While on these inspections, the assessor must heed all security and safety requirements. It is always a good practice to be accompanied by someone familiar with the facility.

5.7.4.5 Performance Testing. Performance testing is used to observe the response of personnel or equipment by creating a specific situation and noting performance. This technique is especially useful when activities of interest would not normally occur during an assessment visit. It is also used when timeliness and appropriateness of the response are critical to an organization (e.g., emergency responses). CANCELE D CANCELE D DOE G 414.1-1A 19 5-31-01 Figure 3. Graphic presentation of assess- ment results. 5.7.5 The Exit Meeting Independent assessment programs can gain value from a postassessment exit meeting. This meeting is used primarily to present the assessment summary and provide the assessed organization an opportunity to verify the factual accuracy of assessment results. To facilitate this, assessors should be prepared to provide detailed supporting information for those results (ideally, a draft assessment report should be available at this time). This meeting also offers an opportunity for the assessed organization to present its management position and any plans for addressing the results. Reasonable time should be allowed to discuss any concerns, but this meeting should not be used to argue the assessment agenda or methodology. 5.7.6 Assessment Reporting Assessment reports are required to communicate the issues identified during an assessment. Assessment team leaders have the overall responsibility for preparing the report and obtaining approval for its release from their senior management. The assessment report may be formal or informal, depending on the level of assessment performed, but should provide a clear picture of the results in terms of the programs, systems, and processes assessed. The report should be clear and easy to understand and should include only facts that directly relate to assessment observations and results. It should include sufficient information to enable the assessed organization to check the report for accuracy (if such a check was not done during the assessment) and to develop and implement appropriate improvement plans. Every effort should be made to ensure assessment reports are concise, accurate, and understandable. For example, summary information may be clearer or more easily understood if presented graphically (figure 3). In preparing the report, authors should also remember that many people who will read the report have had no active role in the assessment and the report may be their only source of information regarding its conduct and results. A recognized good practice is to provide a draft copy of the report to the assessed organization to allow the staff to comment on the factual accuracy (if a draft for this purpose was not presented at the postassessment meeting); however, the review is only to confirm factual accuracy, not to contest or argue the assessment team’s conclusions. Specific report formats may vary considerably from one organization to the next. In developing a report format, the assessment organization should solicit input from report recipients to ensure the CANCELE D CANCELE D 20 DOE G 414.1-1A 5-31-01 report meets their needs. An independent assessment report usually includes the sections described below. (Note: A management assessment report may be less formal and may only require the executive summary.)

5.7.6.1 Executive Summary. This summary should be a brief, stand-alone document. It should describe the programs, systems, and processes assessed and the overall assessment results, including an evaluation of the effectiveness, efficiency, and adequacy of the area(s) assessed and the overall results. The executive summary should describe the strengths and weaknesses affecting the assessed organization, including barriers to performance, so that meaningful action can be taken for improvement. 5.7.6.2 Observation Section. Each part of this section should focus on the established assessment scope and the identified organization mission; otherwise, the recipient of the report will question why a specific area or activity was assessed. The section should include general background on the assessment, including team members, scope of the assessment, methodology used, and a summary of the assessment basis and source documents. This section should also include a detailed discussion of each area assessed, including specific performance criteria used and summaries of interviews, documents reviewed, observations, and inspections. The summaries contained in this section should support the specific items discussed under the results section. Noteworthy practices identified during the assessment should also be documented so that the assessed organization and other organizations can learn and build upon them. 5.7.6.3 Results Section. This section should list and discuss specific problem areas or deficiencies, areas needing improvement, or noteworthy practices identified during the assessment. In addition, this section should highlight any recurring problems as indicators of ineffective corrective action by the assessed organization. For each item listed, the report should include a discussion of the specific performance criteria used and the basis for the nonconformance in sufficient detail to enable further analysis and action by the responsible organization. The report should also include any required postassessment actions by the assessed organization. For example, a series of “like” discrepancies may be symptoms of an underlying system problem. Therefore, a single issue should be developed that cites the individual discrepancies as evidence of a system breakdown. Issues should be defined, labeled, and enumerated in a manner that facilitates a response. While this should be in accordance with the assessing organization’s assessment program, the language used should clearly distinguish objective noncompliances from observations, opinions, and improvement opportunities. 5.7.6.4 Attachments. Attachments provide supplementary information to validate the assessment and its methodology. They can be helpful in planning corrective actions and follow-up. Items frequently included as attachments to assessment reports are the assessment agenda, a list of persons contacted, a list of documents reviewed, performance criteria, and the tools used to perform the assessment. CANCELE D CANCELE D DOE G 414.1-1A 21 5-31-01 5.7.7 Releasing and Responding to Assessment Reports While team leaders have overall responsibility for the report, the entire assessment team should have an opportunity to read and sign the completed report. At a minimum, the final report should be distributed to the management of both the assessed and assessing organizations. Distribution to other organizations (e.g., Defense Nuclear Facilities Safety Board or other regulators) should be defined during the planning phase.

Because the true value of an assessment is the improvement opportunities it identifies, and its value typically diminishes over time, the best time to release a report is immediately after the postassessment meeting, which allows the assessed organization to begin improvement actions, yielding the maximum return from those actions. The assessment report or transmittal correspondence should clearly indicate what response is expected from the assessed organization and a reasonable response date. 5.7.8 Corrective Action Managers responsible for the activities assessed are also responsible for the development of effective corrective actions for the problem areas or deficiencies discovered during the assessment. At a minimum, the corrective actions should include the following: • measures to correct each deficiency, • identification of all root causes for significant deficiencies, • determination of the existence of similar deficiencies, • corrective actions to preclude recurrence of like or similar deficiencies, • assignment of corrective action responsibility, and • completion dates for each corrective action. For independent assessments, the proposed corrective actions should be reviewed for concurrence by the assessment team leader, with input from the assessment team. The senior line management to whom the assessed organization is accountable should approve the corrective actions. This will help ensure that the planned actions will be effective in resolving the problem areas and deficiencies reported by the assessment team. Management assessments should be reported in accordance with the organization’s management assessment reporting protocols. Because of the nature of management assessments, corrective actions may be required by any level of management or by other organizations. Specific, detailed requirements (DOE O 414.1A, attachment 2) and guidance (DOE G 450.4-1B, Integrated Safety Management System Guide, vol. 2, appendix G) exist for responding to assessments conducted by the Department’s independent oversight organizations for safety and CANCELE D CANCELE D 22 DOE G 414.1-1A 5-31-01 emergency management. The requirements and guidance address topics such as line management responsibility, timeliness, corrective action plans, independent review, tracking, reporting, review, implementation, verification, and closure. The concepts are based on consensus standards, compatible with this guide, and may be adapted for use with an organization’s internal management and independent assessment program. 5.7.9 Follow-up After a reasonable period of time has elapsed, follow-up activities should be performed to verify the effectiveness of the corrective actions and how they were implemented. The verification should, at a minimum, sample the corrective actions to determine whether the problem/issue to be addressed has been resolved. The organization’s reporting systems (e.g., noncompliance tracking system, occurrence reporting and processing system, external oversight reports and regulatory violations, performance indicators) should be reviewed for evidence of the problem (or a similar problem) recurring. The same techniques used to conduct assessments may be used for verifying corrective actions; however, there are several common ways to verify the implementation of corrective actions, including the following: • reassessment of the deficient areas; • review of new or revised quality-affecting documents such as manuals, procedures, and training

records; • verification during the next scheduled assessment; and • verification by conducting a surveillance covering the areas of concern. Several DOE directives on special focus assessments also include verification steps: DOE G 450.4-1B, Integrated Safety Management System Guide, vol. 2, appendix G; DOE-STD-3006-95, Planning and Conduct of Operational Readiness Reviews (ORR); and Office of Oversight Environment, Safety, and Health Appraisal Process Protocols, appendix A, “Safety Management Template,”July 1999. The key thing to remember is that verification is necessary. A solution to a problem may look good on paper but may not be readily implemented or long lasting. The failure to adequately identify all root causes will most likely result in a recurrence of the deficiency. Therefore, an appropriate amount of follow-up is necessary to ensure the effectiveness of the corrective action process and to reestablish confidence in the item/service assessed. 5.8 Management Assessment Process Planning management assessments is an organization-specific effort that should be integrated with other assessment processes to avoid redundancy and provide the greatest value to the organization. No one method is appropriate. Organizations are challenged to make the management assessment process a value-added process that will lead to improvement in organizational performance, safety, CANCELE D CANCELE D DOE G 414.1-1A 23 5-31-01 and meeting customer expectations. It is important to remember that while management assessments share many commonalities with audits, they should not become “managers’ audits,” but should instead focus on evaluating organizational performance and identifying barriers that hinder improved performance. Management assessments should contain an introspective, self-evaluative element (What/how should things be?) rather than focusing on compliance alone. 5.8.1 Defining the System The organization should have a written description of the management assessment process. The description should address all elements of the process, including, but not limited to, the following: • management levels that will be expected to perform assessments, • general goals of the management assessment process, • training or mentoring that will be provided to assessing managers, • overall assessment planning process, • expectations for the number and frequency of assessments to be performed, • expectations and guidelines for management assessment reports, • reporting and follow-up process, • management assessment planning process, and • administration of the process. 5.8.2 Assessment Scheduling The organization should review and update its management assessment schedule on a regular basis, either bimonthly or quarterly, to ensure relevance. The review should consider the current conditions, conclusions of recent management assessments, inputs for independent assessments, and organizational performance. Management assessments should be planned with input from all levels of management. Some organizations have found it beneficial to schedule the number of assessments but leave some of the topics or areas of assessment to be determined by the performing manager to allow needed flexibility and to allow the manager the freedom to perform assessments that will result in the greatest opportunity for improvement. The planning process may include and take credit for existing management reviews or similar assessments that routinely occur, such as the following:

• program reviews, • strategic planning sessions, • reviews of performance indicators, • organizational goals- and objectives-setting sessions, CANCELE D CANCELE D 24 DOE G 414.1-1A 5-31-01 • financial reviews, • reviews of outputs of improvement teams, and • reviews of independent assessments. When the above are included in the management assessment system, the requirements for documenting the assessment and follow-up and tracking of conclusions should be applied (refer to section 5.7). Management assessments may include some benchmarking activities, both internal and against other organizations. The assessment of internal and external performance indicators may also be beneficial. There is no fixed number of assessments that must be performed; however, the organization must be able to show that the management assessment program complies with 10 CFR 830, Subpart A. 5.8.3 Performing Management Assessments Management assessments may be performed by individual managers or teams of managers. The primary responsibility for management assessments resides with managers because they are in the best position to identify barriers to improved performance and to effect changes. While some aspects of the assessments, such as collecting information, may be delegated to staff, it is the manager’s responsibility to perform the assessment and determine the conclusions. Personal involvement by the manager will yield the most meaningful information for improving the performance of the organization. 5.8.4 Assessment Reporting Management assessments should be reported in accordance with the organization’s Management Assessment program. As with the reports on independent assessments, management assessment reports should include a concise summary of the topic or area assessed, the conclusions reached, and follow-up actions that may be required. Reports should be available for use by others and for future planning. Provisions may have to be made for reports dealing with sensitive areas, proprietary information, or classified information. Reports identifying potential regulatory compliance issues should be provided to the appropriate managers for any necessary action or reporting. 5.8.5 Follow-up Mechanisms should be in place to provide tracking of and follow-up on identified needed actions. Individuals should be assigned to complete the follow-up actions and provide input back into the system by specific dates. The management assessment process should provide a means for coordinating actions when more than one organization’s response is required. Follow-up for management assessments may include another management assessment, inclusion of the assessed areas in a future independent assessment, and/or evaluation of subsequent performance indicators. CANCELE D CANCELE D DOE G 414.1-1A 25 (and 26) 5-31-01 Senior management should review the follow-up actions periodically to ensure that they are appropriate and that actions are taken in a timely manner. 5.8.6 Feedback Every system should include a mechanism for feedback to improve the overall system. Management should periodically request feedback from all levels on the effectiveness and efficiency of the process, value of the assessments, and actions that should be taken to improve the management assessment process. CANCELE D CANCELE D DOE G 414.1-1A A-1 5-31-01 APPENDIX A—CONSENSUS STANDARDS AND REFERENCES A.1 Consensus Standards

The following consensus standards provide methods for implementing the guidance contained herein. A single standard may not fully implement all elements of the requirements (particularly for management assessments); therefore, these documents should be used in conjunction with 10 CFR 830, Subpart A, and DOE O 414.1A, Quality Assurance, to develop and implement assessment processes that meet the DOE assessment requirements. The organization remains responsible for compliance with 10 CFR 830, Subpart A, and DOE O 414.1A. 1. American National Standards Institute (ANSI)/American Nuclear Society (ANS) 3.2-1994 (R1999), Administrative Controls and Quality Assurance for the Operational Phase of Nuclear Power Plants. 2. ANSI/ANS 15.8-1995, Quality Assurance Program Requirements for Research Reactors. 3. ANSI/ASQ (American Society for Quality) Z 1.13-1999, Quality Guidelines for Research. 4. ANSI/ASQC (American Society for Quality Control) E4-1994, Quality Systems Requirements for Environmental Programs, Part A. 5. ANSI/ASQC Q10011-1-1994, Guidelines for Auditing Quality Systems-Auditing, American Society for Quality. 6. ANSI/ASQC Q10011-2-1994, Guidelines for Auditing Quality Systems- Qualification Criteria for Quality Systems Auditors, American Society for Quality. 7. ANSI/ASQC Q10011-3-1994, Guidelines for Auditing Quality Systems-Management of Audit Programs, American Society for Quality. 8. ANSI/ISO (International Organization for Standardization)/ASQ Q9001:2000, Quality Management Systems: Requirements. (Note: ANSI/ISO/ASQ Q9000 series documents are recognized as being identical to their ISO 9000 series counterparts.) 9. ASME (American Society of Mechanical Engineers) NQA-1-2000, Quality Assurance Requirements for Nuclear Facility Applications. 10. DOE-HDBK-1101-96, Process Safety Management for Highly Hazardous Chemicals, February 1996. 11. DOE-HDBK-3027-99, Integrated Safety Management Systems (ISMS) Verification; Team Leader’s Handbook, June 1999. CANCELE D CANCELE D A-2 DOE G 414.1-1A 5-31-01 12. DOE-EM-STD-5505-96, Operations Assessments, May 1996. 13. DOE-NE-STD-1004-92, Root Cause Analysis Guidance Document, February 1992. 14. DOE-STD-1036-93; Guide to Good Practices for Independent Verification, June 1993 (and Change Notice 1, dated December 1998). 15. DOE-STD-3006-95, Planning and Conduct of Operational Readiness Reviews (ORR), November 1995. 16. ISO 9001, see reference 8. 17. ISO 14001.1, Environmental Management Systems–Specification with Guidance for Use, 1996. 18. ISO 14010.1, Guidelines for Environmental Auditing–General Principles, 1996. 19. ISO 14011/1.1, Guidelines for Environmental Auditing–Audit Procedures; Part 1: Auditing of Environmental Management Systems, 1996. 20. ISO 14012.1, Guidelines for Environmental Auditing–Qualification Criteria for Environmental Auditors, 1996. 21. ISO CD.3 19011, Guidelines on Quality and/or Environmental Management Systems Auditing. (This guide should be used for information only; it is in the final stage before approval as an international standard and will replace ISO 10011, parts 1, 2, and 3; ISO 14010; ISO 14011; and ISO 14012.) A.2 Other References The following references provide additional information concerning assessments. 1. DOE G 414.1-2, Quality Assurance Management System Guide for Use with 10 CFR 830.120 and DOE O 414.1, June 1999.

2. DOE/RW-0333P,Quality Assurance Requirements and Description [for the Civilian Radioactive Waste Management Program], Rev. 10, Section 18.0, “Audits,” Office of Civilian Radioactive Waste Management, April 2000 [http://www.ymp.gov/doclist.htm#q]. 3. DOE P 450.1, Environment, Safety and Health Policy for the Department of Energy Complex, June 1996. CANCELE D CANCELE D DOE G 414.1-1A A-3 (and A-4) 5-31-01 4. DOE G 450.4-1B, Integrated Safety Management System Guide, viols. 1 and 2, March 2001. 5. DOE P 450.5, Line Environment, Safety and Health Oversight, June 1997. 6. DOE, Office of Oversight, Environment, Safety, and Health Appraisal Process Protocols, July 1999 [http://www.eh.doe.gov/oversight/procedures/9907app/html/toc.htm]. 7. U.S. Department of Commerce, National Institute of Standards and Technology, “Malcolm Baldrige National Quality Award Criteria for Performance Excellence,” [http://www.quality.nist.gov]. CANCELE D CANCELE D DOE G 414.1-1A B-1 5-31-01 APPENDIX B—ASSESSMENT FUNCTIONAL AREAS This appendix comprises a list of the essential environment, safety, and health (ES&H) and safeguards and security functional areas/activities that should be included in a comprehensive assessment program (see next page). The list represents basic ES&H requirements, processes, and programs found in regulations, DOE policy, and DOE Orders. This appendix should not be interpreted as limiting application of assessments to only these functional areas that ensure ES&H protection, nor should it be interpreted as minimizing the importance of assessing product/service quality and organizational performance. It does, however, illustrate the wide variety of ES&H programs, systems, and processes currently implemented by DOE and its contractors. These ES&H functional areas comprise a complex mix of people, hardware, software, and resources, all impacting on management and performance of activities. An integrated assessment program must be adaptable to this mix to be responsive to senior management needs and comply with 10 CFR 830, Subpart A, and DOE O 414.1A, Quality Assurance. CANCELE D CANCELE D B-2 DOE G 414.1-1A 5-31-01 • Accelerator Safety • Accident, Incident, and Unusual Occurrence Investigation and Reporting Process • Aviation Safety • Biological Hazards • Calibration Control • Computer Software Control • Conduct of Operations • Configuration Control • Construction Safety • Criticality Control • Corrective Action • DOE Nuclear Safety Rule Compliance (10 CFR 830) • Document Control and Records • Emergency Preparedness (N.H.) • Employee Concerns System • Engineering Design Processes • Environmental Management Systems • Environmental Protection and NEPA Compliance • Equipment Modifications • Experimental Programs • Explosives Safety • Facility Operations • Fire Protection • Firearms Safety • Identification & Control of Items • Industrial Hygiene • Industrial Safety • Inspection & Test Control • Integrated Safety Management System Implementation • Maintenance Management • Motor Carrier/Vehicle Safety • Nonconformance Control • Nuclear Facility Safety • Occupational Medicine • Occurrence Reporting/Trending • Operational Readiness Review Process • Packaging and Transportation • Performance Measures and Indicators • Pollution Prevention • Procurement & Contracts (including supplier control) • Quality Management Systems • Radiation Protection (10 CFR 835) • Reactor Safety • Safeguards and Security

• Safety Management Systems • Safety Analysis Documentation (e.g., Bases for Interim Operation, hazard analyses, and safety analysis reports) • Standards/Requirements Identification Document(s) • Suspect/Counterfeit Items • Technical Safety Requirements • Training of Nuclear Facility Personnel • Unreviewed Safety Questions Process • Voluntary Protection System • Waste Management • Worker Protection/Industrial Safety CANCELE D CANCELE D DOE G 414.1-1A C-1 5-31-01 APPENDIX C—TOOLS FOR ASSESSMENT PLANNING AND CONDUCT C.1 Checklists Checklists (example C.1) are lists of assessment objectives and performance criteria. They usually include a column for the requirements (or references to the requirements) and a column for recording assessment observations/evidence. Checklists are especially useful for organizing assessment time by providing a means to list appraisal objectives sequentially. They may also be structured in a form that can easily be converted into assessment report text. Example C.1. Laboratory Calibration Program Checklist Lab/Appraisal Number: Date: Page 1 of Reference Criteria Results CommentsSat Uniat NL-QAM 1. Is monitoring and data collection equipment calibrated? NL-QAM 2. Is equipment calibration traceable to nationally recognized standards? NL-QA-5.1 3. Is equipment calibration performed using approved instructions? NL-QA-5.1 4. Are calibration records maintained for each piece of equipment? NL-QA-5.1 5. Is a use log maintained? In example C.1, the checklist is used to list the primary elements of a laboratory’s calibration program. The basis or source of each criterion is included in the first column to provide a path back to the requirements document(s). The “comments” column provides a place for the assessor to record additional observations as they are discovered during the assessment, which helps to ensure that important data are not lost. C.2 Matrices Matrices (examples C.2 and C.3) are two-dimensional tables showing the relationship between two sets of information. They can be used to show the logical connecting points between performance criteria and implementing actions or required actions and personnel responsible for those actions. In this way, matrices are used to determine what actions and/or personnel have the greatest impact on an organization’s mission. Matrices are especially useful as a way to focus assessment time and organize assessment conduct. CANCELE D CANCELE D C-2 DOE G 414.1-1A 5-31-01 Example C.2. Organizational Responsibilities Matrix Program Development Deficiency Tracking Training Work Control Documents & Records Retention Assessment Director X X Ops Office X X X Ops Support X X X Tech Support X X X X Admin X X In example C.2, the matrix is used to help the assessor plan the assessment by identifying organizational responsibilities for the different assessment areas. This type of matrix is used to maximize use of assessment time during the site visit. Example C.3. Long-Range Planning Matrix Administration Chemistry Biology Materials Building Services Engineering Industrial Hygiene A A A A Radiological Protection B B B Fire Protection A C C C C Industrial Safety A C A Environmental C A C C Personnel Training B C B C Conduct of Operations C C C Quality Assurance A C A C A = 1st assessment; B = 2nd assessment; C = 3rd assessment

Example C.3 is a much broader use of the matrix that allows the assessor to do the long-range planning necessary for ensuring proper application of the assessment program. In this example, the various assessment areas (Y axis) are correlated with the different organizations to be assessed. CANCELE D CANCELE D DOE G 414.1-1A C-3 5-31-01 C.3 Flowcharts Flowcharts (figure C.1) illustrate the steps or activities in a process. They provide an excellent tool for examining how various steps in a process are related to each other and whether or not each subsequent activity is receiving what it needs from the previous one. Flowcharts are used to help the assessor understand how a function is being implemented based on written programs and procedures. Flowcharts also illustrate reporting relationships and indicate whether the handoff of information or materials is adequate. They are especially useful for locating process bottlenecks, which may hinder the organization’s mission. Figure C.1. Flowchart. In figure C.1, the flowchart diagrams the steps in the design process, which helps identify critical areas and determine whether an individual step affects the design process output. In addition, this flowchart may allow the assessor to divide the design assessment between different visits while ensuring overall coverage. C.4 Cause-and-Effect Diagrams Cause-and-effect diagrams (figure C.2) illustrate the relationship between a known “effect” or outcome and all the “causes” or contributors influencing it. The effect being examined may CANCELE D CANCELE D C-4 DOE G 414.1-1A 5-31-01 represent either a wanted or unwanted outcome. The cause-and-effect diagram is used when the outcome of a process/program is known but the contributors need to be evaluated further. These diagrams are especially useful when the contributors stem from different sources across the organization being assessed. Figure C.2. Cause-and-effect diagram. In this example, the assessor would use the cause-and-effect diagram (figure C.2) to identify all the program elements that should be in place to prevent worker exposure. This tool can be used in two ways by the assessor: (1) to verify the effectiveness of individual elements, thereby verifying that the program is working, and (2) to pinpoint the source of programmatic weaknesses. C.5 Tree Diagrams Tree diagrams (figure C.3) are used to map out systematically, in increasing detail, the full range of controls and tasks needed to achieve a primary goal. They can also be used to map out the barriers needed to prevent an unwanted event (called “causal analysis” or “barrier analysis” trees). Tree diagrams may be used by the assessor to verify whether all planned activities are in place to support a program’s objective. They are especially useful for helping the assessor focus on the big picture: the overall goal of the program, with its supporting subgoals. CANCELE D CANCELE D DOE G 414.1-1A C-5 5-31-01 Figure C.3. Tree diagram. In figure C.3, the tree diagram provides a map of the elements needed to support an effective training and qualification program. Using the diagram, the assessor can plan the assessment to ensure that the appropriate activities are being performed and to evaluate the training organization’s effectiveness overall. As this tree diagram is used for the assessment, the elements must be continually rolled up. This means the “Capability/Proficiency Verified” element should be assessed to determine its impact on “Qualification,” which must be assessed to ensure it supports overall “Training and Qualification.”

C.6 Information Systems Information systems comprise a wide range of different forms and formats. In their simplest form they may include the weekly and monthly reports of laboratory or organizational performance that are used to alert the assessment organization of potential assessment areas. In more complex form, these systems may include computerized databases that link performance to specific performance objectives or track actions to resolve programmatic weaknesses. In either case, information systems are important tools for assessors, providing them the data needed to focus assessment activities. CANCELE D CANCELE D C-6 DOE G 414.1-1A 5-31-01 Figure C.4. Information system output. In figure C.4, information on lost-time injuries is displayed in both tabular and graphical form. This information can be used to focus the assessment on either the location of the injuries or the work procedures involved to identify any weaknesses in the accident prevention program. CANCELE D CANCELE D DOE G 414.1-1A D-1 5-31-01 APPENDIX D—INDEPENDENT ASSESSMENT PLANNING Many scheduling and planning approaches satisfy the assessment requirements for integration, grading, and use of a risk-based approach. The following steps are one approach to detailed planning for independent assessments. This detailed level of planning should be performed on a regular basis, such as annually. D.1 Identifying Assessment Areas Assessments should be managed to achieve specific objectives. Programs, systems, and processes (including functional and organizational areas) and their associated performance objectives and measurements should be reviewed to determine their appropriateness and applicability. In the following sections we discuss the information that should be compiled and maintained to manage the assessment process and its objectives after the determination has been made. D.2 Evaluating Probability and Consequences and Probability (Risk) Evaluate the program, systems, and processes to identify the various factors and elements that could result in a failure to achieve the mission or the objective’s success. Factors to consider include, but are not limited to, the following: • worker health and safety (injuries, deaths, exposures); • public health and safety (injuries, deaths, exposures); • community concerns; • regulatory noncompliance and liability; • business efficiency/productivity; • complexity of the involved processes; • reliability of the engineering and administrative controls; • skills and experience of the personnel involved; • maturity of the program, system, or process (developed, mature techniques/processes versus state-of-the-art or developmental/pilot technology); • changes that may affect performance (including regulatory); • life-cycle phase (new, midlife, closeout of activity); • organizational experience with the program, system, or activity; • economic costs (uncertainty); • schedule/commitment or milestones failure; and • performance measures/indicators, trending downward. CANCELE D CANCELE D D-2 DOE G 414.1-1A 5-31-01 Each of the factors should be evaluated for likelihood of occurrence and severity of consequences. This is a rough estimate, using professional judgement, and the use of a multidisciplined team for this process is encouraged. D.3 Prioritizing Assessment Activities

Rank the program, system, and process risks based on the consequences and likelihood. Those with high risk or multiple lower risks should be ranked higher. This may identify risks that crosscut several programs, activities, or organizations, thus benefiting more than one organization. D.4 Identifying Areas for Further Evaluation List the programs, systems, or processes and the areas of risk; then further evaluate other factors such as • time since the last internal, independent assessment; • time since other assessments (external, management, process improvement teams, investigations, etc.); • opportunities to perform the assessment in conjunction with other organizations (internal or external); • work schedules (will a lower-ranked program start or complete before a higher-ranked program or activity); • other scheduled assessments (management assessments, process improvement teams, etc.) that would be expected to address the area; • availability of assessment personnel, including technical personnel to perform the assessment; and • certifications, registrations, or other scheduled activities that would be expected to evaluate the program, system, or process. D.5 Initiating the Assessment Plan Develop an assessment plan for the specific assessment including the following elements: • scope of the assessment (i.e., the program, system, process, organization, and/or activity to be assessed); • objectives of the assessment; • assessment drivers (e.g., the regulatory requirements, contractual agreements, performance objectives, and/or internal procedures that will be used); CANCELE D CANCELE D DOE G 414.1-1A D-3 5-31-01 • assessment team members, including the lead, supporting assessors, and technical experts (if appropriate); and • assessment schedule, with start and end dates (final planning, notification, kickoff meetings, preparation, investigation, closeout meeting, and report issuance). The assessment plan should establish the depth, scope, and breadth of the assessment. It will provide a tool for scheduling and information exchange for both the assessment team and the management of the assessed organizations. D.6 Allocating Resources The resources for performing assessments are limited and seldom allow for performing all assessments. Likewise, the benefits of performing assessments on low risk areas are marginal. As a result, a realistic estimate of the resources available, including their scheduled availability, should be developed. Additional factors such as the availability of personnel independent of the areas to be assessed, budget constraints, management or customer requests, and response/follow-up to previous external assessments should be considered. D.7 Developing the Assessment Schedule and Plan Using the information developed in D.1 through D.6, an assessment schedule and plan can be developed. The schedule and plan should reflect the areas of greatest risk and the reasonable allocation of resources. Assessments that fall “below the line” should be retained as “targets of opportunity” to be performed if resources become available or if one of the planned assessments changes in risk or schedule. The assessment schedule and plan is a tool that allows management and customers to understand the basis for the assessments and justifies the allocated resources. D.8 Maintaining the Assessment Schedule and Plan

Assessment plans should be reviewed periodically and modified as new information on the facility or organization is obtained that changes the estimated risks or reflects changes in available resources. These reviews should occur at regular intervals, such as monthly or quarterly. The review can be used to finalize schedules, team members, etc., for the next period. Assessment areas that have increased in risk can be moved up in the schedule, while others can be moved down. In some cases, assessments that were “below the line” should be moved up to reflect changes that have occurred since the original planning and ranking was performed. The results of assessments, which identify good performance, should be used to reduce the frequency and depth of future assessments. Areas of poor performance should receive increased attention, especially if there are indications that management is unable to correct identified problems. This is because recurring and cumulative deficiencies, even in a low hazard operation, may decrease the likelihood of achieving its mission. CANCELE D CANCELE D D-4 DOE G 414.1-1A 5-31-01 D.9 Frequency Given the likelihood of some or all of these factors being present and the wide variety of Department of Energy activities, it is impossible to define the “right” assessment frequency in this Guide. Too few assessments will not keep pace with the changes occurring in the program, system, or process. Too many assessments will distract the organization from focused attention on the safe conduct of work and mission accomplishment. Therefore, it is ultimately the responsibility of management, guided by assessment professionals, to determine the appropriate mix of assessments (independent and management) to meet customer requirements and ensure mission success. CANCELE D CANCELE D