DOE G 414.1-1B, Management and Independent Assessments Guide for Use with 10 CFR Part 830, Subpart A, and DOE O 414.1C, Quality Assurance; DOE M 450.4 -1, Integrated Safety Management System Manual; and DOE O 226.1A, Implementation of DOE Oversight Policy

DISTRIBUTION: INITIATED BY: All Departmental Elements Office of Health, Safety and Security DOE G 414.1-1B 9-27-07 Certified: 11-18-10 MANAGEMENT AND INDEPENDENT ASSESSMENTS GUIDE FOR USE WITH 10 CFR, Part 830, Subpart A, and DOE O 414.1C, Quality Assurance; DOE M 450.4-1, Integrated Safety Management System Manual; and DOE O 226.1A, Implementation of Department of Energy Oversight Policy [Guides can serve as reference sources for acceptable, non-mandatory implementation methods to satisfy requirements of Orders, Notices, and Manuals. NOTE: Guides do not establish requirements or constitute the basis for a finding of non-compliance with a specific requirement] U.S. Department of Energy Washington, D.C. 20585 NOT MEASUREMENT SENSITIVE DOE G 414.1-1B i 9-27-07 CONTENTS FOREWORD................................................................................................................................ iii BACKGROUND ........................................................................................................................... v 1. INTRODUCTION.................................................................................................................... 1 2. APPLICATION........................................................................................................................ 3 3. GENERAL INFORMATION................................................................................................. 5 3.1 Assessment Program Expectations........................................................................... 5 3.2 Assessment Benefits ................................................................................................... 5 3.3 Graded Approach ...................................................................................................... 6 3.4 Purpose of Assessment............................................................................................... 7 3.4.1 Assessment Elements .................................................................................. 7 3.5 Types of Assessment .................................................................................................. 8 3.5.1 Management Assessment............................................................................ 8 3.5.2 Independent Assessment ............................................................................ 9 3.6 Organizational Activity Levels ............................................................................... 10 3.6.1 Process Level Assessments ....................................................................... 10 3.6.2 System Level Assessments........................................................................ 11 3.6.3 Program Level Assessments..................................................................... 11 3.7 Assessing for Compliance and Performance ......................................................... 12 3.7.1 Compliance Assessment............................................................................ 12 3.7.2 Performance-Based Assessment .............................................................. 12 4. GUIDELINES ........................................................................................................................ 15 4.1 Assessment Personnel .............................................................................................. 15 4.2 Assessment Program Planning ............................................................................... 16

4.2.1 Assessment Programs ............................................................................... 16 4.2.2 Management Assessment Planning ......................................................... 17 4.2.3 Independent Assessment Planning .......................................................... 18 4.2.4 Planning Updates ...................................................................................... 18 4.3 Assessment Integration............................................................................................ 18 4.4 Assessment Plan ....................................................................................................... 19 4.5 Performance Criteria............................................................................................... 20 4.6 Assessment Planning Tools ..................................................................................... 21 4.7 Independent Assessment Process............................................................................ 22 4.7.1 Pre-assessment Meetings .......................................................................... 22 4.7.2 The Entrance Meeting .............................................................................. 22 4.7.3 Performing Independent Assessments .................................................... 23 4.7.4 Independent Assessment Techniques...................................................... 24 4.7.5 The Exit Meeting....................................................................................... 25 4.7.6 Assessment Reporting............................................................................... 25 4.7.7 Releasing and Responding to Assessment Reports ................................ 26 4.7.8 Corrective Actions .................................................................................... 26 4.7.9 Follow-up Assessments ............................................................................. 27 ii DOE G 414.1-1B 9-27-07 4.8 Management Assessment Process........................................................................... 27 4.8.1 Defining the System .................................................................................. 28 4.8.2 Assessment Scheduling ............................................................................. 28 4.8.3 Performing Management Assessments ................................................... 29 4.8.4 Assessment Reporting............................................................................... 29 APPENDIX A—CONSENSUS STANDARDS AND REFERENCES................................. A-1 APPENDIX B—ASSESSMENT FUNCTIONAL AREAS ................................................... B-1 APPENDIX C—TOOLS FOR ASSESSMENT PLANNING AND CONDUCT ................ C-1 APPENDIX D—MANAGEMENT AND INDEPENDENT ASSESSMENT PLANNING D-1 APPENDIX E—ACRONYMS AND ABBREVIATIONS .................................................... E-1 DOE G 414.1-1B iii (and iv) 9-27-07 FOREWORD This Guide is approved for use by all Department of Energy (DOE) and National Nuclear Security Administration (NNSA) components and contractors. Throughout this Guide the use of this DOE term includes DOE and NNSA. Suggestions for improving this Guide are welcome and should be sent to: James O’ Brien HS-21/270CC U.S. Department of Energy 1000 Independence Ave., SW Washington D.C. 20585-270 Phone: (301) 903-1408 e:mail: james.o’brien@hq.doe.gov

Guides provide preferred, non-mandatory, supplemental information about acceptable methods for implementing requirements, including lessons learned, suggested practices, instructions, and suggested performance measures. Guides do not impose requirements but may quote requirements if the sources are adequately cited. Alternate methods may be used if it can be demonstrated that they provide an equivalent or better level of performance. DOE G 414.1-1B v (and vi) 9-27-07 BACKGROUND Since 2001, assessment practices and DOE requirements have evolved. This revision reflects the following assessment practices, international standards, and changes in DOE expectations related to quality Assurance (QA): • new requirements in DOE O 414.1C, Quality Assurance, dated 6-17-05, for corrective action plans in response to independent oversight assessments; • commitments in the DOE implementation plan for Defense Nuclear Safety Board Recommendation 2004-1; • DOE O 226.1A, Implementation of Department of Energy Oversight Policy, dated 7-31-07; • DOE G 414.1-4, Safety Software Guide for Use with 10 CFR 830, Subpart A, Quality Assurance Requirements, and DOE O 414.1C, Quality Assurance, dated 6-17-05; • DOE G 414.1-5, Corrective Action Program Guide, dated 3-2-06; • DOE O 450.1 Chg 3, Environmental Protection Program, dated 1-15-03; • QC-1 Rev. 10, Nuclear Weapons Quality Policy; • ASME NQA-1-2004 with 2005 Addenda; Quality Assurance Program for Nuclear Facilities, and • new and revised international management system standards [e.g., International Organization for Standardization (ISO) 9001 and ISO 14001] and the integrated environmental and quality audit standard (ISO 19011). DOE G 414.1-1B 1 (and 2) 9-27-07 1. INTRODUCTION DOE and its contractors are required to perform management and independent assessments in accordance with: • Title 10, Code of Federal Regulations (CFR), Part 830, Subpart A, “Quality Assurance Requirements”; • DOE O 414.1C, Quality Assurance; • DOE M 450.4-1, Integrated Safety Management System Manual; • DEAR 970.5223-1, Integration of Environment, Safety, and Health into Work Planning and Execution; • DOE O 226.1A, Implementation of Department of Energy Oversight Policy; and • QC-1 Rev. 10, Nuclear Weapons Quality Policy. This Guide provides information on establishing processes for performing effective assessments. See Appendix A for a list of consensus standards and other references related to assessments. Assessments add value to products and services by providing feedback and linking the management and conduct of work to meaningful improvement actions. Assessment programs should embody the following principles: • Managers are involved in the assessment process to ensure results contribute to improved performance of the programs, systems, and work processes. • Managers receive timely, objective feedback from assessments. • Managers take timely, appropriate actions to resolve quality problems. • Feedback addresses the effectiveness of policies, requirements, standards, processes, procedures, and their implementation. • The process is planned, coordinated, and integrated to promote efficiency and effectiveness. • Organizational culture seeks quality improvement, and assessments are accepted as contributors. • Assessment processes and results support the goal of protecting people and the environment. • Quality problems (including environment, safety, and health issues) are identified for resolution.

DOE G 414.1-1B 3 (and 4) 9-27-07 2. APPLICATION All DOE products and services, and the programs, systems, and processes that deliver them can be assessed over their entire life cycles. DOE directives require integration of environment, safety and health (ES&H), safeguards and security, and emergency management assessments to ensure that DOE and its contractors perform assessments as stated in 10 CFR 830, Subpart A; DOE O 414.1C; DOE M 450.4-1; and DOE O 226.1A. Examples of functional areas for assessments are listed in Appendix B. This Guide expands upon the details of the assessment criteria discussed in DOE G 414.1-2A, Quality Assurance Management System Guide, dated 6-15-05. The Quality Assurance Management System Guide also describes the relationship between quality assurance (QA) and integrating the safety management system requirements. DOE G 450.4-1B Integrated Safety Management System Guide (Vol. 2, Appendices D and G), dated March 1, 2001, describes the role of assessment in the feedback and improvement of safety management functions. DOE line managers fulfill their safety responsibilities in part through line management ES&H oversight and have unfettered access to information and facilities in accordance with safety and security requirements. Contractor line managers fulfill safety responsibilities in part through the implementation of their assessment programs. Contractors are responsible for establishing robust, rigorous, and credible ES&H, safeguards and security, and emergency management assessment programs, integrated with their safety management systems (DOE O 226.1A, DOE O 414.1C and DOE M 450.4-1). Assessment programs conducted in accordance with this Guide, and appropriately adopted standards will satisfy the assessment requirements of 10 CFR 830, Subpart A; DOE O 414.1C; DOE O 226.1A; and DOE M 450.4-1. The Guide, however, does not lessen the requirement to comply with DOE O 414.1C or 10 CFR 830, Subpart A, including those requirements related to management and independent assessments. Alternative methods demonstrated to achieve adequate levels of safety and quality may be acceptable to DOE. This Guide also provides a basis for determining the adequacy of QA programs and Integrated Safety Management System (ISMS) descriptions (DOE M 450.4-1 and DOE O 226.1A). DOE G 414.1-1B 5 9-27-07 3. GENERAL INFORMATION 3.1. Assessment Program Expectations The development of effective assessment and management programs should focus on achieving the following DOE expectations: • Documented assessment programs define the systems that will be used to plan, perform, and follow up on assessments. • Responsibilities are defined for both performing and responding to assessments. • Senior management endorses and actively supports the assessment program and views it as a value-added process. • Management at all levels is responsive to any identified issues, regardless of how they are identified. • Individuals performing assessments are appropriately trained and qualified in the assessment process. • Objective evidence is available to substantiate assessment findings. • Corrective actions are taken promptly to prevent recurrence. • Feedback is solicited from management, workers, independent evaluators and customers. • Measurable organization goals and objectives are identified and progress can be demonstrated. 3.2. Assessment Benefits

The success of an organization depends upon the extent to which its products and services satisfy customer requirements and expectations. Each member of an organization is responsible for customer satisfaction. The results-oriented quality program described in 10 CFR 830, Subpart A, Quality Assurance Requirements (QA Rule) and DOE O 414.1C, Quality Assurance (QA Order) focuses on customer requirements and expectations, and embraces continuous improvement. Assessments build confidence that organizations can meet customer expectations and self-identify areas where improvement is needed and correct problems before they become major issues or events. Effective internal assessments prepare an organization for external governmental and nongovernmental assessments of performance, and conformity with national and international standards. 6 DOE G 414.1-1B 9-27-07 Voluntary third-party conformity assessments are conducted by quality/environmental management system registrars, laboratory accrediters, and product certifiers who evaluate compliance with national or international standards. DOE contractors that participate in the Voluntary Protection Program can attain recognition for excellence in safety and health management. Other national and state quality awards include the Presidential Award for Quality and the Malcolm Baldrige National Quality Award, which use comprehensive assessments that focus on integrated management systems and customer service. Regulatory or oversight bodies such as the Nuclear Regulatory Commission (NRC), the Environmental Protection Agency, or the Occupational Safety and Health Administration (OSHA) measure compliance with regulatory requirements, standards, and related commitments (involuntary third-party assessments). For the NRC and OSHA this represents a relatively small number of DOE facilities. The benefit of an involuntary third-party assessment is the confirmation of compliance or noncompliance with regulatory requirements. An effective assessment process, when coupled with prompt improvements and corrective actions by management, may be considered a mitigating factor by third party assessors. Additionally it can reduce the time and frequency of third-party assessments. This translates into lower conformity assessment service costs for the organization. 3.3. Graded Approach This Guide and the technical standards referenced herein should be applied using a graded approach. The graded application is dependent upon the hazards and/or level of risk associated with the activity or structures, systems, and components under consideration. The scope, depth, and rigor of assessments should be determined by the use of a grading process before performing the activity. The purpose of grading is to select assessment activities consistent with their importance to safety, cost, schedule, and success of the program. The grading process provides the flexibility to design an assessment that best suits the facility or activity. The grading process is not used to obtain exemption from the requirements of the QA Rule or QA Order. Items, services, or programs that contribute the greatest risk to quality, safety, and mission are assessed with the greatest rigor and frequency. The grading process is used to determine the appropriate assessment rigor and is based on activity-specific or facility-specific factors such as: • relative importance to safety, safeguards, and security; • magnitude of any hazard or risk involved;

• life-cycle stage of a facility or activity; • impact/consequences on the programmatic mission of a facility; • unique characteristics of a facility or activity; DOE G 414.1-1B 7 9-27-07 • nuclear safety classification or hazard category of the item or activity; • adequacy of existing safety documentation; • relative importance of radiological and non-radiological hazards; • complexity of products or services involved; • performance history of a facility or activity; and • other relevant factors. 3.4. Purpose of Assessment Establishing and implementing an effective assessment program is an integral part of every management system. Assessment is an important feedback mechanism that provides management with information to evaluate and improve the following: • organizational progress in reaching strategic goals and objectives; • adequacy and implementation of management programs for mission achievement; • performance capability of ISMSs; • products and service quality; and • regulatory and contractual compliance. Simply stated, an assessment is an opportunity to: • identify the gaps between where you are and where you want to be; • identify the reasons for the gaps; • identify the actions that will be taken to close the gaps (corrective actions); • close the gaps between where you are and where you want to be; and • verify that corrective actions have been effective and lasting. 3.4.1. Assessment Elements An assessment program should have the following elements: • defined roles, responsibilities, authorities, and accountabilities for the staff performing and responding to assessments; • documented description, defining the purpose and the processes that will be used to plan, perform, and follow up on assessments; 8 DOE G 414.1-1B 9-27-07 • use of Safety Information Programs (SIPs) to plan and direct assessment resources on ES&H issues; • both management and independent assessments; • performance metrics that reflect the assessment process itself (e.g., customer satisfaction, value of assessment topics, and findings); • a process to periodically evaluate the effectiveness of site assessment programs (both independent and management) in meeting regulatory and management objectives; • clear links to measurable organizational goals and objectives; • inclusion of appropriate technical expertise within the assessment team; • a training program to assure that assessment participants have the proper skills to perform assessments; and • assessment results that feed the corrective action system. The assessment program should be included as part of the Integrated Safety Oversight Plans (ISOP). 3.5. Types of Assessment Assessments are tools for improvement. DOE’s QA Rule and QA Order establish distinct requirements for management and independent assessments. DOE O 226.1A refers to contractor self-assessment programs that include line and independent evaluations. In this context, the assessments are those that a contractor conducts on its own ES&H performance. Management and independent assessments as described in this Guide will satisfy the requirements of DOE O 226.1A. Contractors should clearly describe in writing how their self-assessment programs satisfy the requirements for management and/or independent assessment. Management and Independent Assessments may be performed on the same functions or organizations; however, each has a specific focus defined by the QA Rule and QA Order as described below. For some organizations, the only difference between Management and Independent Assessments may be the actual performer of the assessment.

It is essential to clearly define the criteria and/or objectives intended for the assessment through the assessment planning process and in the Criteria Review and Approach Documents (CRADs). 3.5.1. Management Assessment Managers perform management assessments to comply with the QA Rule and QA Order and to improve performance. In general, the purpose of this type of assessment is to identify the DOE G 414.1-1B 9 9-27-07 management systems, processes, and programs that affect performance and to make improvements. Management assessments look at the total picture: • how well the management systems and processes meet the customer’s requirements; • compliance with standards and requirements; • meeting the expectations for safely performing work; • clarity of the organizational mission, goals, and objectives; and • identifying and correcting problems that hinder the organization from achieving its objectives. The emphasis of management assessment is on issues that affect performance, strategic planning, personnel qualification and training, staffing and skills mix, communication, cost control, organizational interfaces, and mission objectives. Management assessment is a periodic introspective self-analysis to determine whether the organization’s activities are properly focused on achieving desired results. This includes reviewing the processes, systems, and programs that are important to the organization’s mission and objectives. Results of management as well as independent assessments can be used, in addition to formulating approaches and corrective actions for improvements, to develop plans for the subsequent management assessments. Additionally, independent assessment results may also be used as the basis for determining the focus and frequency of management assessments. It should be noted that effective management assessments could result in less frequent independent assessments, and independent assessment findings could affect the frequency and rigor of management assessments. In general, management and independent assessments are complementary; however, management assessments are generally performed at a greater frequency and cover a broader spectrum than independent assessments. 3.5.2. Independent Assessment An independent assessment may be an audit, surveillance, “for cause” review or inspection conducted by individuals within the organization or company but independent from the work or process being evaluated, or by individuals from an external organization or company. In general, the purpose of this assessment is to perform the following: • evaluate compliance with standards and requirements; • evaluate the performance of work; • measure the quality of the item or service; • examine process effectiveness/adequacy; and • promote improvement. 10 DOE G 414.1-1B 9-27-07 Figure 1 The organization or staff performing independent assessments should have sufficient authority and freedom from the line organization to carry out its responsibilities. Individuals should be technically qualified and knowledgeable in assessment techniques and in the areas being assessed. Independent assessments evaluate the performance of work processes with regard to requirements, compliance, and expectations for safely performing the work and achieving the goals of the organization. The focus of independent assessments should be the items and services produced and their associated processes. Thus, management receives an objective view of the assessed activity. Independent assessments are typically performed less frequently than management assessments but go into greater depth.

Management is responsible for developing and implementing a coherent plan that balances management and independent assessments and other forms of feedback and improvement to satisfy the requirements of the QA Rule and QA Order. 3.6. Organizational Activity Levels To develop a comprehensive assessment program that optimizes the application of each assessment type, it may be helpful to visualize the organization as having three interlinked levels of activity (Figure 1). This is not meant to imply a hierarchy of assessments. For this discussion, these levels are referred to as “process,” “system,” and “program.” Management and independent assessments can be applied at all three levels, but examine different aspects of each level. A process is a collection of steps or actions that yield some intermediate outcome. A system is made up of two or more processes that may operate independently or interdependently to yield a complete product or service. A program is the most complex level, and consists of multiple interdependent systems that often require several interfaces to provide the desired product or service. 3.6.1. Process Level Assessments Process level assessments involve the examination of work controls, and verification that they are being implemented effectively, and should assess the effectiveness of the processes from a quality and customer satisfaction perspective. This level of assessment is critical for ensuring that the worker, the public, and the environment are protected from harm. DOE G 414.1-1B 11 9-27-07 At the process level, assessments are performed to verify compliance with procedures and process objectives or criteria, and to ensure that work-control documents (e.g., procedures, instructions, radiation surveys, permits, and safety checklists) accurately reflect tasks and their associated hazards. 3.6.2. System Level Assessments System level assessments focus on whether the appropriate leadership and support systems are in place to enable the implementation of work processes, and may range from informal daily performance oversight to formal periodic evaluations using established protocols. System level assessments are performed to ensure that human and material resources are being properly applied to achieve an organization’s mission and objectives. The collection of “processes” that have been assembled to form an effective “system” is also evaluated. At the system level, assessments are performed to determine whether all the necessary elements and interfaces are addressed, and to ensure that the system is capable of consistently meeting requirements and customer expectations. For example, a management assessment of the work control system might identify cost and resource allocation issues that affect the system. Some of the elements within the work control systems that might be assessed are: • planning the work; • identifying hazards associated with the work; • identifying hazard controls; • scheduling and performing work; • verifying/testing completed work; • critiquing work processes and results; and • documenting the work performed. 3.6.3. Program Level Assessments Program level assessments are used to determine whether the overall organizational programs are properly established and implemented, and are used to evaluate complex organizations from several perspectives. They usually examine the integration of the systems designed to achieve organizational goals and customer expectations (with an emphasis on ES&H factors).

At the program level for example, a maintenance management program, which relies on the work control system, would use results from the process (i.e., work control documents) and system level assessments to determine the effectiveness of the entire maintenance program. This program level assessment could be performed as either a management assessment or an independent assessment. 12 DOE G 414.1-1B 9-27-07 3.7. Assessing for Compliance and Performance There are two different methods commonly used for accomplishing assessments. These are usually known as compliance assessment and performance-based assessment. While each method has distinct characteristics, a good assessment will usually gauge, at some level, effectiveness of the processes, systems, and programs in meeting the mission and objectives of the organization. In practice, an assessment is likely to include both compliance and performance based methods. 3.7.1. Compliance Assessments Compliance assessments focus on verifying compliance with requirements through the implementation of procedures, and begin with a determination of the contractual and regulatory requirements governing the assessed organization. Assessors should become familiar with requirements and procedures and then verify that requirements flow down to implementing documents such as procedures, whose implementation is in turn verified. Assessing for compliance alone may not adequately identify higher-level systemic or programmatic problems or determine the effectiveness of the program. For example, an organization may have written procedures that appear to implement the requirements, however, in practice the intent of those requirements may not be fully achieved because of variables such as poorly executed procedures. 3.7.2. Performance-Based Assessments Performance-based assessments take a different approach by focusing first on the adequacy of the process that produced a product or service, and then on the product itself. If problems are found in the product or work processes, the assessor evaluates the methods and procedures used to implement the applicable requirements in an effort to find the failure that led to the problems. The assessor is expected to determine whether a non-compliance or series of non-compliances with procedures could result in a failure to satisfy top-level requirements. Results of prior compliance assessments may help the assessor in determining the focus areas for planning performance-based assessments. In performance-based assessments, great emphasis is placed on getting the full story on a problem before coming to a conclusion. If an assessor sees a problem with the execution of a welding process, the next step should determine the extent of the problem. Is it limited to one welder? Is it limited to one process? Can the problem be traced to the qualification program for the welder or to the qualification program for the welding process? Or is there a problem with the weld material itself, indicating a problem such as engineering or procurement? While the assessor should be familiar with requirements and procedures, in performance-based assessments the assessor’s experience and knowledge play an integral part in determining whether requirements are satisfied. Therefore, participants in performance-based assessments should be technically competent in the areas they are assessing. For example, if an assessor is evaluating a welding process, the assessor relies heavily on his or her knowledge of welding

DOE G 414.1-1B 13 (and 14) 9-27-07 codes, welding processes, and metallurgy, rather than just verifying simple procedure compliance. Performance-based assessments usually provide the most useful information to management; however, it requires a much higher level of competence on the part of the assessment team. Results of performance-based assessments may provide useful insight for management’s pursuit of excellence. DOE G 414.1-1B 15 9-27-07 4. GUIDELINES Organizations should establish procedures for planning and performing management and independent assessments. These procedures should be a part of the overall assessment program for the organization. They should address personnel training and qualifications, the assessment planning processes, performance protocols and tools, reporting, including report distribution, corrective action development and implementation, and other follow-up activities. The following guidelines are presented to assist organizations in developing their procedures and protocols. Management assessments share procedural and protocol commonalities with Independent Assessments. Because of this, the organization should ensure that assessment procedures are well defined and integrated, while maintaining the separate and unique focuses of the two types of assessments. For example, management and independent assessments look at the results of internal and external assessments to determine compliance with defined system requirements. Management assessments; however, also need to focus on how well the system is meeting organizational objectives and achieving improvement goals. 4.1. Assessment Personnel Assessment personnel facilitate continuous process improvement by identifying ways programs, systems, and processes can be improved and by providing information to management and process owners. The assessor should be able to collect performance data through interviews, document reviews, observation, and inspection. It is very important that the assessor also be able to communicate effectively, both orally and in writing, and demonstrate effective interpersonal skills. Both management and independent assessments should be performed by qualified individuals who are knowledgeable about the program, system, or process being assessed and have been trained to ensure full understanding of the assessment processes, including reporting. Individuals performing independent assessments should not currently perform, supervise, or be directly responsible for performing the activities being assessed. Independence is determined based on an individual not having bias, rather than on organizational affiliation. The independent assessor should have both the personal and organizational freedom to communicate with the management of the assessed organizations. Organizations should establish a formal training and qualification programs for assessors, including both assessment team leaders and team members, that reflects both regulatory and customer requirements. Organizations may adopt third-party personnel qualification programs such as the American Society for Quality’s “Quality Auditor Certification” (http://www.asq.org/certification/index.html) or the Registrar Accreditation Board’s certification program (http://www.rabqsa.com/). The International Organization for Standardization and the American Society of Mechanical Engineers (ASME) provide additional guidance for training and qualification of assessors (ISO-19011 and ASME NQA-1 respectively). For assessments of nuclear facilities and activities, ASME NQA-1 is the appropriate national standard to be used by DOE and contractor organizations for guidance on training of assessment personnel. At a

16 DOE G 414.1-1B 9-27-07 minimum, however, training and qualification programs should be based on a recognized, relevant standard such as (DOE-STD-1150-2002, Quality Assurance Functional Area Qualification Standard). Effective assessments may be accomplished through the use of an assessment team with combined skills and experiences. Training for assessors should address the policies and procedures of the assessing organization. To enhance assessment performance and capability, new assessment personnel should participate in on-the-job training with qualified, experienced assessors before being considered fully trained or receiving a required qualification. Further guidance on assessor training and qualification is provided in DOE Orders and Guides, and the standards in Appendix A. 4.2. Assessment Program Planning 4.2.1. Assessment Programs Assessment programs should be developed to the level of rigor and detail required to ensure adequate review of programs, systems, and processes. An assessment program provides the structure for the overall process and ensures that assessments are conducted in a cost-effective, efficient manner. Items considered essential for a comprehensive assessment program include the following: • assessment scheduling, planning approach, and logic (including how management and independent assessments are balanced); • methodology for determining/developing performance criteria; • recognition and use of third-party assessment results (accreditation, certification, registration, and regulatory); • assessment ethics and behaviors; • qualification and training of assessment team personnel; • protocols for conduct, including interfaces and meetings; • format/review of assessment plans and agendas; DOE G 414.1-1B 17 9-27-07 • reporting methods/procedures for concerns, findings, observations, and improvement opportunities, including distribution, and mechanisms for addressing imminent danger issues; • procedures/processes for concern verification and follow–up; • assessment records management program, including identification of records that will be retained, retention periods, and protection; and • central repository or website to ensure that assessment supporting documentation (e.g., assessment plans, corrective action plans, transmittal letters, etc.) can be retrieved. Assessments should be conducted commensurate with the hazards, status, and importance of the program goals, systems, or work processes and should be focused on worker health and safety, public health and safety, environmental protection, community concern, strategic planning, organizational resources, compliance and liability, and business efficiency and productivity. Complexity, reliability, risk, and economic issues should also be considered when planning and scheduling assessments. The application of a graded approach using a risk-based decision-making process will ensure that resources are applied in a manner that provides the greatest benefit to the assessed organizations and their customers. 4.2.2. Management Assessment Planning Management assessments should be planned in a systematic manner by the individual managers at each level of management, to evaluate the effectiveness and adequacy of their management systems. Assessments should be planned with appropriate consideration for other management and/or independent assessments that could conflict with or duplicate their efforts (see Section 4.3). Management should retain overall responsibility for the planning and performance of management assessments, and the results of the planning process should be documented in an assessment plan. The following items should be evaluated:

• how well management is providing the leadership to enable an organization to continuously meet internal and external customer requirements and expectations; • processes and their effectiveness, internally and across organizational boundaries, including ensuring that the staff is receiving the support to do their work (training, procedures, tools, and cooperation from others; and • performance information from other assessments (Independent, Self, and External Assessments including financial, employee feedback systems, customer surveys, etc.). Management assessment planning should include the mechanics of performing the assessment such as the expected time-frame, assessment tools that may be used, reporting requirements, and how areas for improvement will be identified, tracked, and closed (see Section 4.8 for additional information). 18 DOE G 414.1-1B 9-27-07 4.2.3. Independent Assessment Planning The level of planning for an independent assessment will vary significantly depending on scope, breadth, and complexity of the system or process being assessed. The planning process should provide for input from the assessed organizations and consideration of their customers’ and stakeholders’ needs, requirements and expectations. Specific budget requirements and required support should be identified as early in the planning process as possible. Administrative issues that need to be anticipated and addressed prior to the site visit typically include: • requests for documentation; • travel planning; • security clearances; • access privileges; • facility authorizations; • site logistics; and • points of contact. There are various scheduling and planning approaches to satisfying these requirements. Appendix C describes tools to aid in assessment planning and Appendix D provides an example of how to plan and schedule independent assessments using an integrated, risk-based approach. 4.2.4. Planning Updates Required assessments should be planned and completed within a reasonable period of time. Assessment planning should be reviewed and modified periodically as new information about the facility or organization is obtained. An assessment that finds good performance should be used as justification for reducing the depth and frequency of future assessments. The frequency of assessments may also be reduced based on good performance unless the frequency is established by a requirement (e.g., DOE O 151.1C). Areas of poor performance should receive increased attention, especially if there are indications that management is unable to correct the identified problems, because recurring and cumulative deficiencies, even in low-hazard operations, may decrease the likelihood of successfully achieving the mission. 4.3. Assessment Integration Routine communication and trust among the various assessment bodies is essential when coordinating assessment plans. This task involves identifying overlapping and redundant assessments to reduce their negative impact on work performance. Once identified, assessments should be coordinated and consolidated in several ways. DOE G 414.1-1B 19 9-27-07 • Assessment scope should be coordinated and integrated to prevent multiple assessment organizations from performing similar assessments on the same subject. • Separate assessment teams may be combined to evaluate the subject in a single visit. • Assessments may be canceled based on agreements to share the results of other assessments.

Where significant redundancies exist, as when contractors perform Supplier Quality Assessments of analytical laboratory services, the recognition and use of accredited third-party assessment results and/or shared information from other contractors should be considered. Each of these methods requires staff time to develop, implement, and manage; however, they can result in substantial resource savings while enhancing the ability of the assessing organization to cover a greater number of activities. Having an integrated assessment schedule including management and independent assessments will assist the organization controlling overlapping and duplicate assessments to minimize impact to organizations. Additionally, consider developing a multi-year assessment schedule (along with a current year schedule) to identify the total scope of all required assessments. This multi-year schedule could be utilized in a rolling fashion. This will facilitate resource identification, resource leveling, and allow for reconciliation from year-to-year. Assessment schedules are meant to be living documents that are adjusted periodically based on the analysis of operational awareness information, and the results of internal and external assessment activities. 4.4. Assessment Plan The use and level of detail of an assessment plan will vary depending on what is being assessed, and whether the assessment is a management or independent assessment. Plans are used to scope and plan individual assessments, and should include input not only from the assessed organizations but also from their customers. A documented assessment plan not only allows expectations to be communicated to the assessed organization, but also allows the assessment team to focus its activities more effectively. The scope of the assessment should be defined in terms related to the assessed organization’s mission and goals so the focus and value of the assessment will be clearly understood. The level of detail included in the assessment plan should be commensurate with the protocols of both the assessed and assessing organizations. The assessment plan should include the following items: • team members and their qualifications and biographies; • description of the assessment scope and performance criteria; • dates of the assessment; • schedule of assessment meetings-pre-assessment, daily, and post-assessment; • list of documents to be provided to the assessment team upon arrival; • requests for office space, phone lines, computers, and other administrative support as required; 20 DOE G 414.1-1B 9-27-07 • requests for site-specific training, dosimetry, and access requirements; and • request for points-of-contact for each functional area. The organization being assessed should be contacted and provided with the assessment plan as early as possible before the assessment (unless regulations or contracts specify other contact protocols). Management or independent assessments in smaller organizations may not involve as much information as that listed above, nor the degree of formality required for assessments performed by external organizations. It is important to remember that each assessment needs to evaluate a program, system, or process during a finite time period. 4.5. Performance Criteria

Assessments seek to ensure that performance expectations defined by management and process owners are being met. Assessors should clearly understand the programs, systems, or processes being assessed, including their goals, and associated objectives and requirements for efficient, effective performance of operations. Performance requirements can be found in the following source documents: • federal and state regulatory requirements; • appropriate codes and standards; • contract requirements; • DOE Orders, Manuals, and Notices; • implementation plans; • implementation procedures; • facility safety documents; • policy and mission statements; • DOE-approved “Work Smart” standards ; • standards/requirements identification documents (S/RIDs); • plans and programs; • management, business, operating and/or strategic plans; and • applicable standards, permits, authorizations and regulatory agreements. DOE G 414.1-1B 21 9-27-07 Much information about performance and additional performance requirements may be available to assessors in existing documents and reports, such as: • reports from outside regulators; • facility operations/activity/metrics reports; • performance reviews; • previous assessment reports, including self-assessment reports; • internal inspections, reviews, and reports; • corrective action plans and status reports; • concerns and occurrence reports; • performance indicators; • monitoring and survey data, and modeling data and analyses; and • Price-Anderson Amendments Act Nonconformance Tracking System reports (http://www.hss.energy.gov/enforce/). Requirements contained in these documents are selected based upon impact on the assessed organization’s mission and the relationship to the scope of the assessment. From selected requirements, objective statements (performance measures) are developed for determining whether a program, system, or process is working efficiently and effectively. From these measures, the specific performance criteria (based on written programs, DOE Orders, Rules, etc.) are developed and tools selected for conducting the assessments. In developing performance criteria, assessment personnel should not reinterpret or redefine requirements specified in the source documents. It is critical for a successful assessment that the requirements are understood and clearly explained in the assessment documentation. 4.6. Assessment Planning Tools Assessment planning tools such as checklists are an essential element of an effective assessment. They vary in format, content, and level of detail, but all have one thing in common: they help focus the assessor on the mission and objectives of the program, system, or process being assessed. Application of planning tools before an assessment ensures that time will be used effectively and that the assessment’s focus is identified and maintained. Assessment planning tools are often used to relate the performance criteria to the established assessment scope and may include lists of interview questions, major elements of programs, or detailed process work steps. Similar to a road map, each tool is used to remind the assessor of where he/she is going and the items likely to be encountered along the way. Planning tools are extremely useful when the assessment basis is complex or the requirements come from multiple sources. Typical planning tools include matrices, flowcharts, cause-effect diagrams, tree diagrams, checklists, 22 DOE G 414.1-1B 9-27-07

CRADs, and information systems. (See Appendix C for examples and further discussion of these tools. An example of a QA CRAD can be found at http://www.hss.energy.gov/CSA/CSP/qa/QA_CRAD.html) 4.7. Independent Assessment Process As discussed previously, there are commonalities shared by management and independent assessments. To avoid unnecessary duplication, the following sections of this Guide are very detailed, pointing out areas where the two assessment types differ. Section 4.8 is a briefer discussion concentrating on elements unique to management assessments. Appendix D provides additional insight and guidance for independent assessment planning. 4.7.1. Pre-assessment Meetings The effectiveness and efficiency of independent assessments can be improved greatly if representatives of the assessment team meet informally with key members of the assessed organization. This should be done at least a week before the assessment fieldwork actually begins. Pre-assessment meetings are particularly useful when the assessment team is completely external to the organization being assessed. It is recommended that these meetings be used to confirm that there is a common understanding between the assessing organization, the assessed organization, and the customer of the independent assessment (DOE program or field office management), of the criteria and/or performance measures. These meetings are also an opportunity to clarify the assessment team’s plan and to work out logistical problems. The assessment team should identify and schedule the individuals to be interviewed so that arrangements can be made to ensure their availability. Documents that require extra effort to retrieve can be identified in advance, and the requirements for assessment team work space can be resolved. 4.7.2. The Entrance Meeting An entrance meeting involving personnel from the assessing organization and the managers of the organization being assessed, is held immediately before the assessment fieldwork begins to “set the stage” for a positive and productive independent assessment. This meeting is usually held at the assessed organization’s location/facility and allows the assessment team to meet the assessed organization’s managers and answer any questions they may have about the assessment. This meeting is also used to establish how concerns involving imminent danger or regulatory non-compliance will be communicated. The protocols to be followed during the assessment should be clarified during the entrance meeting, including discussions of the following: • purpose and scope of the assessment, including authority for conducting the assessment; • assessed organization’s mission, program, systems, and processes; • scheduled duration of the assessment; DOE G 414.1-1B 23 9-27-07 • source documents and performance information that form the basis for the performance criteria to be used; • a list of knowledgeable individuals from the assessed organization as “points of contact” for each assessor; • any restrictions on the collection and/or disposal of assessment notes/records by the assessors; • logistics, including work area, working hours, lunch hours, etc.; • time and location of periodic status meetings; and • time and place of the post assessment meeting. 4.7.3. Performing Independent Assessments

The assessment should be conducted in accordance with established protocols developed by the assessing organization. Site/facility protocols should be followed for what to do if an imminent danger situation or a reportable noncompliance or violation is encountered during the course of an assessment. Any assessment schedules or specific protocols established during the pre-assessment meeting are used to ensure that the assessment is conducted effectively and safely. Assessors should keep their points of contact informed of their activities to preclude surprises during the post-assessment conference. This may include requests for additional assistance or the communication of concerns that require immediate action on the part of the assessed organization. Timely communication, verbal and written, will allow the assessed organization to verify the accuracy of observations and provide relevant facts and background on the issues. One way to accomplish this is to meet periodically (daily or every other day) with the organization being assessed to convey questions/concerns and provide a status update. Daily team meetings may be helpful in ensuring continuity and overall focus by providing assessment team leaders with information about the completion status of the assessment checklists, and offering the opportunity for inquiry into issues requiring additional action (e.g., clearances, access, requests for personnel or material, and impasse resolution). These meetings also provide the setting for advising other team members of issues that may be of interest in their assigned scope, or for integrating data gathered by the various assessors. The meetings should be brief so that they do not significantly reduce the team members’ field time with the processes they are to assess and the people they are to interview. It is important that sufficient information be gathered during the assessment to determine whether an activity meets the performance criteria established. The assessor should be able to state clearly the criterion impacted by the activity and whether identified findings impact the mission/goals of the organization. To accomplish this, the assessor may deviate from the assessment schedule to determine the extent and significance of an issue. Deviations that affect 24 DOE G 414.1-1B 9-27-07 the assessor’s ability to complete the assessment team’s interview schedule should immediately be made known to the organization being assessed and the team leader. 4.7.4. Independent Assessment Techniques Effective assessments use a combination of tools and techniques to maximize the productivity of the assessment team and resources. Such assessment techniques include interviews, document reviews, observation, inspection, and performance testing. The planning tools discussed in Appendix C also allow for more complex analysis and systematic coverage of the areas being assessed. In using these techniques, the assessor should not forget that the objective is to verify accomplishment of an organization’s mission. To save time, the assessor should gather only data and information relevant to overall program performance and the achievement of program objectives. Assessments should be thorough and information gathered with sufficient diligence such that accurate, detailed conclusions can be provided to the organizations that will receive the final report.

When using any of these techniques, assessors should maintain good records of the assessment results. These may include personal notes or other information to support the assessment, and may be included in the checklist information. These records are useful in writing the report, and any associated findings and recommendations, and will be valuable if questions arise during the report review process. All classified notes should be disposed of properly in accordance with established and agreed-upon procedures. A discussion of each of the techniques follows. 4.7.4.1. Interviews provide the means of verifying the results of observation, document review, inspection, and performance testing; allow the responsible person to explain and clarify those results; help to eliminate misunderstandings about program implementation; and provide a venue where apparent conflicts or recent changes can be discussed and organization and program expectations can be described. 4.7.4.2. Document Reviews provide the objective evidence to substantiate compliance with applicable requirements. A drawback is that the accuracy of the records cannot be ascertained by review alone. This technique should be combined with interviews, observation, inspection, and/or performance testing to complete the performance picture. Records and documents should be selected carefully to ensure that they adequately characterize the program, system, or process being assessed. 4.7.4.3. Observation, the viewing of actual work activities is often considered the most effective technique for determining whether performance is in accordance with requirements. Assessors should understand the effect their presence has on the person being observed and convey an attitude that is helpful, constructive, positive, and unbiased. The primary goal during observation is to obtain the most complete picture possible of the performance, which should then be put into perspective relative to the overall program, system, or process. 4.7.4.4. Inspections are performed in accordance with acceptance criteria to verify the condition of physical facilities, systems, equipment, and components. DOE G 414.1-1B 25 9-27-07 4.7.4.5. Performance Testing is used to observe the response of personnel or equipment by creating a specific situation and noting the resulting performance. This technique is especially helpful when activities of interest would not normally occur during an assessment visit. It is also useful when the timeliness and appropriateness of the response are critical (e.g., emergency responses). 4.7.5. The Exit Meeting This meeting is used primarily by the assessment team to present the assessment summary. Reasonable time should be allowed to discuss any concerns, but this meeting should not be used to argue the assessment findings or methodology. There should be no surprises during the exit meeting since the assessment team should have taken every effort possible during the conduct of the assessment to ensure that the assessed organization was aware of the team’s findings and concerns. Prior to the exit meeting the assessment team should consider combining related findings into a small number of well-supported findings to help focus management’s opportunities for improvement. 4.7.6. Assessment Reporting

Assessment reports are required for documentation of assessment results. Assessment team leaders have the overall responsibility for preparing the report and obtaining appropriate approval for its release as applicable. The report may be formal (e.g., distributed by memorandum) or informal (e.g., letter to file or email), depending on the level of assessment performed, but should provide a clear picture of the results in terms of the programs, systems, and processes assessed. The assessment report should be clear, concise, accurate, and easy to understand, and should include only facts that directly relate to assessment observations and results. It should include sufficient information to enable the assessed organization to develop and implement appropriate improvement plans. Specific report formats may vary considerably from one organization to the next. An independent assessment report usually includes the sections described below. (Note: A management assessment report may not require all of the content listed below and may only require an executive summary.) 4.7.6.1. Assessment Report Content • executive summary; • assessment scope; • identification of team members; • identification of personnel contacted; • documents reviewed; • work performance observed; 26 DOE G 414.1-1B 9-27-07 • assessment process and criteria (e.g. CRADs); and • results of the assessment including identification of areas for improvement, and/or strengths. 4.7.7. Releasing and Responding to Assessment Reports While team leaders have overall responsibility for the report, the entire assessment team should have an opportunity to read and sign the completed report. At a minimum, the final report should be distributed to the management of both the assessed and assessing organizations. Distribution to other organizations (e.g., Defense Nuclear Facilities Safety Board or other regulators) should be defined during the planning phase and communicated in advance to the assessed group. Because the true value of an assessment is the improvement opportunities it identifies, and its value typically diminishes over time, the best time to release a report is immediately after the post-assessment meeting, which allows the assessed organization to begin improvement actions, yielding the maximum return for those actions. The assessment report or transmittal correspondence should clearly indicate what response is expected from the assessed organization and a reasonable response date. 4.7.8. Corrective Actions Managers responsible for the activities assessed are also responsible for the development of effective corrective actions for the problem areas/deficiencies discovered during the assessment. At a minimum, these corrective actions should include the following: • measures to correct each deficiency; • identification of all root causes for significant deficiencies; • determination of the existence of similar deficiencies or underlying causes (i.e., extent of condition, extent of cause); • actions to preclude recurrence of like or similar deficiencies; • assignment of corrective action responsibility; and • completion dates for each corrective action. Managers should verify that corrective actions are likely to fully address the identified deficiency and when actions are completed, validate that the actions have corrected the deficiency. Specific, detailed requirements (DOE O 414.1C and DOE O 470.2B, Independent Oversight and Performance Assurance Program) and guidance (DOE G 450.4-1B, Integrated Safety Management System Guide, Vol. 2, Appendix G, and DOE G 414.1-5, Corrective Action

DOE G 414.1-1B 27 9-27-07 Program Guide) exist for responding to assessments conducted by the Department’s independent oversight organization. 4.7.9. Follow-up Assessments A follow-up assessment with special focus may be performed and should be completed in accordance with applicable corrective action documents. Particularly, this follow-up assessment should evaluate the effectiveness of corrective actions. A reasonable subset of corrective actions should be reviewed for effectiveness Several DOE special focus assessments also include verification steps: DOE G 450.4-1B, Integrated Safety Management System Guide, Vol. 2, Appendix G; DOE-STD-3006-2000, Planning and Conduct of Operational Readiness Reviews (ORR); and Office of Independent Oversight Environment, Safety, and Health, Safeguards and Security, and Emergency Management Appraisal Process Protocols. See Appendix A, “Consensus Standards and References.” 4.8. Management Assessment Process Management assessments are a tool used by managers to identify and resolve issues that contribute to success or failure in fulfilling mission, goals, requirements, or expectations. During management assessments, focus should be on evaluating system performance issues and related processes such as personnel skills, communications, cost control, and other similar processes. Some sites have identified management assessment subject matter experts (SME) typically within the central assessment organization who have responsibility for reviewing the management assessments performed by the various program and division managers within the organization. The SMEs provide feedback as to how well the various management assessments are meeting company expectations, areas for improvement, etc. This helps to alleviate one of the deficiencies typically associated with management assessments; inconsistencies in the scope and effectiveness of such assessments across the organization. Planning management assessments is an organization-specific effort that should be integrated with other assessment processes. No single method is appropriate for every situation. Either quantitative or qualitative assessment methods may be used as appropriate to the assessment scope. Managers are challenged to make the assessment a value-added process that will lead to improvement in organizational performance, safety, meeting customer expectations, and achieving mission goals in full compliance with regulatory and DOE requirements. It is important to remember that while management assessments share some commonalities with audits, they should focus on evaluating organizational performance and identifying barriers that hinder improved performance. The levels of formality and scope of management assessments can vary greatly. For example, at the informal extreme of this range, a manager may evaluate the skills of personnel involved in a single work function and document the need for additional training or staffing levels adjustments through a memorandum. At the formal extreme, an assessment team made up of senior managers and process consultants from outside the organization, can perform an extensive evaluation of an entire organization using checklists, interviews, document reviews, work observations, and 28 DOE G 414.1-1B 9-27-07 objective evidence evaluation, followed by the issuance of a formal report. Appendix D provides additional insight and guidance on management assessment planning, performance and reporting.

4.8.1. Defining the System The organization should have a written description of the management assessment process. The description should address all elements of the process including, but not limited to, the following: • management levels that will be expected to perform assessments; • general goals of the management assessment process; • training or mentoring that will be provided to assessing managers; • expectations for the number and frequency of assessments to be performed; • expectations and guidelines for management assessment reports; • reporting and follow-up processes; • interfaces with other oversight programs and processes (e.g., the corrective action reporting and tracking process); • management assessment planning process; and • administration of the process. 4.8.2. Assessment Scheduling Schedules for management assessments need to be established with the expected frequency of assessment performance specified. However, scheduling of management assessments should be as flexible as possible to meet operational and management needs. The organization should review and update its management assessment schedule on a regular basis, either bi-monthly or quarterly, to ensure relevance. The review should consider the current conditions, conclusions of recent management assessments, inputs for independent assessments, and organizational performance. Management assessments should be planned with input from all levels of management. Some organizations have found it beneficial to schedule the number of assessments, but leave the determination of the topics/areas of assessment to the performing manager, thus providing flexibility and the freedom to perform assessments that will result in the greatest opportunity for improvement. The planning process may include and take credit for the existing management reviews or similar assessments that routinely occur, such as the following: • program reviews; DOE G 414.1-1B 29 (and 30) 9-27-07 • strategic planning sessions; • reviews of performance indicators; • reviews of organizational goals-setting and objective-setting sessions; • financial reviews; • reviews of the outputs of improvement teams; and • reviews of independent assessments. Management assessments may include some benchmarking activities, both internal and against other organizations. The assessment of internal and external performance indicators may also be beneficial. There is no fixed number of assessments to be performed; however, the organization should be able to show that the management assessment program complies with the QA Rule and QA Order. 4.8.3. Performing Management Assessments Management assessments may be performed by individual managers or teams of managers. The primary responsibility for management assessments resides with managers because they are in the best position to identify barriers to improved performance and to effect changes. While some aspects of the assessments, such as collecting information, may be delegated to staff, it is the manager’s responsibility to perform the assessment and determine the conclusions. Personal involvement by the manager will yield the most meaningful information for improving the performance of the organization. Although the primary purpose of a management assessment is not verification of compliance with regulatory or DOE requirements, occasionally a condition is found that is a violation of programmatic requirements. When this happens, the appropriate oversight organizations should be notified and corrective action processes implemented.

Refer to Appendix D for additional guidance on performance methods. 4.8.4. Assessment Reporting Management assessments should be reported in accordance with the organization’s assessment program. As with independent assessment reports, management assessment reports should include a concise summary of the topics or areas assessed, the conclusions reached, and any follow-up actions that may be required. Reports should be available for use by others and for future planning. Special provisions may be required for reports dealing with sensitive areas, proprietary information, or classified information. Assessments identifying potential regulatory compliance issues should be communicated directly to the appropriate managers for any necessary action as well as in the final assessment report. Refer to Appendix D for additional guidance on report content. DOE G 414.1-1B A-1 9-27-07 APPENDIX A—CONSENSUS STANDARDS AND REFERENCES A.1 Consensus Standards The following are some of the consensus standards that provide methods for implementing the guidance contained herein. A single standard may not fully implement all elements of the requirements (particularly for management assessments); therefore, these documents should be used in conjunction with 10 CFR 830, Subpart A, and DOE O 414.1C, Quality Assurance, to develop and implement assessment processes that meet DOE assessment requirements. The organization remains responsible for compliance with 10 CFR 830, Subpart A, and DOE O 414.1C. 1. American National Standards Institute (ANSI)/American Nuclear Society (ANS) 3.2-1994 (R1999), Administrative Controls and Quality Assurance for the Operational Phase of Nuclear Power Plants. 2. ANSI/ANS 15.8-1995 R2005, Quality Assurance Program Requirements for Research Reactors. 3. ANSI/ASQ (American Society for Quality) Z 1.13-1999, Quality Guidelines for Research. 4. ANSI/ASQC (American Society for Quality Control) E4-2004, Quality Systems Requirements for Environmental Data and Technology Programs. 5. ANSI/ISO (International Organization for Standardization)/ASQ Q9001:2000, Quality Management Systems: Requirements. (Note: ANSI/ISO/ASQ Q9000 series documents are recognized as being identical to their ISO 9000 series counterparts.) 6. ASME (American Society of Mechanical Engineers) NQA-1-2000, Quality Assurance Requirements for Nuclear Facility Applications. 7. ASME (American Society of Mechanical Engineers) NQA-1-2004, Quality Assurance Requirements for Nuclear Facility Applications. 8. DOE-HDBK-1101-04, Process Safety Management for Highly Hazardous Chemicals, February 2004. 9. DOE-HDBK-3027-99, Integrated Safety Management Systems (ISMS) Verification; Team Leader’s Handbook, June 1999. 10. DOE-NE-STD-1004-92, Root Cause Analysis Guidance Document, February 1992. 11. DOE-STD-1036-93; Guide to Good Practices for Independent Verification, June 1993 (and Change Notice 1, dated December 1998). 12. DOE-STD-3006-2000, Planning and Conduct of Operational Readiness Reviews (ORR), June 2000. A-2 DOE G 414.1-1B 9-27-07 13. ISO 14012-96, Guidelines for Environmental Auditing–Qualification Criteria for Environmental Auditors, 1996. 14. ANSI/ISO/ASQ QE 19011S-2004, Guidelines on Quality and/or Environmental Management Systems Auditing. May 2004 15. DOE-STD-1150-2002, Quality Assurance Functional Area Qualification Standard 16. DOE-STD-1172-2003, Safety Software Quality Assurance Functional Area Qualification Standard A.2 Other References

The following references provide additional information concerning assessments. 1. DOE O 151.1C, Comprehensive Emergency Management System 2. DOE G 414.1-2A, Quality Assurance Management System Guide for Use with 10 CFR 830.120 and DOE O 414.1. 3. DOE G 414.1-4, Safety Software Quality Assurance Guide for Use with 10 CFR 830.120 and DOE O 414.1 4. DOE/RW-0333P,Quality Assurance Requirements and Description [for the Civilian Radioactive Waste Management Program], Rev. 18, Section 18.0, “Audits,” Office of Civilian Radioactive Waste Management, October 2006 http://www.ocrwm.doe.gov/documents/qard/48027/index.htm 5. DOE G 450.4-1B, Integrated Safety Management System Guide, vols. 1 and 2, dated 3-1-01. 6. DOE, Office of Independent Oversight, Environment, Safety, and Health, Safeguards and Security, and Emergency Management Appraisal Process Protocols, http://www.hss.doe.gov/IndepOversight/docs.html 7. U.S. Department of Commerce, National Institute of Standards and Technology, “Malcolm Baldrige National Quality Award Criteria for Performance Excellence” http://www.quality.nist.gov. 8. DOE O 226.1A, Implementation of Department of Energy Oversight Policy, dated 7-31-07. 9. DOE O 450.1 Chg 3, Environmental Protection Program, dated 1-15-03. 10. DOE O 470.2B, Independent Oversight and Performance Assurance Program. dated 10-31-02. 11. QC-1 Rev. 10, Nuclear Weapons Quality Policy. DOE G 414.1-1B A-3 (and A-4) 9-27-07 12. Quality Assurance Criteria Review and Approach Documents http://www.hss.energy.gov/csa/csp/qa/ 13. Safety Software Quality Assurance Criteria Review and Approach Documents http://www.hss.energy.gov/CSA/CSP/sqa/assessments.htm 14. The Certified Manager of Quality/Organizational Excellence Handbook, 3rd edition: Russell T. Westcott, editor; ASQ Quality Press 2005. DOE G 414.1-1B B-1 9-27-07 APPENDIX B—ASSESSMENT FUNCTIONAL AREAS This appendix lists environment, safety, and health (ES&H) functional areas, as well as safeguards and security and emergency management activities that should be included in a comprehensive assessment program. The list represents basic ES&H requirements, processes, and programs found in regulations, DOE policy, and DOE Orders. This appendix should not be construed as limiting assessments to only these functional areas, nor should it be interpreted as minimizing the importance of assessing product/service quality and organizational performance. It does, however, illustrate the wide variety of ES&H programs, systems, and processes currently implemented by DOE and its contractors. These ES&H functional areas comprise a complex mix of people, hardware, software, and resources, all impacting on management and performance of activities. An integrated assessment program should be adaptable to this mix in order to be responsive to senior management needs and comply with 10 CFR 830, Subpart A, and DOE O 414.1C, Quality Assurance. • Accelerator Safety • Accident, Incident, and Unusual Occurrence Investigation and Reporting Process • Aviation Safety • Biological Hazards • Calibration Control • Chemical Safety (process safety management) • Computer Software Control • Conduct of Operations • Configuration Control • Construction Safety • Criticality Control • Corrective Action • Dissenting Professional Opinion Program • DOE Nuclear Safety Rule Compliance (10 CFR 830) • Document Control and Records • Electrical Safety • Emergency Management • Employee Assistance Program • Employee Concerns System • Engineering Design Processes • Environmental Management Systems • Environmental Protection and NEPA

Compliance • Equipment Modifications • Experimental Programs • Explosives Safety • Facility Operations • Fire Protection • Firearms Safety • Hoisting and Rigging • Identification & Control of Items • Industrial Hygiene • Industrial Safety • Inspection & Test Control • Integrated Safety Management System Implementation • Lessons Learned Processes • Maintenance Management • Motor-Carrier/Vehicle Safety • Non-conformance Control • Nuclear Facility Safety • Occupational Medicine • Occurrence Reporting/Trending • Operational Readiness Review Process • Packaging and Transportation • Performance Measures and Indicators • Pollution Prevention • Pressure Systems • Procurement & Contracts (including supplier control) • Quality Management Systems • Radiation Protection (10 CFR 835) • Reactor Safety B-2 DOE G 414.1-1B 9-27-07 • Safeguards and Security • Safety Management Systems • Safety Basis Documentation (e.g., Bases for Interim Operation, hazard analyses, and safety analysis reports) • Software Quality Assurance • Standards/Requirements Identification Documents(s) • Suspect/Counterfeit Items • Technical Safety Requirements • Training of Nuclear Facility Personnel • Un-reviewed Safety Questions Process • Voluntary Protection System • Waste Management • Waste Minimization • Welding • Work Planning and Control • Worker Protection/Industrial Safety DOE G 414.1-1B C-1 9-27-07 APPENDIX C—TOOLS FOR ASSESSMENT PLANNING AND CONDUCT C.1 Checklists Checklists (Example C.1) are lists of assessment objectives and performance criteria. They usually include a column for the requirements (or references to the requirements) and a column for recording assessment observations/evidence. Checklists are especially useful for organizing assessment time, keeping the assessment focused, and providing a means to list appraisal objectives sequentially. They may also be structured in a form that can easily be converted into an assessment report. Example C.1 Laboratory Calibration Program Checklist Lab/Appraisal Number: Date: Page 1 of Results Reference Criteria Sat Unsat Comments NL-QAM 1. Is monitoring and data collection equipment calibrated? NL-QAM 2. Is equipment calibration traceable to nationally recognized standards? NL-QA-5.1 3. Is equipment calibration performed using approved instructions? NL-QA-5.1 4. Are calibration records maintained for each piece of equipment? NL-QA-5.1 5. Is a use log maintained? In Example C.1, the checklist is used to list the primary elements of a laboratory’s calibration program. The basis or source of each criterion is included in the first column to provide a path back to the requirements document(s). The “comments” column provides a place for the assessor to record additional observations as they are discovered during the assessment, thus helping to ensure that important data is not lost. C.2 Matrices Matrices (Examples C.2 and C.3) are two-dimensional tables showing the relationship between two sets of information. They can be used to show the logical connecting points between performance criteria and implementing or required actions, and personnel responsible for those actions. In this way, matrices are used to determine what actions and/or personnel have the C-2 DOE G 414.1-1B 9-27-07 greatest impact on an organization’s mission. Matrices are especially useful as a way to focus assessment time and organize assessment conduct. Example C.2 Organizational Responsibilities Matrix

Program Development Deficiency Tracking Training Work Control Documents & Records Retention Assessment Director X X Ops Office X X X Ops Support X X X Tech Support X X X X Admin X X In Example C.2, the matrix is used to help the assessor plan the assessment by identifying organizational responsibilities for the different assessment areas. This type of matrix is used to maximize use of assessment time during the site visit. Example C.3 Long-Range Planning Matrix Administration Chemistry Biology Materials Building Services Engineering Industrial Hygiene A A A A Radiological Protection B B B Fire Protection A C C C C Industrial Safety A C A Environmental C A C C Personnel Training B C B C Conduct of Operations C C C Quality Assurance A C A C Example C.3 is a much broader use of the matrix that allows the assessor to do the long-range planning necessary for ensuring proper application of the assessment program. In this example, the various assessment areas (Y axis) are correlated with the different organizations to be assessed. DOE G 414.1-1B C-3 9-27-07 C.3 Flowcharts Flowcharts (Figure C.1) illustrate the steps or activities in a process. They provide an excellent tool for examining how various steps in a process are related to each other and whether or not each subsequent activity is receiving what it needs from the previous one. Flowcharts are used to help the assessor understand how a function is being implemented based on written programs and procedures. Flowcharts also illustrate reporting relationships and indicate whether the handoff of information or materials is adequate. They are especially useful for locating process bottlenecks, which may hinder the organization’s mission. In Figure C.1, the flowchart diagrams the steps in the design process, which helps identify critical areas and determine whether an individual step affects the design process output. In addition, this flowchart may allow the assessor to divide the design assessment between different visits while ensuring overall coverage. C.4 Cause-and-Effect Diagrams Cause-and-effect diagrams (Figure C.2) illustrate the relationship between a known “effect” or outcome and all the “causes” or contributors influencing it. The effect being examined may represent either a wanted or unwanted outcome. The cause-and-effect diagram is used when the outcome of a process/program is known but the contributors need to be evaluated further. These diagrams are especially useful when the contributors stem from different sources across the organization being assessed. Figure C.1 C-4 DOE G 414.1-1B 9-27-07 In this example, the assessor would use the cause-and-effect diagram (Figure C.2) to identify all the program elements that should be in place to prevent worker exposure. This tool can be used in two ways by the assessor: (1) to verify the effectiveness of individual elements, thereby verifying that the program is working and (2) to pinpoint the source of programmatic weaknesses. C.5 Tree Diagrams Tree diagrams (Figure C.3) are used to map out, systematically and in increasing detail, the full range of controls and tasks needed to achieve a primary goal. They can also be used to map out the barriers needed to prevent unwanted events (called “causal analysis” or “barrier analysis” trees). Tree diagrams may be used by the assessor to verify whether all planned activities are in place to support a program’s objective. They are especially useful for helping the assessor gain a big-picture view of the overall goals of the program, with its supporting sub-goals.

In Figure C.3, the tree diagram provides a map of the elements needed to support an effective training and qualification program. Using the diagram, the assessor can plan the assessment to Figure C.2 DOE G 414.1-1B C-5 9-27-07 ensure that the appropriate activities are being performed and to evaluate the training organization’s overall effectiveness. As this tree diagram is used for the assessment, the elements should be continually rolled up. This means the “Capability/Proficiency Verified” element should be assessed to determine its impact on “Qualification,” which should be assessed to ensure it supports overall “Training and Qualification.” C.6 Information Systems Information systems comprise a wide range of different forms and formats. In their simplest form they may include the weekly and monthly laboratory or organizational performance reports that may be used to alert the organization to potential assessment areas. In more complex form, these systems may include computerized databases that link performance to specific performance objectives or track actions to resolve programmatic weaknesses. In either case, information systems are important tools for assessors, providing much of the necessary data to focus assessment activities. In Figure C.4, information on lost-time injuries is displayed in both tabular and graphical form. This information can be used to focus the assessment on either the location of the injuries or the work procedures involved, to identify any weaknesses in the accident prevention program. Figure C.3 C-6 DOE G 414.1-1B 9-27-07 Figure C.4 DOE G 414.1-1B D-1 9-27-07 APPENDIX D—MANAGEMENT AND INDEPENDENT ASSESSMENT PLANNING, PERFORMANCE AND REPORTING Various scheduling and planning approaches can be used to satisfy the assessment requirements for integration, grading, and use of a risk-based approach. The following steps present one approach to planning for management and independent assessments and describe the information that should be compiled and maintained to manage the process to ensure that its objectives are achieved. D.1 Identifying Assessment Areas Management assessments are performed to comply with the QA Rule and QA Order and to improve performance. In general, the purpose of this type of assessment is to identify the management systems, processes, and programs that affect performance and to make improvements. Management assessments look at the total picture: • how well the management systems and processes meet the customer’s requirements; • compliance with standards and requirements; • meeting the expectations for safely performing work; • clarity of the organizational mission, goals, and objectives; and • identifying and correcting problems that hinder the organization from achieving its objectives. The emphasis of management assessment is on issues that affect performance, strategic planning, personnel qualification and training, staffing and skills mix, communication, cost control, organizational interfaces, and mission objectives. Independent assessments are also performed to comply with the QA Rule and QA Order and may be in the form of an audit, surveillance, or inspection conducted by individuals within the organization or company but independent from the work or process being evaluated or by individuals from an external organization or company. In general, the purpose of this type of assessment is to perform the following: • evaluate compliance with standards and requirements,

• evaluate the performance of work, • measure the quality of the item or service, • examine process effectiveness, adequacy, and • promote improvement. D-2 DOE G 414.1-1B 9-27-07 Independent assessments evaluate the performance of work processes with regard to requirements, compliance, and expectations for safely performing the work and achieving the goals of the organization. The focus of independent assessments should be the items and services produced and their associated processes. Independent assessments are typically performed less frequently than management assessments but go into greater depth. D.2 Prioritizing Assessment Activities Assessments areas should be made based on risk and ranked based on the consequences and likelihood of occurrence. Those with high risk or multiple lower risks should be ranked higher. This may result in identifying risks that crosscut several programs, activities, or organizations, thus benefiting more than one organization. D.3 Evaluating Probability and Consequences (Risk) Assessment areas should be evaluated to identify the various factors and elements that could result in a failure to achieve the mission objectives. Factors to consider include, but are not limited to the following: • worker health and safety (injuries, exposures, deaths); • public health and safety (injuries, exposures, deaths); • community concerns; • regulatory and DOE Order noncompliance and liability requirements; • business efficiency/productivity; • complexity of the involved processes; • reliability of the engineering and administrative controls; • personnel skills and experience; • maturity of the program, system, or process (developed, mature techniques/processes versus state-of-the-art or developmental/pilot technology); • changes that may affect performance (including regulatory); • life-cycle phase (new, midlife, closeout of activity); • organizational experience with the program, system, or activity; • economic uncertainty; DOE G 414.1-1B D-3 9-27-07 • schedule/commitment or milestones failure; and • performance measures/indicators, trending downward. Each of the factors should be evaluated for likelihood of occurrence and severity of consequences. This is a rough estimate, using professional judgment, and the use of a multi-disciplined team for this process is encouraged. D.4 Developing the Assessment Schedule and Plan The proposed management or independent assessment plan should be reviewed and approved by the senior line management to whom the assessed organizations are accountable. This will help ensure that the organizations to be assessed are prepared for the planned assessments. The schedule and plan should reflect the areas of greatest risk and the reasonable allocation of resources. Assessment areas that fall “below the line” should be retained as “targets of opportunity” to be performed if resources become available or if one of the planned assessments changes in risk or schedule. The assessment schedule and plan is a tool that allows management and customers to understand the basis for the assessments and justifies the allocated resources. Assessment plan documentation can vary from the relatively informal memo for small, simple scoped independent assessments, to a formally reviewed, approved and distributed assessment plan for complex and extensive management assessments. In all cases, however, the following elements should be documented:

• scope of the assessment (i.e., the program, system, process, organization, and/or activity to be assessed); • objectives of the assessment; • assessment drivers (e.g., the regulatory requirements, contractual agreements, performance objectives, and/or internal procedures that will be used); • assessment team members, including the lead, supporting assessors, and technical experts (if appropriate); • assessment process and criteria (e.g. CRADs); and • assessment schedule, with start and end dates (final planning, notification, kickoff/entrance meetings, preparation, investigation, closeout/exit meeting, and report issuance). The management or independent assessment plan should establish the depth, scope, and breadth of the assessment. It provides a tool for scheduling and information exchange for both the assessment team and the management of the assessed organizations. D-4 DOE G 414.1-1B 9-27-07 D.5 Allocating Resources The resources for performing management or independent assessments may be limited and not allow for performing an assessment in all areas. Likewise, the benefits of performing assessments on low risk areas are marginal. As a result, a realistic estimate of the resources available, including their scheduled availability, should be developed. Additional factors such as the availability of personnel independent of the areas to be assessed, budget constraints, management or customer requests, and response/follow-up to previous external assessments should be considered. D.6 Maintaining the Assessment Schedule and Plan Assessment plans should be reviewed periodically and modified as new information about the facility or organization is obtained that changes the estimated risks or reflects changes in available resources. These reviews can be used to finalize schedules, assessment areas, team members, etc. Assessment areas that have increased in risk can be moved up in the schedule, while others can be moved down. Assessment schedules are meant to be living documents that are adjusted periodically based on the analysis of operational awareness information, and the results of internal and external assessment activities. In some cases, assessment areas that were “below the line” should be moved up to reflect changes that have occurred since the original planning and ranking was performed. The results of past assessments, which identified areas of good/noteworthy performance, should be used to reduce the frequency and depth of future assessments. Areas of poor performance should receive increased attention, especially if there are indications that management has been unable to correct identified problems. This is because recurring and cumulative deficiencies, even in a low hazard operation, may indicate systemic problems and may decrease the likelihood of achieving its mission. D.7 Management and Independent Assessment Performance There are several methods that can be used in evaluating areas during a management or independent assessment. These can be both quantitative and qualitative and typically include: • written surveys or questionnaires to be filled out by an organization’s “customers,” • interviews with personnel, either one-on-one or in groups, • review of documents or records, • direct observation of work performance or facilities; and • performance of drills or exercises. For all methods, review criteria should be established through the use of regulatory and DOE requirements, management expectations, experience and judgment.

DOE G 414.1-1B D-5 9-27-07 D.8 Identifying Areas for Further Evaluation List the programs, systems, or processes and the areas of risk, and then further evaluate other factors such as the following: • time since the last management or independent assessment; • opportunities to perform the assessment in conjunction with other organizations (internal or external), • work schedules (will a lower-ranked program start or complete before a higher-ranked program or activity); • other scheduled assessments (e.g., process improvement teams, etc.) that would be expected to address the same area; • availability of assessment personnel, including technical personnel to perform the assessment; and • certifications, registrations, or other scheduled activities that would be expected to evaluate the program, system, or process. D.9 Management and Independent Assessment Reporting The results of management or independent assessment performance should be documented and transmitted to appropriate management levels. Similar to assessment plans, formality of documentation can vary widely from a memo for small simple scope independent assessments to reviewed, approved, and distributed reports. In general, the following elements should be addressed: • executive summary • assessment scope • identification of team members • identification of personnel contacted • documents reviewed • work performance observed • assessment process and criteria (e.g. CRADs) • results of the assessment including identification of deficiencies and/or strengths D-6 DOE G 414.1-1B 9-27-07 D.10 Frequency Given the likelihood of some or all of these factors being present and the wide variety of DOE activities, it is impossible to define the “right” management or independent assessment frequency in this Guide. Too few assessments will not keep pace with the changes occurring in the program, system, or process. Too many assessments will distract the organization from focused attention on the safe conduct of work and mission accomplishment. Therefore, it is ultimately the responsibility of management, guided by assessment professionals, to determine the appropriate mix of assessments (management and independent) to meet customer requirements and ensure mission success. DOE G 414.1-1B E-1 (and E-2) 9-27-07 APPENDIX E—ACRONYMS AND ABBREVIATIONS ANS American Nuclear Society ANSI American National Standards Institute ASME American Society of Mechanical Engineers ASQ American Society for Quality ASQC American Society for Quality Control CFR Code of Federal Regulations CRAD Criteria Review and Approach Document DNFSB Defense Nuclear Facility Safety Board ES&H Environment, Safety, and Health ISMS Integrated Safety Management System ISO International Organization for Standardization ISOP Integrated Safety Oversight Plans NEPA National Environmental Policy Act ORR Operational Readiness Review QA Quality Assurance S/RID Standards/Requirements identification document SIPS Safety Information Programs