DOE G 414.1-1C, Management and Independent Assessments Guide

G NOT MEASUREMENT SENSITIVE DOE G 414.1-1C 3-27-2014 MANAGEMENT AND INDEPENDENT ASSESSMENTS GUIDE [Guides can serve as reference sources for acceptable, non-mandatory implementation methods to satisfy requirements of Regulations, Orders, and Notices. NOTE: Guides do not establish requirements or constitute the basis for a finding of non-compliance with a specific requirement] U.S. Department of Energy Washington, D.C. 20585 AVAILABLE ONLINE AT: INITIATED BY: www.directives.doe.gov Office of Health, Safety and Security http://www.directives.doe.gov/ i DOE G 414.1-1C 3-27-4014 CONTENTS FOREWORD..............................................................................................................................................iii BACKGROUND ......................................................................................................................................... v 1. INTRODUCTION................................................................................................................................ 1 2. APPLICATION.................................................................................................................................... 3 3. GENERAL INFORMATION.............................................................................................................. 4 3.1. Assessment Program Expectations............................................................................................... 4 3.2. Assessment Benefits ..................................................................................................................... 5 3.3. Graded Approach.......................................................................................................................... 5 3.4. Purpose of Assessment ................................................................................................................. 7 3.4.1. Assessment Elements................................................................................................... 7 3.5. Types of Assessment .................................................................................................................... 8 3.5.1. Management Assessment ............................................................................................. 8 3.5.2. Independent Assessment .............................................................................................. 9 3.6. Organizational Activity Levels................................................................................................... 10 3.6.1. Process Level Assessments ........................................................................................ 11 3.6.2. System Level Assessments ........................................................................................ 11 3.6.3. Program Level Assessments ...................................................................................... 12 3.7. Assessing for Compliance and Performance .............................................................................. 12 3.7.1. Compliance Assessments ........................................................................................... 12 3.7.2. Performance-Based Assessments............................................................................... 12 4. GUIDELINES..................................................................................................................................... 14

4.1. Assessment Personnel ................................................................................................................ 14 4.2. Assessment Program Planning ................................................................................................... 15 4.2.1. Assessment Programs ................................................................................................ 15 4.2.2. Management Assessment Planning............................................................................ 16 4.2.3. Independent Assessment Planning............................................................................. 17 4.2.4. Planning Updates ....................................................................................................... 17 4.3. Assessment Integration............................................................................................................... 18 4.4. Assessment Plan ......................................................................................................................... 18 4.5. Performance Criteria .................................................................................................................. 20 4.6. Assessment Planning Tools........................................................................................................ 21 4.7. Independent Assessment Process ............................................................................................... 22 4.7.1. Pre-assessment Meetings ........................................................................................... 22 4.7.2. The Entrance Meeting................................................................................................ 22 4.7.3. Performing Independent Assessments ....................................................................... 23 4.7.4. Independent Assessment Techniques......................................................................... 24 4.7.5. The Exit Meeting ....................................................................................................... 25 4.7.6. Assessment Reporting................................................................................................ 25 ii DOE G 414.1-1C 3-27-2014 4.7.7. Releasing and Responding to Assessment Reports.................................................... 26 4.7.8. Corrective Actions ..................................................................................................... 26 4.7.9. Follow-up Assessments.............................................................................................. 27 4.8. Management Assessment Process .............................................................................................. 27 4.8.1. Defining the System................................................................................................... 28 4.8.2. Assessment Scheduling.............................................................................................. 29 4.8.3. Performing Management Assessments ...................................................................... 29 4.8.4. Assessment Reporting................................................................................................ 30 APPENDIX A - CONSENSUS STANDARDS AND REFERENCES................................................A-1 APPENDIX B - ASSESSMENT FUNCTIONAL AREAS ..................................................................B-1

APPENDIX C - TOOLS FOR ASSESSMENT PLANNING AND CONDUCT ...............................C-1 APPENDIX D - ACRONYMS AND ABBREVIATIONS ...................................................................D-1 TABLE OF EXAMPLES AND FIGURES Figure 3-1: Organizational Activity Levels ............................................................................................ 10 Figure 4-1: Examples of Assessment Program Development Considerations.................................... 15 Table C-1: Example of Laboratory Calibration Program Checklist ................................................C-1 Table C-2: Example of Oganizational Responsibilities Matrix .........................................................C-2 Table C-3: Example of Long-Range Planning Matrix........................................................................C-2 Figure C-1: Steps in a Design Process..................................................................................................C-3 Figure C-2: Cause and Effect Diagram................................................................................................C-4 Figure C-3: Tree Diagram.....................................................................................................................C-5 Figure C-4: Information Systems in Tabular and Graphical Format ..............................................C-6 DOE G 414.1-1C iii (and iv) 3-27-2014 FOREWORD This Guide is approved for use by all Department of Energy (DOE) and National Nuclear Security Administration (NNSA) components and contractors. Throughout this Guide the use of the DOE term includes DOE and NNSA. Suggestions for improving this Guide are welcome and should be sent to: Colette A. Broussard HS-33 U.S. Department of Energy 1000 Independence Ave., SW Washington D.C. 20585-270 Phone: (301) 903-5452 email: colette.broussard@hq.doe.gov Guides provide preferred, non-mandatory, supplemental information about acceptable methods for implementing requirements, including lessons learned, suggested practices, instructions, and suggested performance measures. Guides do not impose requirements but may quote requirements if the sources are adequately cited. Alternate methods may be used if it can be demonstrated that they provide an equivalent or better level of performance. mailto:colette.broussard@hq.doe.gov DOE G 414.1-1C v (and vi) 3-27-2014 BACKGROUND Since 2001, assessment practices and DOE requirements have evolved. This revision reflects the following assessment practices, international standards, and changes in DOE expectations related to quality assurance (QA): • DOE Order (O) 414.1D, Quality Assurance, dated 5-8-13; • related commitments in the DOE implementation plan for Defense Nuclear Facilities Safety Board (DNFSB) Recommendation 2004-1; • DOE O 226.1B, Implementation of Department of Energy Oversight Policy, dated 4-25­ 11; • DOE Guide (G) 226.1-2, Federal Line Management Oversight of Department of Energy Nuclear Facilities, dated 6-21-2012; • DOE O 227.1, Independent Oversight Program, dated 8-30-11; • DOE G 414.1-4, Safety Software Guide for Use with 10 CFR 830, Subpart A, Quality Assurance Requirements, and DOE O 414.1C, Quality Assurance, dated 6-17-05; • DOE O 436.1, Departmental Sustainability, dated 5-2-11; • NNSA Policy (NAP)-24, Weapon Quality Policy, dated 6-20-13; • American Society of Mechanical Engineers (ASME) Standard Nuclear Quality

Assurance (NQA)-1-2008 with the NQA-1a-2009 addenda; Quality Assurance Requirements for Nuclear Facility Applications, dated 8-31-09; and • international and national management system standards, such as from the International Organization for Standardization (ISO), the Institute of Nuclear Power Operations (INPO), and ASME . 1 DOE G 414.1-1C 3-27-2014 1. INTRODUCTION DOE and its contractors are required to perform management and independent assessments in accordance with DOE and NNSA requirements, including: • Title 10, Code of Federal Regulations (C.F.R.), Part 830, Subpart A, Quality Assurance Requirements, dated 1-10-01; • DOE O 414.1D, Quality Assurance, 5-8-13; • DOE O 450.2, Integrated Safety Management, dated 4 -25-11; • DEAR 970.5223-1, Integration of Environment, Safety, and Health into Work Planning and Execution; • DOE O 226.1B, Implementation of Department of Energy Oversight Policy, dated 4-25­ 11; • DOE G 226.1-2, Federal Line Management Oversight of Department of Energy Nuclear Facilities, dated 6-21-2012; • DOE O 227.1, Independent Oversight Program, dated 8-30-11; and • NNSA Policy (NAP)-24, Weapon Quality Policy, dated 6-20-20-13. Addressing the management or independent assessment requirements of criterion 9 and 10 found in DOE O 414.1D and 10 C.F.R. 830 can be accomplished by following the structure laid out in the Guide, and by including any additional implementation requirements to address the criteria contained in the approved site Quality Assurance Program (QAP). This Guide is intended as a method to perform and meet criterion 9 and 10 by providing a basis as to what should be considered when developing an assessment program, including, having the right people on the assessments teams. Once the assessment is complete, the development and implementation of effective corrective action (criterion 3) is vital to a successful program. Guidance for federal oversight of hazard category 1, 2, and 3 nuclear facilities is contained in DOE G 226.1-2. DOE G 414.1C can be used to supplement performance of federal oversight performed per DOE G 226.1-2. See Appendix A for a list of consensus standards and other references related to assessments. • Assessments add value to products and services by providing feedback and linking the management and conduct of work to meaningful improvement actions. Assessment programs should embody the following principles:Managers are involved in the assessment process to ensure results contribute to improved performance of the programs, systems, and work processes. • Managers receive timely, objective feedback from assessments. 2 DOE G 414.1-1C 3-27-2014 • Using a graded approach, managers take timely, appropriate actions to resolve quality problems. • Feedback addresses the effectiveness of policies, requirements, standards, processes, procedures, and their implementation. • Assessments are planned, coordinated, and integrated to promote efficiency and effectiveness. • Organizational culture seeks quality improvement, and assessments are accepted as contributors. • Assessment processes and results support the Departmental goal of protecting people, national security, environment, and capital assets. • Quality problems (including project management, engineering, design, construction, environment, safety, health, and security issues) are identified for resolution. 3 DOE G 414.1-1C 3-27-2014 2. APPLICATION

All DOE products and services, and the programs, systems, and processes that deliver them, can be assessed to improve their performance over their entire life cycles. Many DOE directives require assessments. This guide provides an approach to conduct effective assessments. Integration with other directive requirements helps to ensure assessments focus on the execution of the operational mission and reduce redundancy of effort in the application of resources. Examples of functional areas for assessments are listed in Appendix B. This Guide expands upon the details of the assessment criteria discussed in DOE G 414.1-2B, Quality Assurance Program Guide, dated 8-16-11. DOE G 450.4-1C Integrated Safety Management System Guide, dated 9-29-11, describes the role of assessments in the feedback and improvement of safety management functions. DOE G 226.1-2 describes the role of assessments in the nuclear oversight function. DOE line managers fulfill their safety responsibilities in part through line management QA and environment, safety and health (ES&H) oversight and have unfettered access to information and facilities in accordance with safety and security requirements. Contractor line managers fulfill safety responsibilities in part through the implementation of their assessment programs. Contractors are responsible for establishing and implementing robust, rigorous, and credible QA, ES&H, safeguards and security, and emergency management assessment programs, integrated with their safety management systems.). Assessment programs conducted in accordance with this Guide, and appropriately adopted standards will satisfy the assessment requirements of 10 CFR 830, Subpart A (QA Rule) and DOE O 414.1D (QA Order) and can supplement guidance, if needed, for other DOE Directives. The Guide, however, does not lessen the requirement to comply with the DOE QA Order or QA Rule, including those requirements related to management and independent assessments. Alternative methods demonstrated to achieve adequate levels of safety and quality may be acceptable to DOE. Finally, this guide can be used to supplement other existing rigorous assessment/review requirements contained in directives dealing with areas such as project management, envronmental protection, safeguards & security (S&S), emergency management, classification and controlled unclassified information, and cyber security; however, note that there may be additional actions, not described in this Guide, that are necessary to fully meet those rigorous requirements. 4 DOE G 414.1-1C 3-27-2014 3. GENERAL INFORMATION 3.1. Assessment Program Expectations The development of effective Management and Independent Assessment Programs should focus on achieving the following expectations: • Documented assessment programs provide for identification of the people, processes, and systems that will be used to plan, perform, and follow up on assessments. • Responsibilities are defined for both performing and responding to assessments. • Senior management endorses and actively supports the assessment program and views it as a value-added process, and ensures adequate fiscal and human resources are assigned to support the assessment function. Management at all levels is responsive to any identified issues, regardless of how they are identified. (Note: terminology used to identify issues or suggested improvements should be defined up front prior to the start of the assessment. Example, “findings” are those issues that are a direct result of non­ compliance with a requirement, and “opportunities for improvement” are those issues that are not contrary to a requirement, but would enhance the results of the implementation of requirements.)

• Individuals performing assessments are appropriately trained and qualified in the assessment process. • Objective evidence is available to substantiate the basis for assessment findings, as well as to support other assessment determinations, such as “in compliance with requirements.” • Corrective actions are taken promptly to prevent recurrence (that is, implement prompt and effective corrective actions by determining the root cause and ensuring the cause is addressed to prevent recurrenc: Criterion 3). • Feedback is solicited from management, workers, independent evaluators and customers, as appropriate. • Measurable organization goals and objectives are identified and progress can be demonstrated. • Assessments are planned, scheduled, and prioritized using a graded approach, as delineated in the Organization’s Quality Assurance Program. • Project activities and documentation are adequate to support Critical Decision approval in the areas of project management, engineering, design, safety, environment, security and quality assurance. 5 DOE G 414.1-1C 3-27-2014 3.2. Assessment Benefits The success of an organization depends upon the extent to which its products and services satisfy customer requirements and expectations. Each member of an organization is responsible for customer satisfaction. The results-oriented quality program described in the QA Rule and QA Order focuses on customer requirements and expectations, and embraces continuous improvement. Well documented and detailed assessments build confidence that organizations can meet customer expectations, self-identify areas where improvement is needed, and correct problems before they become major issues or events. Effective internal assessments prepare an organization for external governmental and nongovernmental assessments of performance, and verify compliance with applicable laws as well as conformity with national and international standards (e.g., internal classification self- assessments prepare sites for possible inspections by the external Information Security Oversight Office). Performance assessments are proactive analysis activities that identify performance gaps before they reveal themselves Voluntary third-party conformity assessments include those conducted by quality/environmental management system registrars, laboratory accreditation entities, and product certifiers who evaluate compliance with national or international standards. DOE contractors that participate in the Voluntary Protection Program (VPP), modeled after the Occupational Safety and Health Administration (OSHA) VPP, can attain recognition for excellence in safety and health management. Other national and state quality awards include the Presidential Award for Quality and the Malcolm Baldrige National Quality Award, which use comprehensive assessments that focus on integrated management systems and customer service. In addition to DOE oversight and enforcement, DOE sites are subject to assessment and regulation by the Environmental Protection Agency. Some DOE facilities are also regulated by the Nuclear Regulatory Commission and/or OSHA instead of DOE. These agencies may evaluate compliance with regulatory requirements, standards, and related commitments (involuntary third-party assessments) at DOE facilities. The benefit of an involuntary third-party assessment is the confirmation of compliance or non-compliance with regulatory requirements. Finally, DOE organizations are also subject to reviews by the Office of the Inspector General and the Government Accountability Office. The use of existing audit results from any of these independent authorities can help the assessment team be more effective and efficient when evaluating the facility.

An effective assessment process, when coupled with prompt and effective corrective actions, may be considered a mitigating factor by an assessor and can reduce needed assessment frequency. 3.3. Graded Approach This Guide and the technical standards referenced should be applied using a graded approach. The graded application is dependent upon the hazards and/or level of risk associated with the 6 DOE G 414.1-1C 3-27-2014 activity or structures, systems, and components under consideration. Assessments areas should be selected based on risk and ranked based on the consequences and likelihood of occurrence. Those with higher risks should be ranked higher. Those activities with multiple lower risks might need to be ranked higher depending on the combined consequences and likelihood of occurrence. This may result in identifying risks that crosscut several programs, activities, or organizations, thus benefiting more than one organization. The scope, depth, and rigor of assessments should be determined by the use of a grading process before performing the activity. The purpose of grading is to select assessment activities consistent with their importance to safety, cost, schedule, and success of the program. The grading process provides the flexibility to design an assessment that best suits the facility or activity. The grading process is not used to obtain exemption from the requirements of the QA Rule or QA Order. Items, services, or programs that contribute the greatest risk to quality, safety, and mission are assessed with the greatest rigor and frequency. The grading process is used to determine the appropriate assessment rigor and is based on activity-specific or facility-specific factors such as: • relative importance to worker health and safety; • relative importance to public health and safety; • relative importance to safeguards and security; • relative importance to national security; • protection of the environment; • magnitude of any hazard or risk involved; • life-cycle stage of a facility or activity; • impact/consequences on the programmatic mission of a facility; • unique characteristics of a facility, item or activity; • nuclear safety classification of equipment or facility hazard category; • adequacy of existing safety documentation; • relative importance of radiological and non-radiological hazards; • complexity of products or services involved; • performance history of a facility or activity; and • other relevant factors. 7 DOE G 414.1-1C 3-27-2014 3.4. Purpose of Assessment Establishing and implementing an effective assessment program is an integral part of every management system. Assessment is an important feedback mechanism that provides management with information to evaluate and improve the following: • organizational progress in reaching strategic goals and objectives; • adequacy and implementation of management programs for mission achievement; • performance capability of Integrated Safety Management Systems (ISMSs); • products and service quality; • regulatory and contractual compliance; and • for construction projects, assure safety and QA are integrated into the design. Simply stated, an assessment is an opportunity to: • identify the gaps between where you are and where you want to be; • identify the reasons for the gaps; • identify the actions that will be taken to close the gaps (corrective actions); • close the gaps between where you are and where you want to be; and

• verify that corrective actions for previously identified findings have been effective and lasting. 3.4.1. Assessment Elements An assessment program should have the following elements: • defined roles, responsibilities, authorities, and accountabilities for the staff performing and responding to assessments; • documented description, defining the purpose and the processes that will be used to plan, perform, and follow up on assessment; • identification of all planned assessments (including management and independent assessments); • performance metrics that reflect the assessment process itself (e.g., customer satisfaction, and findings); 8 DOE G 414.1-1C 3-27-2014 • a process to periodically evaluate the effectiveness of site assessment programs (both independent and management) in meeting regulatory and management objectives; • clear links to measurable organizational goals and objectives; • a training program to assure that assessment participants have the proper skills to perform assessments; • assessment results that feed the corrective action system; and • appropriate technical expertise on the assessment team. 3.5. Types of Assessment Assessments are tools to verify compliance, evaluate adequacy of procedures, review systems and processes, and perhaps more importantly, to drive improvement. DOE's QA Rule and QA Order establish distinct requirements for management and independent assessments. DOE O 226.1B refers to contractor self-assessment programs that include line and independent evaluations. In this context, the assessments are those that a contractor conducts on its own ES&H performance. Management and independent assessments as described in this Guide can also be used to supplement DOE G 226.1-2. Self-assessments are considered a type of assessment. In general, a management assessment is a form of self-assessment. A self-assessment is performed by the people doing the work that is being assessed. A management assessment is a self-assessment performed of the management’s organization and work, with direct management involvement. DOE and contractors should clearly describe in writing how their assessment programs satisfy the requirements for management, as well as independent assessments. The QA Order and QA Rule require that QAPs include a description of how these requirements are met. Implementing procedures for management and independent assessments should describe how these requirements are met. Management and independent assessments may be performed on the same functions or organizations; however, each has a specific focus defined by the QA Rule and QA Order as described below. It is essential to clearly define the criteria and/or objectives intended for the assessment through the assessment planning process and in the Criteria Review and Approach Documents (CRADs). 3.5.1. Management Assessment Managers perform management assessments to comply with the QA Rule and QA Order and to improve performance. The purpose of this type of assessment is to identify and evaluate the management systems, processes, and programs that affect performance and to identify areas for improvement. Management assessments look at the total picture: 9 DOE G 414.1-1C 3-27-2014 • how well the management systems and processes meet the customer's requirements; • compliance with standards and requirements; • effectiveness of change management; • meeting the expectations for safely performing work;

• clarity of the organizational mission, goals, and objectives; • identifying and correcting problems that hinder the organization from achieving its objectives; and • promoting improvement. The emphasis of management assessment is on issues that affect performance, strategic planning, personnel qualification and training, staffing and skills mix, communication, cost control, organizational interfaces, and mission objectives. Management assessment is a periodic introspective self-analysis to determine whether the organization's activities are achieving desired results. This includes reviewing the processes, systems, and programs that are important to the organization's mission and objectives. Results of management as well as independent assessments can be used, in addition to formulating approaches and corrective actions for improvements, to develop plans for the subsequent management assessments. Additionally, independent assessment results may also be used as the basis for determining the focus and frequency of management assessments. It should be noted that effective management assessments could result in less frequent independent assessments, and independent assessment findings could affect the frequency and rigor of management assessments. In general, management and independent assessments are complementary; however, management assessments are generally performed at a greater frequency and cover a broader spectrum than independent assessments. 3.5.2. Independent Assessment An independent assessment may be an audit, "for cause" review or inspection conducted by individuals within the organization or company but independent from the work or process being evaluated, or by individuals from an external organization or company. In general, the purpose of this assessment is to perform the following: • evaluate compliance with standards and requirements; • evaluate the performance of work; • measure the quality of the item or service; • examine process effectiveness/adequacy; 10 DOE G 414.1-1C 3-27-2014 • promote improvement. As required by the QA Rule and QA Order, organizations or staff performing independent assessments are to have sufficient authority and freedom from the line organization to carry out its responsibilities, and individuals are to be technically qualified and knowledgeable in assessment techniques and in the areas being assessed. Independent assessments evaluate the performance of work processes with regard to requirements, compliance, and expectations for safely performing the work and achieving the goals of the organization. The focus of independent assessments should be the items and services produced and their associated processes. Thus, management receives an objective view of the assessed activity. Independent assessments are typically performed less frequently than management assessments but go into greater depth. Management is responsible for developing and implementing a plan that balances management and independent assessments and other forms of feedback and improvement to satisfy the requirements of the QA Rule and QA Order. 3.6. Organizational Activity Levels ASSESSMENTS To develop a comprehensive assessment program that optimizes the application of each assessment type, it may be helpful to visualize the organization as having three interlinked levels of activity (Figure 3-1). This is not meant to imply a hierarchy of assessments. For this discussion, these levels are referred to as "process," "system," and "program." Management and independent assessments can be applied at all three levels, but examine different aspects of each level.

A process is a series of actions or operations leading to an end result such as a continuous operation or treatment especially in manufacture. A system is made up of two or more processes that may operate independently or interdependently to yield a complete product or service. A program is the most complex level, and consists of multiple interdependent systems that often require several interfaces to provide the desired product or service. Process System Program Results Figure 3-1: Organizational Activity Levels 11 DOE G 414.1-1C 3-27-2014 3.6.1. Process Level Assessments Process level assessments involve the examination of work controls, and verification that they are being implemented effectively, and should assess the effectiveness of the processes from a quality and customer satisfaction perspective. This level of assessment is critical for ensuring that the worker, the public, and the environment are protected from harm. At the process level, assessments are performed to verify compliance with procedures and process objectives or criteria, and to ensure that work-control documents (e.g., procedures, instructions, radiation surveys, permits, and safety checklists) accurately reflect tasks and their associated hazards. 3.6.2. System Level Assessments System level assessments focus on whether the appropriate leadership and support systems are in place to enable the implementation of work processes, and may range from informal daily performance oversight to formal periodic evaluations using established protocols. System level assessments are performed to ensure that human and material resources are being properly applied to achieve an organization's mission and objectives. System level assessments should also evaluate how well the systems produce the desired results. The collection of "processes" that have been assembled to form an effective "system" is also evaluated. At the system level, assessments are performed to determine whether all the necessary elements and interfaces are addressed, and to ensure that the system is capable of consistently meeting requirements and customer expectations. For example, a management assessment of the work control system might identify cost and resource allocation issues that affect the system. Some of the elements within the work control systems that might be assessed are: • planning the work; • identifying hazards associated with the work; • identifying hazard controls; • scheduling and performing work; • verifying/testing completed work; • evaluating human performance elements; • critiquing work processes and results; and • documenting the work performed. 12 DOE G 414.1-1C 3-27-2014 3.6.3. Program Level Assessments Program level assessments are used to determine whether the overall organizational programs are properly established and implemented, and are used to evaluate complex organizations from several perspectives. Program level assessments should also evaluate how well the program produces the desired results. They usually examine the integration of the systems designed to achieve organizational goals and customer expectations (with an emphasis on ES&H factors). At the program level for example, a maintenance management program, which relies on the work control system, would use results from the process (i.e., work control documents) and system level assessments to determine the effectiveness of the entire maintenance program. This program level assessment could be performed as either a management assessment or an independent assessment.

3.7. Assessing for Compliance and Performance There are two different methods commonly used for accomplishing assessments. These are usually known as compliance assessment and performance-based assessment. While each method has distinct characteristics, a good assessment will usually gauge, at some level, effectiveness of the processes, systems, and programs in meeting the mission and objectives of the organization. In practice, an assessment is likely to include both compliance and performance based methods. 3.7.1. Compliance Assessments Compliance assessments focus on verifying compliance with requirements through the implementation of procedures, and begin with a determination of the contractual and regulatory requirements governing the assessed organization. Assessors should become familiar with requirements and procedures, then verify that requirements flow down to implementing documents, such as procedures, whose implementation is in turn verified. Assessing for compliance alone may not adequately identify higher-level systemic or programmatic problems or determine the effectiveness of the program. For example, an organization may have written procedures that appear to implement the requirements; however, in practice the intent of those requirements may not be fully achieved because of variables such as poorly executed procedures. 3.7.2. Performance-Based Assessments Performance-based assessments take a different approach by focusing first on the adequacy of management behaviors and the processes that produced a product or service, and then on the product itself. If problems are found in the product or work processes, the assessor evaluates the methods and procedures used to implement the applicable requirements in an effort to find the failure that led to the problems. The assessor is expected to determine whether a non-compliance or series of non-compliances with procedures could result in a failure to satisfy top-level requirements. Results of prior compliance assessments may help the assessor in determining the focus areas for planning performance-based assessments. 13 DOE G 414.1-1C 3-27-2014 In performance-based assessments, greater emphasis is placed on the impact of issues discovered rather than on simply the existence of non-compliance issues (i.e., a compliance assessment). The assessor addresses the impact of the issues against the adequacy of what is being assessed. As an example, if an assessor sees a problem with the execution of a welding process, the next step should be to determine the impact of the problem, e.g., what is the impact on the quality of the final weld? Is it limited to one welder? Is it limited to one process? Can the problem be traced to the qualification program for the welder or to the qualification program for the welding process? Is there a problem with the weld material itself, indicating a problem such as engineering or procurement? While the assessor should be familiar with requirements and procedures, in performance-based assessments the assessor's experience and knowledge play an integral part in determining whether requirements are satisfied. Therefore, participants in performance-based assessments should be technically competent in the areas they are assessing. For example, if an assessor is evaluating a welding process, the assessor relies heavily on his or her knowledge of welding codes, welding processes, and metallurgy, rather than just verifying simple procedure compliance. Performance-based assessments usually provide the most useful information to management; however, it requires a much higher level of competence on the part of the assessment team. Results of performance-based assessments may provide useful insight for management's pursuit of excellence.

Managers use performance assessment to do the following: - Detect performance issues at a low level before they become consequential; - Assist in resource management by identifying and eliminating low-value assessment or monitoring activities; - Assist in identifying the most risk-significant or important issues on which to act. - Identify issues that need further analysis and intervention; and - Assist in identifying and resolving cross-organizational performance issues. 14 DOE G 414.1-1C 3-27-2014 4. GUIDELINES Organizations should establish procedures for planning and performing management and independent assessments. These procedures should be a part of the overall assessment program for the organization. They should address personnel training and qualifications, the assessment planning processes, performance protocols and tools, reporting, including report distribution, corrective action development and implementation, and other follow-up activities. The following guidelines are presented to assist organizations in developing their procedures and protocols. Management assessments share procedural and protocol commonalities with Independent Assessments. Because of this, the organization should ensure that assessment procedures are well defined and integrated, while maintaining the separate and unique focuses of the two types of assessments. For example, management and independent assessments look at the results of internal and external assessments to determine compliance with defined system requirements. Management assessments, however, also need to focus on how well the system is meeting organizational objectives and achieving improvement goals. 4.1. Assessment Personnel Assessment personnel facilitate continuous process improvement by identifying ways programs, systems, and processes can be improved and by providing information to management and owners. The assessor should be able to collect performance data through interviews, document reviews, observation, and inspection. It is very important that the assessor also be able to communicate effectively, both orally and in writing, and demonstrate effective interpersonal skills. Both management and independent assessments should be performed by qualified individuals who are knowledgeable about the program, system, or process being assessed and have been trained to ensure full understanding of the assessment processes, including reporting. Individuals performing independent assessments should not currently perform, supervise, or be directly responsible for performing the activities being assessed. Independence is determined based on an individual not having bias rather than on organizational affiliation. The independent assessor should have both the personal and organizational freedom to communicate with the management of the assessed organizations. Organizations should establish formal training and qualification programs for assessors, including both assessment team leaders and team members, that reflects both regulatory and customer requirements. Organizations may adopt third-party personnel qualification programs such as the American Society for Quality 's "Quality Auditor Certification" (http://www.asq.org/certification/index.html) or the Registrar Accreditation Board's certification program (http://www.rabqsa.com/). The International Organization for Standardization and the American Society of Mechanical Engineers (ASME) provide additional guidance for training and qualification of assessors (lSO-19011 and ASME NQA-l, respectively). For assessments of nuclear facilities and activities, ASME NQA- 1 is the appropriate national standard to be used by

http://www.asq.org/certification/index.html http://www.rabqsa.com/ assessment team personnel; se 15 DOE G 414.1-1C 3-27-2014 DOE and contractor organizations for guidance on training of assessment personnel. At a minimum, however, training and qualification programs for Federal QA personnel overseeing nuclear operations should be based on a recognized, relevant standard, such as DOE-STD-1150­ 2002, Quality Assurance Functional Area Qualification Standard. Effective assessments may be accomplished through the use of an assessment team with combined skills and experiences. Training for assessors should address the policies and procedures of the assessing organization. To enhance assessment performance and capability, new assessment personnel should participate in on-the-job training with qualified, experienced assessors before being considered fully trained or receiving a required qualification. Further guidance on assessor training and qualification is provided in DOE Orders and Guides, and the standards listed in Appendix A. 4.2. Assessment Program Planning 4.2.1. Assessment Programs Assessment programs should be developed to the level of rigor and detail required to ensure adequate review of programs, systems, and processes. An assessment program provides the structure for the overall process and ensures that assessments are conducted in a cost-effective, efficient manner. Items generally considered essential for a comprehensive assessment program include the following (see Figure 4-1): • assessment scheduling, planning approach, and logic (including how management and independent assessments are balanced); • methodology for determining/developing performance criteria; • recognition and use of third- party assessment results (accreditation, certification, registration, and regulatory); • assessment ethics and Figure 4-1: Examples of Assessment Program Development behaviors; Considerations • qualification and training of • protocols for conduct, including interfaces and meetings; Previous Assessment ssessment concern- the construction associated with Injector Project and in one finding and recommendations. ORPS Report January 8 – Loss of offsite Power caused blackout. T diesels fallen The root cau determined Schedule Jan Feb Mar Apr May WBS 1 WBS 2 WBS 3 WBS 4 WBS 5 16 DOE G 414.1-1C 3-27-2014 • format/review of assessment plans and agendas; • reporting methods/procedures for concerns, findings, observations, and improvement opportunities, including distribution, and mechanisms for addressing imminent danger issues; • procedures/processes for addressing concerns or issues (e.g., which type issue is expected to be addressed and what else is needed to be included in a corrective action plan); • verification of corrective action implementation and follow-up; • assessment records management program, including identification of records that will be retained, retention periods, and protection; and • central repository or website to ensure that assessment supporting documentation (e.g., assessment plans, corrective action plans, transmittal letters, etc.) can be retrieved.

Assessments should be conducted commensurate with the hazards, status, and importance of the program goals, systems, or work processes and should be focused on worker health and safety, public health and safety, national security, environmental protection, community concern, strategic planning, organizational resources, compliance and liability, and business efficiency and productivity. Complexity, reliability, risk, and economic issues should also be considered when planning and scheduling assessments. The application of a graded approach using a risk- based decision-making process will ensure that resources are applied in a manner that provides the greatest benefit to the assessed organizations and their customers. 4.2.2. Management Assessment Planning Management assessments should be planned in a systematic manner by the individual managers at each level of management, to evaluate the effectiveness and adequacy of their management systems. Assessments should be planned with appropriate consideration for other management and/or independent assessments that could conflict with or duplicate their efforts (see Section 4.3). [Please note- other assessments of related or similar topics may not always be conflicting or duplicative; they may actually enhance the assessment by providing additional information not readily obtained through the lines of inquiry. Duplicative topics in assessments may also serve to validate the findings of another assessment.] Management should retain overall responsibility for the planning and performance of management assessments, and the results of the planning process should be documented in an assessment plan. The following items should be evaluated: • how well management is providing the leadership to enable an organization to continuously meet internal and external customer requirements and expectations; • processes and their effectiveness, internally and across organizational boundaries, including ensuring that the staff is receiving the support to do their work (training, procedures, tools, and cooperation from others); and • performance information from other assessments (independent, Self, and External Assessments including financial, employee feedback systems, customer surveys, etc.). 17 DOE G 414.1-1C 3-27-2014 Management assessment planning should include the mechanics of performing the assessment such as the expected time-frame, assessment tools that may be used, reporting requirements, and how areas for improvement will be identified, tracked, and closed (see Section 4.8 for additional information). 4.2.3. Independent Assessment Planning The level of planning for an independent assessment will vary significantly depending on scope, breadth, and complexity of the system or process being assessed. The planning process should provide for input from the assessed organizations and consideration of their customers' and stakeholders' needs, requirements and expectations. Specific budget requirements and required support should be identified as early in the planning process as possible. Administrative issues that need to be anticipated and addressed prior to the site visit typically include: • requests for documentation; • travel planning; • security clearances; • access privileges; • facility authorizations; • facility orientation and training ; • site logistics; and • points of contact. There are various scheduling and planning approaches to satisfying these requirements. Appendix C describes tools to aid in assessment planning.

4.2.4. Planning Updates Assessments should be planned and completed within a reasonable period of time. Assessment planning should be reviewed and modified periodically as new information about the facility or organization is obtained. An assessment that finds good performance should be used as justification for reducing the depth and frequency of future assessments. The frequency of assessments may also be reduced based on good performance unless the frequency is established by a requirement (e.g., DOE O 475.2A, Identifying Classified Information). Areas of poor performance should receive increased attention, especially if there are indications that management is unable to correct the identified problems, because recurring and cumulative deficiencies, even in low-hazard operations, may decrease the likelihood of successfully achieving the mission. 18 DOE G 414.1-1C 3-27-2014 4.3. Assessment Integration Routine communication and trust among the various assessment bodies is essential when coordinating assessment plans. This task involves identifying overlapping and redundant assessments to reduce their negative impact on work performance. Once identified, assessments should be coordinated and consolidated in several ways. • Assessment scope should be coordinated and integrated to prevent multiple assessment organizations from performing similar assessments on the same subject. • Separate assessment teams may be combined to evaluate the subject in a single visit. • Assessments may be canceled based on agreements to share the results of other assessments. Where significant redundancies exist, such as when multiple contractors perform supplier quality assessments of the same analytical laboratory services, the recognition and use of accredited third-party assessment results and/or shared information from the other contractors’ evaluations of the same supplier, vendor or subcontractor, can be considered and used as part of the current assessment for planning purposes or as part of the objective evidence of compliance. Each of these methods requires staff time to develop, implement, and manage; however, they can result in substantial resource savings while enhancing the ability of the assessing organization to cover a greater number of activities. Having an integrated assessment schedule including management and independent assessments will assist the organization controlling overlapping and duplicate assessments to minimize impact to organizations. Additionally, consider developing a multi- year assessment schedule (along with a current year schedule) to identify the total scope of all required assessments. This multi-year schedule could be utilized in a rolling fashion. This will facilitate resource identification, resource leveling, and allow for reconciliation from year-to­ year. Assessment schedules are meant to be living documents that are adjusted periodically based on the analysis of operational awareness information, and the results of internal and external assessment activities. 4.4. Assessment Plan

The use and level of detail of an assessment plan will vary depending on what is being assessed, and whether the assessment is a management or independent assessment. There can be different levels of planning and documentation for assessments. An informal daily oversight activity requires less documentation and planning as opposed to a full blown assessment. The key is that the level of documentation depends on the purpose and scope of the assesment (i.e., a graded approach). Plans are used to scope and plan individual assessments, and should include input not only from the assessed organizations but also from their customers. A documented assessment plan not only allows expectations to be communicated to the assessed organization, but also allows the assessment team to focus its activities more effectively. The scope of the assessment should be defined in terms related to the assessed organization's mission and goals so the focus and value of the assessment will be clearly understood. The level of detail included in the assessment plan should be commensurate with the protocols of both the assessed and assessing 19 DOE G 414.1-1C 3-27-2014 organizations. Assessment plan documentation can vary from the relatively informal memo for small, simple scoped assessments, to a formally reviewed, approved and distributed assessment plan for complex and extensive assessments. In all cases, however, the following elements should be documented: • dates of the assessment; • scope of the assessment (i.e., the program, system, process, organization, and/or activity to be assessed); • objectives of the assessment; • assessment drivers (e.g., the regulatory requirements, contractual agreements, performance objectives, and/or internal procedures that will be used); • assessment team members, including the lead, supporting assessors, and technical experts (if appropriate) and their qualifications and biographies; • assessment process and criteria (e.g. CRADs); • assessment schedule, with start and end dates (notification; kickoff/entrance meetings; pre-assessment, daily, and post-assessment meetings; preparation; investigation; closeout/exit meeting; and report issuance). • list of documents to be provided to the assessment team prior to and upon arrival; • requests for site-specific training, dosimetry, and access requirements; • request for personnel availability for on-site interviews; • request for a list of in-process work activities during the assessment; and • request for points-of-contact for each functional area. The organization being assessed should be contacted and provided with the assessment plan as early as possible before the assessment (unless regulations or contracts specify other contact protocols). The proposed plan should be reviewed and accepted by the senior line management to whom the assessed organizations are accountable. This will help ensure that the organizations to be assessed are prepared for the planned assessments. Management or independent assessments in smaller organizations may not involve as much information as that listed above, nor the degree of formality required for assessments performed by external organizations. It is important to remember that each assessment needs to evaluate a program, system, or process during a finite time period. Assessment plans should be reviewed periodically and modified as new information about the facility or organization is obtained that changes the estimated risks or reflects changes in

20 DOE G 414.1-1C 3-27-2014 available resources. These reviews can be used to finalize schedules, assessment areas, team members, etc. Assessment areas that have increased in risk can be moved up in the schedule, while others can be moved down. Assessment schedules are meant to be living documents that are adjusted periodically based on the analysis of operational awareness information, and the results of internal and external assessment activities. 4.5. Performance Criteria Assessments seek to ensure that performance expectations defined by management and process owners are being met and are effective. Assessors should clearly understand the programs, systems, or processes being assessed, including their goals, and associated objectives and requirements for efficient, effective performance of operations. Performance requirements can be found in the following source documents: • federal and state regulatory requirements; • appropriate codes and standards; • contract requirements; • DOE Directives (Note: changes in Directives should be reviewed to ensure the organization that is being assessed has implemented them); • implementation plans; • implementation and operational procedures; • facility safety documents; • policy and mission statements; • DOE-approved "Work Smart" standards; • standards/requirements identification documents (S/RIDs); • plans and programs; • management, business, operating and/or strategic plans; and • applicable standards, permits, authorizations and regulatory agreements. Much information about performance and additional performance requirements as well as lessons learned may be available to assessors in existing documents and reports, such as: • reports from outside regulators; • facility operations/activity/metrics reports; 21 DOE G 414.1-1C 3-27-2014 • performance reviews; • previous assessment reports, including both internal and external assessments; • internal inspections, reviews, and reports; • corrective action plans and status reports; • concerns and occurrence reports; • performance indicators; • monitoring and survey data, and modeling data and analyses; and • DOE Non-compliance Tracking System reports (http://www.hss.energy.gov/enforce/nts.html/). Requirements contained in these documents are selected based upon impact on the assessed organization's mission and the relationship to the scope of the assessment. From selected requirements, objective statements (performance measures) are developed for determining whether a program, system, or process is working efficiently and effectively. From these measures, the specific performance criteria (based on written programs, DOE Orders, Rules, etc.) are developed and tools selected for conducting the assessments. In developing performance criteria, assessment personnel should not reinterpret or redefine requirements specified in the source documents. It is critical for a successful assessment that the requirements are understood and clearly explained in the assessment documentation. 4.6. Assessment Planning Tools

Typical planning tools include matrices, flowcharts, cause-effect diagrams, tree diagrams, checklists, CRADs, information systems, and others as applicable and deemed appropriate. Assessment planning tools such as checklists are an essential element of an effective assessment. They vary in format, content, and level of detail, but all have one thing in common: they help focus the assessor on the mission and objectives of the program, system, or process being assessed. Application of planning tools before an assessment ensures that time will be used effectively and that the assessment's focus is identified and maintained. Assessment planning tools are often used to relate the performance criteria to the established assessment scope and may include lists of interview questions, major elements of programs, or detailed process work steps. Similar to a road map, each tool is used to remind the assessor of where he/she is going and the items likely to be encountered along the way. Planning tools are extremely useful when the assessment basis is complex or the requirements come from multiple sources. (See Appendix C for examples and further discussion of these tools. An example of a QA CRAD can be found at http://www.hss.doe.gov/nuclearsafety/qa/qa_crad.html) http://www.hss.energy.gov/enforce/nts.html http://www.hss.energy.gov/enforce/ http://www.hss.energy.gov/enforce/ http://www.hss.doe.gov/nuclearsafety/qa/qa_crad.html 22 DOE G 414.1-1C 3-27-2014 4.7. Independent Assessment Process As discussed previously, there are commonalities shared by management and independent assessments. To avoid unnecessary duplication, the following sections of this Guide are very detailed, pointing out areas where the two assessment types differ. Section 4.8 is a briefer discussion concentrating on elements unique to management assessments. 4.7.1. Pre-assessment Meetings The effectiveness and efficiency of independent assessments can be improved greatly if representatives of the assessment team meet informally with key members of the assessed organization. This should be done at least a week before the assessment fieldwork actually begins. Pre-assessment meetings are particularly useful when the assessment team is completely external to the organization being assessed. It is recommended that these meetings be used to confirm that there is a common understanding between the assessing organization, the assessed organization, and the customer of the independent assessment (DOE program or field office management), of the criteria and/or performance measures. These meetings are also an opportunity to clarify the assessment team's plan and to work out logistical problems. The assessment team should identify and schedule the individuals to be interviewed so that arrangements can be made to ensure their availability. It is recommended that document handling and disposition protocols be clarified during the pre-assessment meeting to ensure protection of sensitive information. Documents that require extra effort to retrieve can be identified in advance, and the requirements for assessment team work space can be accomodated. 4.7.2. The Entrance Meeting

An entrance meeting involving personnel from the assessing organization and the managers of the organization being assessed, is held immediately before the assessment fieldwork begins to "set the stage" for a positive and productive independent assessment. This meeting is usually held at the assessed organization's location/facility and allows the assessment team to meet the assessed organization's managers and answer any questions they may have about the assessment. This meeting is also used to establish how concerns involving imminent danger or regulatory non-compliance will be communicated. The protocols to be followed during the assessment should be clarified during the entrance meeting, including discussions of the following: • purpose and scope of the assessment, including authority for conducting the assessment; • assessed organization's mission, program, systems, and processes; • scheduled duration of the assessment; • source documents and performance information that form the basis for the performance criteria to be used; • a list of knowledgeable individuals from the assessed organization as "points of contact" for each assessor; 23 DOE G 414.1-1C 3-27-2014 • any restrictions on the collection and/or disposal of assessment notes/records by the assessors (to include the procedure for identifying and protecting the site's classified and controlled unclassified information).; • logistics, including work area, working hours, lunch hours, etc.; • time and location of periodic status meetings; • time and place of the post assessment meeting; • specific safety concerns (e.g., PPE, escorts, emergencies); and • site-specific training. 4.7.3. Performing Independent Assessments The assessment should be conducted in accordance with established protocols developed by the assessing organization. Site/facility protocols should be followed for what to do if an imminent danger situation or a reportable non-complianceis encountered during the course of an assessment. Any assessment schedules or specific protocols established during the pre­ assessment meeting are used to ensure that the assessment is conducted effectively and safely. Assessors should keep their points of contact informed of their activities to preclude surprises during the exit meeting. This may include requests for additional assistance or the communication of concerns that require immediate action on the part of the assessed organization. Timely communication, verbal and written, will allow the assessed organization to verify the accuracy of observations and provide relevant facts and background on the issues. One of the ways to accomplish this is to meet periodically (daily or every other day) with the organization being assessed to convey questions/concerns, ensure a clear understanding of findings and observations and provide a status update. Daily team meetings may be helpful in ensuring continuity and overall focus by providing assessment team leaders with information about the completion status of the assessment checklists, and offering the opportunity for inquiry into issues requiring additional action (e.g., clearances, access, requests for personnel or material, and impasse resolution). These meetings also provide the setting for advising other team members of issues that may be of interest in their assigned scope, or for integrating data gathered by the various assessors. The meetings should be brief so that they do not significantly reduce the team members' field time with the processes they are to assess and the people they are to interview.

It is important that sufficient information be gathered during the assessment to determine whether an activity meets the performance criteria established. The assessor should be able to state clearly the criterion impacted by the activity and whether identified findings impact the mission/goals of the organization. To accomplish this, the assessor may deviate from the assessment schedule to determine the extent and significance of an issue. Deviations that affect the assessor's ability to complete the assessment team's interview schedule should immediately be made known to the organization being assessed and the team leader. 24 DOE G 414.1-1C 3-27-2014 4.7.4. Independent Assessment Techniques To maximize the effectivness of the assessment, productivity of the assessment team and resources, assessment teams should use a combination of tools and techniques. Such assessment techniques include interviews, document reviews, observation, inspection, and performance testing (Note: Some of these techniques can be used in a management assessment). The planning tools discussed in Appendix C also allow for more complex analysis and systematic coverage of the areas being assessed. In using these techniques, the assessor should not forget that the objective is to verify accomplishment of an organization's mission. To save time, the assessor should gather only data and information relevant to overall program performance and the achievement of program objectives. Assessments should be thorough and information gathered with sufficient diligence such that accurate, detailed conclusions can be provided to the organizations that will receive the final report. When using any of these techniques, assessors should maintain good records of the assessment results. These may include personal notes or other information to support the assessment, and may be included in the checklist information. These records are useful in writing the report, and any associated findings and recommendations, and will be valuable if questions arise during the report review process. All notes containing classified or controlled unclassified information must be disposed of properly in accordance with established and agreed-upon procedures.. A discussion of each of the techniques follows. 4.7.4.1 Interviews provide the means of verifying the results of observation, document review, inspection, and performance testing; allow the responsible person to explain and clarify those results; help to eliminate misunderstandings about program implementation; and provide a venue where apparent conflicts or recent changes can be discussed and organization and program expectations can be described. 4.7.4.2. Document Reviews provide the objective evidence to substantiate compliance with applicable requirements. A drawback is that the accuracy of many records cannot be ascertained by review alone. This technique should be combined with interviews, observation, inspection, and/or performance testing to complete the performance picture. Records and other documents should be selected carefully to ensure that they adequately characterize the program, system, or process being assessed. 4.7.4.3. Observation is the viewing of actual work activities. This is often considered the most effective technique for determining whether performance is in accordance with requirements. Assessors should understand the effect their presence has on the person being observed and convey an attitude that is helpful, constructive, positive, and unbiased. The primary goal during observation is to obtain the most complete picture possible of the performance, which should then be put into perspective relative to the overall program, system, or process.

4.7.4.4. Inspections are performed in accordance with acceptance criteria to verify the condition of physical facilities, systems, equipment, and components. 25 DOE G 414.1-1C 3-27-2014 4.7.4.5. Performance Testing is used to observe the response of personnel or equipment by creating a specific situation and noting the resulting performance. This technique is especially helpful when activities of interest would not normally occur during an assessment visit. It is also useful when the timeliness and appropriateness of the response are critical (e.g., emergency responses). 4.7.5. The Exit Meeting This meeting is used primarily by the assessment team to present the assessment summary. Reasonable time should be allowed to discuss any concerns, but this meeting should not be used to argue the assessment findings or methodology. There should be no surprises during the exit meeting since the assessment team should have taken every effort possible during the conduct of the assessment to ensure that the assessed organization was aware of the team's findings and concerns. Prior to the exit meeting, the assessment team should consider combining related weaknesses or performance issues into a smaller number of well-supported findings to help focus management's follow-up actions. A written summary of the assessment conclusions and results should be provided at the exit meeting. 4.7.6. Assessment Reporting Assessment reports are required for documentation of assessment results. Assessment team leaders have the overall responsibility for preparing the report and obtaining appropriate approval for its release as applicable. The report may be formal (e.g., distributed by memorandum) or informal (e.g., letter to file or email), depending on the level of assessment performed, but should provide a clear picture of the results in terms of the programs, systems, and processes assessed. The assessment report should be clear, concise, accurate, and easy to understand, and should include only facts that directly relate to assessment observations and results. It should include sufficient information to enable the assessed organization to develop and implement appropriate improvement plans. Specific report formats may vary considerably from one organization to the next. An independent assessment report usually includes the sections described below. (Note: A management assessment report may not require all of the content listed below and may only require an executive summary.) 4.7.6.1. Assessment Report Content • Executive Summary; • Assessment scope; • Identification of team members; • Identification of personnel contacted; • Documents reviewed; 26 DOE G 414.1-1C 3-27-2014 • Work performance observed; • Assessment process and criteria (e.g. CRADs); • Results of the assessment including identification of areas for improvement, and/or strengths; and • Identified issues of non-compliance (Findings) should include specific references to the applicable law, requirement, and/or standard, to ensure the responding organization has clear written understanding of the deficiency. 4.7.7. Releasing and Responding to Assessment Reports While team leaders have overall responsibility for the report, the entire assessment team should have an opportunity to read and sign the completed report. At a minimum, the final report should be distributed to the management of both the assessed and assessing organizations. Distribution to other organizations (e.g., Defense Nuclear Facilities Safety Board) should be defined during the planning phase and communicated in advance to the assessed group.

Because the true value of an assessment is the improvement opportunities it identifies, and its value typically diminishes over time, if possible, a draft final report should be released immediately after the exit meeting. This allows the assessed organization to begin improvement actions, yielding the maximum return for those actions. If this is not possible, the written summary of the assessment conclusions and results can facilitate the initiation of the improvement actions. The organization being assessed should have the opportunity for a factual accuracy review of the draft final report prior to the final report being issued. The assessment report or transmittal correspondence should clearly indicate what response is expected from the assessed organization and a reasonable response date. The assessed organization should evaluate the assessment report and provide a response that identifies the improvements and actions that will be taken; the responsible organization; and the expected completion date. 4.7.8. Corrective Actions Managers responsible for the activities assessed are also responsible for the development of effective corrective actions for the problem areas/deficiencies discovered during the assessment. At a minimum, these corrective actions should include the following: • measures to correct each deficiency; • identification of all root causes for significant deficiencies (Note: The term “significant deficiencies” should have a threshhold defined by the assessning organization that is shared with the assessed organization prior to the performance of the assessment); • determination of the existence of similar deficiencies or underlying causes (i.e., extent of condition, extent of cause); 27 DOE G 414.1-1C 3-27-2014 • actions to preclude recurrence of like or similar deficiencies (e.g., effectiveness of corrective actions); • assignment of corrective action responsibility; and • completion dates for each corrective action. Managers should verify that corrective actions are likely to fully address the identified deficiency and when actions are completed, validate that the actions have corrected the deficiency. Additional guidance for corrective actions can be found in DOE G 414.1-2B and DOE G 226.1­ 2 4.7.9. Follow-up Assessments A follow-up assessment with special focus may be performed and should be completed in accordance with applicable corrective action documents. Particularly, this follow-up assessment should evaluate the effectiveness of corrective actions. A reasonable subset of corrective actions should be reviewed for effectiveness. To increase the validity of the effectiveness review, a sufficient amount of time for implementation of the corrective action should be allowed before performing the review (e.g., six months). The results of past assessments, which identified areas of good/noteworthy performance, should be used to reduce the frequency and depth of future assessments. Areas of poor performance should receive increased attention, especially if there are indications that management has been unable to correct identified problems. This is because recurring and cumulative deficiencies, even in a low hazard operation, may indicate systemic problems and may decrease the likelihood of the organization achieving its mission. 4.8. Management Assessment Process

Management assessments are a tool used by managers to identify and resolve issues that contribute to success or failure in fulfilling mission, goals, requirements, or expectations. During management assessments, focus should be on evaluating system performance issues and related processes such as personnel skills, communications, cost control, and other similar processes. Some sites have identified management assessment subject matter experts (SME) typically within the central assessment organization who have responsibility for reviewing the management assessments performed by the various program and division managers within the organization. The SMEs provide feedback as to how well the various management assessments are meeting organization expectations, areas for improvement, etc. This helps to alleviate one of the deficiencies typically associated with management assessments; inconsistencies in the scope and effectiveness of such assessments across the organization. Planning management assessments is an organization-specific effort that should be integrated with other assessment processes. No single method is appropriate for every situation. Either quantitative or qualitative assessment methods may be used as appropriate to the assessment scope. Managers are challenged to make the assessment a value-added process that will lead to 28 DOE G 414.1-1C 3-27-2014 improvement in organizational performance, safety, meeting customer expectations, and achieving mission goals in full compliance with regulatory and DOE requirements. It is important to remember that while management assessments share some commonalities with independent assessments, they should focus on evaluating organizational performance and identifying barriers that hinder improved performance. The levels of formality and scope of management assessments can vary greatly. For example, at the informal extreme of this range, a manager may evaluate the skills of personnel involved in a single work function and document the need for additional training or staffing levels adjustments through a memorandum. At the formal extreme, an assessment team made up of senior managers and process consultants from outside the organization, can perform an extensive evaluation of an entire organization using checklists, interviews, document reviews, work observations, and objective evidence evaluation, followed by the issuance of a formal report. The two links below provide one approach for DOE Offices to perform a management assessment on the ten criteria and other requirements from DOE O 414.1D. The first link is applicable to DOE Program and Field Offices and their contractors. The second link is applicable to DOE staff and support offices. http://hss.doe.gov/nuclearsafety/qa/docs/Management_Assessment_Questions_for_Program_and _Field_Offices.pdf http://hss.doe.gov/nuclearsafety/qa/docs/Management_Assessment_Questions_for_Staff_and_Su pport_Offices.pdf 4.8.1. Defining the System The organization should have a written description of the management assessment process. The description should address all elements of the process including, but not limited to, the following: • management levels that will be expected to perform assessments; • general goals of the management assessment process; • training or mentoring that will be provided to assessing managers; • expectations for the number and frequency of assessments to be performed; • expectations and guidelines for management assessment reports;

• reporting and follow-up processes; • interfaces with other oversight programs and processes (e.g., the corrective action reporting and tracking process); • management assessment planning process; and http://hss.doe.gov/nuclearsafety/qa/docs/Management_Assessment_Questions_for_Program_and_Field_Offices.pdf http://hss.doe.gov/nuclearsafety/qa/docs/Management_Assessment_Questions_for_Program_and_Field_Offices.pdf http://hss.doe.gov/nuclearsafety/qa/docs/Management_Assessment_Questions_for_Staff_and_Support_Offices.pdf http://hss.doe.gov/nuclearsafety/qa/docs/Management_Assessment_Questions_for_Staff_and_Support_Offices.pdf 29 DOE G 414.1-1C 3-27-2014 • administration of the process. 4.8.2. Assessment Scheduling Schedules for management assessments need to be established with the expected frequency of assessment performance specified. However, scheduling of management assessments should be as flexible as possible to meet operational and management needs. The organization should review and update its management assessment schedule on a regular basis, either bi-monthly or quarterly, to ensure relevance. The review should consider the current conditions, conclusions of recent management assessments, inputs for independent assessments, and organizational performance. Management assessments should be planned with input from all levels of management. Some organizations have found it beneficial to schedule the number of assessments, but leave the determination of the topics/areas of assessment to the performing manager, thus providing flexibility and the freedom to perform assessments that will result in the greatest opportunity for improvement. The planning process may include and take credit for the existing management reviews or similar assessments that routinely occur, such as the following: • program reviews; • strategic planning sessions; • reviews of performance indicators; • reviews of organizational goals-setting and objective-setting sessions; • financial reviews; • reviews of the outputs of improvement teams; and • reviews of independent assessments. Management assessments may include some benchmarking activities, both internal and against other organizations. The assessment of internal and external performance indicators may also be beneficial. There is no fixed number of assessments to be performed; however, the organization should be able to show that the management assessment program complies with the QA Rule and QA Order. 4.8.3. Performing Management Assessments Management assessments may be performed by individual managers or teams of managers. The primary responsibility for management assessments resides with managers because they are in the best position to identify barriers to improved performance and to effect changes. While some 30 DOE G 414.1-1C 3-27-2014 aspects of the assessments, such as collecting information, may be delegated to staff, it is the manager's responsibility to perform the assessment and determine the conclusions. Personal involvement by the manager will yield the most meaningful information for improving the performance of the organization. The primary purpose of a management assessment is not verification of compliance with regulatory or DOE requirements; however, if a noncompliance with requirements is found the appropriate organizations should be notified and corrective action processes implemented. Due to the limited time nature of assessment activities, managers must consider the need to perform additional evaluations of problem areas/deficiencies to ensure potential extent of condition concerns and actual root causes are determined, prior to developing corrective actions.

4.8.4. Assessment Reporting Management assessments should be reported in accordance with the organization's assessment program. As with independent assessment reports, management assessment reports should include a concise summary of the topics or areas assessed, the conclusions reached, and any follow-up actions that may be required. Reports should be available for use by others and for future planning. If reports contain sensitive information such as classified information or controlled unclassified information, including Official Use Only (e.g., proprietary, personal privacy, export controlled, etc.) and Unclassified Controlled Nuclear Information (UCNI), special provisions are required. Assessments identifying potential regulatory or DOE requirements compliance issues should be communicated directly to the appropriate managers for any necessary action as well as in the final assessment report. . DOE G 414.1-1C Appendix A 3-27-2014 A-1 APPENDIX A - STANDARDS, HANDBOOKS, AND OTHER REFERENCES A.1. Standards and Handbooks 1. The following are some of the DOE standards and handbooks as well as consensus standards that provide methods for implementing the guidance contained herein. A single standard may not fully implement all elements of the requirements (particularly for management assessments); therefore, these documents should be used in conjunction with 10 CFR 830, Subpart A, and DOE O 414.1D, Quality Assurance, to develop and implement assessment processes that meet DOE assessment requirements. The organization remains responsible for compliance with 10 CFR 830, Subpart A, and DOE O 414.1D.American National Standards Institute (ANSI)/American Nuclear Society (ANS) 3.2-2012, Administrative Controls and Quality Assurance for the Operational Phase of Nuclear Power Plants. 2. ANSI/ANS 15.8-1995 R2013, Quality Assurance Program Requirements for Research Reactors. 3. ANSI/ASQ (American Society for Quality) Z 1.13-1999, Quality Guidelines for Research. 4. ANSI/ASQC (American Society for Quality Control) E4-2004, Quality Systems Requirements for Environmental Data and Technology Programs. 5. ANSI/ISO (International Organization for Standardization)/ASQ Q9001 :2008, Quality Management Systems: Requirements. (Note: ANSI/ISO/ASQ Q9000 series documents are recognized as being identical to their ISO 9000 series counterparts.) 6. ASME (American Society of Mechanical Engineers) NQA-1-2000, Quality Assurance Requirements for Nuclear Facility Applications. 7. ASME NQA-1-2004, Quality Assurance Requirements for Nuclear Facility Applications. 8. ASME NQA-1-2008 with the NQA-1a-2009 addenda, Quality Assurance Requirements for Nuclear Facility Applications 9. ASME NQA-1-2012, Quality Assurance Requirements for Nuclear Facility Applications. (Specifically Part II, 2.22, Part I, R 18, etc) 10. DOE-HDBK-1101-04, Process Safety Management for Highly Hazardous Chemicals, February 2004. 11. DOE-HDBK-3027-99, Integrated Safety Management Systems (ISMS) Verification; Team Leader's Handbook, June 1999. 12. DOE-NE-STD-1004-92, Root Cause Analysis Guidance Document, February 1992. Appendix A DOE G 414.1-1C A-2 3-27-2014 13. DOE-STD-1036-93; Guide to Good Practices for Independent Verification, June 1993 (and Change Notice 1, dated December 1998). 14. DOE-STD-3006-2010, Planning and Conducting Readiness Reviews , June 2000. 15. ISO 19011 2002, Guidelines for Environmental Auditing- Qualification Criteria for Environmental Auditors, 1996.

16. ANSI/ISO/ASQ QE 1901\ S-2004, Guidelines on Quality and/or Environmental Management Systems Auditing, May 2004. 17. DOE-STD-1150-2013, Quality Assurance Functional Area Qualification Standard, December 2013 18. DOE-STD-1172-2011, Safety Software Quality Assurance Functional Area Qualification Standard, February 2011 19. INPO 07-007, Performance Assessment and Trending A.2. Other References The following references provide additional information concerning assessments. 1. DOE O 151.1C, Comprehensive Emergency Management System, November 2005. 2. DOE O 413.3B, Program and Project Management for the Acquisition of Capital Assets, November 2010. 3. DOE G 413.3-2, Quality Assurance Guide for Project Management, June 2008. 4. DOE G 414.1-2B Admin Change 2, Quality Assurance Management System Guide for Use with 10 CFR 830.120 and DOE O 414.1., May 2013 5. DOE G 414.1-4, Safety Software Guide for Use with 10 CFR 830, Subpart A, Quality Assurance Requirements, and DOE O 414.1C, Quality Assurance, June 2005. 6. DOE O 471.1B, Identification and Protection of Unclassified Controlled Nuclear Information, March 2010. 7. DOE O 475.2A, Identifying Classified Information, February 2011. 8. DOE/RW-0333P, Quality Assurance Requirements and Description [for the Civilian Radioactive Waste Management Program], Rev. 21, Section 18.0, "Audits," Office of Civilian Radioactive Waste Management, January 2009. 9. DOE G 450.4-1C, Integrated Safety Management System Guide, September 2011. DOE G 414.1-1C Appendix A 3-27-2014 A-3 (and A-4) 10. U.S. Department of Commerce, National lnstitute of Standards and Technology, "Malcolm Baldrige National Quality Award Criteria for Performance Excellence" http://www.nist.gov/baldrige/. 11. DOE O 226.1B, Implementation of Department of Energy Oversight Policy, April 2011. 12. DOE G 226.1-2, Federal Line Management Oversight of Department of Energy Nuclear Facilities, June 2012. 13. DOE O 426.2, Personnel Selection, Training, Qualification, and Certification Requirements for DOE Nuclear Facilities, April 2010. 14. DOE O 227.1, Independent Oversight Program, August 2011. 15. NAP-24, Weapon Quality Policy, dated 6-20-13. 16. Quality Assurance Criteria and ReviewApproach Documents http://www.hss.doe.gov/nuclearsafety/qa/qa_crad.html. 17. Safety Software Quality Assurance Criteria and Review Approach Documents http://www.hss.doe.gov/nuclearsafety/qa/sqa/sqa_crad.htm. 18. The Certified Manager of Quality/Organizational Excellence Handbook, 3rd edition: Russell T. Westcott, editor; ASQ Quality Press 2005. http://www.hss.doe.gov/nuclearsafety/qa/qa_crad.html http://www.hss.doe.gov/nuclearsafety/qa/sqa/sqa_crad.htm http://www.nist.gov/baldrige DOE G 414.1-1C Appendix B 3-27-2014 B-1 APPENDIX B - ASSESSMENT FUNCTIONAL AREAS This appendix lists environment, safety, and health (ES&H) functional areas, as well as other areas such as safeguards and security, and emergency management programs that should be included in a comprehensive assessment program. The list represents requirements, processes, and programs found in regulations, DOE policy, and DOE Orders. This appendix should not be construed as limiting assessments to only these functional areas (i.e., this is not an all inclusive list), nor should it be interpreted as minimizing the importance of assessing product/service quality and organizational performance. It does, however, illustrate the wide variety of programs, systems, and processes currently implemented by DOE and its contractors. These functional areas comprise a complex mix of people, hardware, software, and resources, all impacting on management and performance of activities. An integrated assessment program should be adaptable to this mix in order to be responsive to senior management needs and comply with 10 CFR 830, Subpart A, and DOE O 414.1D. These areas can also be considered when using DOE O 226.1B or the corresponding 226 Guide when performing oversight of nuclear facilities or operations.

• Accelerator Safety • Accident, Incident, and Unusual Occurrence Investigation and Reporting Process • Acquisition Strategy • Aviation Safety • Biological Hazards • Calibration Control • Chemical Safety (process safety management) • Classification • Computer Software Control • Conduct of Operations • Configuration Management including design control, document control, and change control • Construction Projects • Construction Safety • Criticality Control • Corrective Action • Design Activities • Differing Professional Opinion Program • DOE Nuclear Safety Management Rule (10 CFR 830) • Document Control and Records • Electrical Safety • Emergency Management • Employee Assistance Program • Employee Concerns Program • Engineering Design Processes • Environmental Management Systems/Sustainabiltiy • Environmental Protection and National Environmental Policy Act Compliance • Equipment Modifications • Experimental Programs • Explosives Safety • Facility Operations • Fire Protection • Firearms Safety • Hoisting and Rigging • Identification & Control of Items • Industrial Hygiene • Industrial Safety B-2 Appendix B DOE G 414.1-1C • Inspection & Test Control • Integrated Safety Management System Implementation • Maintenance Management • Motor-Carrier/Vehicle Safety • Natural Phenomenon Hazards • Non-conformance Control • Nuclear Facility Safety • Occupational Medicine • Occurrence Reporting/Trending • Operating Experience Program • Operational Readiness Review Process • Packaging and Transportation • Performance Measures and Indicators • Pollution Prevention • Pressure Systems • Procurement & Contracts (including supplier control) • Quality Management Systems • DOE Occupational Radiation Protection Rule (10 CFR 835) 3-27-2014 • Reactor Safety • Safeguards and Security • Safety Management Systems • Safety Basis Documentation (e.g., Bases for Interim Operation, hazard analyses, and documented safety analysis) • Software Quality Assurance • Standards/Requirements Identification Document(s) • Suspect/Counterfeit Items • Technical Safety Requirements • Training of Nuclear Facility Personnel • Unreviewed Safety Questions Process • Voluntary Protection System • Waste Management • Waste Minimization • Welding • Work Planning and Control • Worker Protection/Industrial Safety DOE G 414.1-1C Appendix C 3-27-2014 C-1 APPENDIX C - TOOLS FOR ASSESSMENT PLANNING AND CONDUCT C.1. Checklists Checklists (Table C-1) are lists of assessment objectives and performance criteria. They usually include a column for the requirements (or references to the requirements) and a column for recording assessment observations/evidence. Checklists are especially useful for organizing assessment time, keeping the assessment focused, and providing a means to list appraisal objectives sequentially. They may also be structured in a form that can easily be converted into an assessment report. Table C-1: Example of Laboratory Calibration Program Checklist Lab/Appraisal Number: Date: Page 1 of Reference Criteria Results Comments Sat Unsat NL-QAM 1. Is monitoring and data collection equipment calibrated? NL-QAM 2. Is equipment calibration traceable to nationally recognized standards? NL-QA-5.1 3. Is equipment calibration performed using approved instructions? NL-QA-5.1 4. Are calibration records maintained for each piece of equipment? NL-QA-5.1 5. Is a use log maintained?

In Table C-l, the checklist example is used to list the primary elements of a laboratory's calibration program. The basis or source of each criterion is included in the first column to provide a path back to the requirements document(s). The "comments" column provides a place for the assessor to record additional observations as they are discovered during the assessment, thus helping to ensure that important data is not lost. C.2. Matrices Matrices (Tables C-2 and C-3) are two-dimensional tables showing the relationship between two sets of information. They can be used to show the logical connecting points between performance criteria and implementing or required actions, and personnel responsible for those actions. In this way, matrices are used to determine what actions and/or personnel have the Appendix C DOE G 414.1-1C C-2 3-27-2014 greatest impact on an organization's mission. Matrices are especially useful as a way to focus assessment time and organize assessment conduct. Table C-2: Example of Oganizational Responsibilities Matrix Program Development Deficiency Tracking Training Work Control Documents & Record Retention Assessment Director X X Ops Office X X X Ops Support X X X Tech Support X X X Admin X X In Table C-2, the matrix example is used to help the assessor plan the assessment by identifying organizational responsibilities for the different assessment areas. This type of matrix is used to maximize use of assessment time during the site visit. Table C-3: Example of Long-Range Planning Matrix Administration Chemistry Biology Materials Building Services Engineering Industrial Hygiene A A A A Radiological Protection B B B Fire Protection A C C C C Industrial Safety A C A Environmental C A C C Personnel Training B C B B Conduct of Operations C C C Quality Assurance A C A C A=1st assessment, B=2nd assessment, C=3rd assessment DOE G 414.1-1C Appendix C 3-27-2014 C-3 Table C-3 is a much broader example of the use of the matrix that allows the assessor to do the long-range planning necessary for ensuring proper application of the assessment program. In this example, the various assessment areas (Y axis) are correlated with the different organizations to be assessed (X axis). C.3. Flowcharts Flowcharts (Figure C-1) illustrate the steps or activities in a process. They provide an excellent tool for examining how various steps in a process are related to each other and whether or not each subsequent activity is receiving what it needs from the previous one. Flowcharts are used to help the assessor understand how a function is being implemented based on written programs and procedures. Flowcharts also illustrate reporting relationships and indicate whether the handoff of information or materials is adequate. They are especially useful for locating process bottlenecks, which may hinder the organization's mission. Identify Design Project Prepare Scoping Statement/Receive Funding Approval Prepare Conceptual Design Establish Design Baseline Establish Design Criteria Sufficient Info on Existing Configuration Validate Existing Design Validate Existing Design Prepare Supporting Analysis Coordinate Cross-Discipline Review Verify Design Criteria and Configuration Finalize Design Documents Comments? Approved? Issue Design Modify Design No No Yes Yes Yes Figure C-1: Steps in a Design Process Appendix C DOE G 414.1-1C C-4 3-27-2014

In Figure C-1, the flowchart diagrams the steps in the design process, which helps identify critical areas and determine whether an individual step affects the design process output. In addition, this flowchart may allow the assessor to divide the design assessment between different visits while ensuring overall coverage. C.4. Cause-and-Effect Diagrams Cause-and-effect diagrams (Figure C-2) illustrate the relationship between a known "effect" or outcome and all the "causes" or contributors influencing it. The effect being examined may represent either a wanted or unwanted outcome. The cause-and-effect diagram is used when the outcome of a process/program is known but the contributors need to be evaluated further. These diagrams are especially useful when the contributors stem from different sources across the organization being assessed. In this example, the assessor would use the cause-and-effect diagram (Figure C-2) to identify all the program elements that should be in place to prevent worker exposure. This tool can be used in two ways by the assessor: (1) to verify the effectiveness of individual elements, thereby verifying that the program is working and (2) to pinpoint the source of programmatic weaknesses. Design/ Rad Protection Controls Standards Protective Specify Training Clothing Use Endorse ALARA Provide Principles Appropriate Dose Limits Shielding Material/ Provided Equipment Use Internal Barriers Bioassay Provided Contamination Material Survey Labeled Personal Dosimetry Areas Posted Air Sampling PREVENT WORKER EXPOSURE Access Monitoring Figure C-2: Cause and Effect Diagram DOE G 414.1-1C Appendix C 3-27-2014 C-5 C.5. Tree Diagrams Tree diagrams (Figure C-3) are used to map out, systematically and in increasing detail, the full range of controls and tasks needed to achieve a primary goal. They can also be used to map out the barriers needed to prevent unwanted events (called "causal analysis" or "barrier analysis" trees). Tree diagrams may be used by the assessor to verify whether all planned activities are in place to support a program's objective. They are especially useful for helping the assessor gain a big-picture view of the overall goals of the program, with its supporting sub-goals. In Figure C-3, the tree diagram provides a map of the elements needed to support an effective training and qualification program. Using the diagram, the assessor can plan the assessment to ensure that the appropriate activities are being performed and to evaluate the training organization's overall effectiveness. As this tree diagram is used for the assessment, the elements should be continually rolled up. This means the "Capability/Proficiency Verified" element should be assessed to determine its impact on "Qualification," which should be assessed to ensure it supports overall "Training and Qualification." Job Requirements Incorporates Includes Established Requirements Process/Tools Initial Training Documented Emphasizes Correct Provided Performance Capability/Proficiency Approved Use of Instructional Verified Methods On-Going Training Periodically Reviewed Validation Performed Provided Figure C-3: Tree Diagram C.6. Information Systems Information systems comprise a wide range of different forms and formats. In their simplest form they may include the weekly and monthly laboratory or organizational performance reports Appendix C DOE G 414.1-1C C-6 3-27-2014

that may be used to alert the organization to potential assessment areas. In more complex form, these systems may include computerized databases that link performance to specific performance objectives or track actions to resolve programmatic weaknesses. In either case, information systems are important tools for assessors, providing much of the necessary data to focus assessment activities. In Figure C-4, information on lost-time injuries is displayed in both tabular and graphical form. This information can be used to focus the assessment on either the location of the injuries or the work procedures involved, to identify any weaknesses in the accident prevention program. Lost-Time Accident Monthly Summary Date Type Area Work Procedure Work Crew Days Lost 5/3 Sprain Bldg 12 CAP-101 Mech 4 5/5 Sprain Bldg 5 MAP-2-12 Elec 5 5/12 Burn Area 8 PMP-1-4 Mech 2 5/15 Abrasion Area 10 PMP-3-7 Grnds 3 5/23 Burn Bldg 12 CAP-103 Elec 1 5/25 Sprain Admin Bldg N/A N/A 1 5/29 Cut Bldg 5 MAP-2-17 Elec 1 Lost Work This Month 6 5 4 3 2 1 0 Abrasions Breaks Burns Cuts Sprains Electrical Mechanical Other Figure C-4: Information Systems in Tabular and Graphical Format DOE G 414.1-1C Appendix D 3-27-2014 D-1 (and D-2) APPENDIX D - ACRONYMS AND ABBREVIATIONS ANS American Nuclear Society ANSI American National Standards Institute ASME American Society of Mechanical Engineers ASQ American Society for Quality ASQC American Society for Quality Control CFR Code of Federal Regulations CRAD Criteria and Review Approach Document CRD Contractor Requirements Document DNFSB Defense Nuclear Facilities Safety Board ES&H Environment, Safety, and Health ISMS Integrated Safety Management System ISO International Organization for Standardization NEPA National Environmental Policy Act OSHA Occupational Safety and Health Administration ORPS Occurrence Reporting and Processing System QA Quality Assurance QAP Quality Assurance Program S/RID Standards/Requirements Identification Document S&S Safeguards and Security VPP Voluntary Protection Program CONTENTS FOREWORD BACKGROUND 1. INTRODUCTION 2. APPLICATION 3. GENERAL INFORMATION 3.1. Assessment Program Expectations 3.2. Assessment Benefits 3.3. Graded Approach 3.4. Purpose of Assessment 3.4.1. Assessment Elements 3.5. Types of Assessment 3.5.1. Management Assessment 3.5.2. Independent Assessment 3.6. Organizational Activity Levels 3.6.1. Process Level Assessments 3.6.2. System Level Assessments 3.6.3. Program Level Assessments 3.7. Assessing for Compliance and Performance 3.7.1. Compliance Assessments 3.7.2. Performance-Based Assessments 4. GUIDELINES 4.1. Assessment Personnel 4.2. Assessment Program Planning 4.2.1. Assessment Programs 4.2.2. Management Assessment Planning 4.2.3. Independent Assessment Planning 4.2.4. Planning Updates 4.3. Assessment Integration 4.4. Assessment Plan 4.5. Performance Criteria 4.6. Assessment Planning Tools 4.7. Independent Assessment Process 4.7.1. Pre-assessment Meetings 4.7.2. The Entrance Meeting 4.7.3. Performing Independent Assessments 4.7.4. Independent Assessment Techniques 4.7.5. The Exit Meeting 4.7.6. Assessment Reporting 4.7.7. Releasing and Responding to Assessment Reports 4.7.8. Corrective Actions 4.7.9. Follow-up Assessments 4.8. Management Assessment Process 4.8.1. Defining the System 4.8.2. Assessment Scheduling 4.8.3. Performing Management Assessments 4.8.4. Assessment Reporting APPENDIX A - STANDARDS, HANDBOOKS, AND OTHER REFERENCES A.1. Standards and Handbooks A.2. Other References APPENDIX B - ASSESSMENT FUNCTIONAL AREAS APPENDIX C - TOOLS FOR ASSESSMENT PLANNING AND CONDUCT C.1. Checklists C.2. Matrices C.3. Flowcharts C.4. Cause-and-Effect Diagrams C.5. Tree Diagrams C.6. Information Systems APPENDIX D - ACRONYMS AND ABBREVIATIONS