DOE G 414.1-1, Assessment Guide for QA

DOE G 414.1-1 August 1996 IMPLEMENTATION GUIDE for Use with INDEPENDENT and MANAGEMENT ASSESSMENT REQUIREMENTS of 10 CFR PART 830.120 and DOE 5700.6C QUALITY ASSURANCE ASSISTANT SECRETARY FOR ENVIRONMENT, SAFETY AND HEALTH INTERIM GUIDE - FOR INTERIM USE AND COMMENT CANCELE D CANCELE D DOE G 414.1-1 August 1996 FOREWORD This Department of Energy (DOE) Guide is approved for use by the Office of Environment, Safety, and Health, Office of Nuclear Safety Policy and Standards, and is available for use by all DOE components and their contractors. Beneficial comments (recommendations, additions, deletions, and any other pertinent data) that may improve this document should be sent to: Gustave E. Danielson, Jr. EH-31/CXXI-3 U.S. Department of Energy 19901 Germantown Road Germantown, MD 20874-1290 Phone (301) 903-2954 Fax (301) 903-6172 e:mail - "bud.danielson@hq.doe.gov" As part of the DOE directives system, guides are issued to provide supplemental information regarding the Department's expectations for fulfilling requirements contained in rules, Orders, Notices, and regulatory standards. Guides are also used to identify government and non-government standards and methods that DOE finds acceptable for implementing the Department's requirements. Guides are not substitutes for requirements, nor do they replace technical standards, which are used to describe established practices and procedures for implementing requirements. CANCELE D CANCELE D August 1996 DOE G 414.1-1 ii BACKGROUND This Guide was prepared by a Policy/Standards Process Improvement Team established by the Assistant Secretary for Environment, Safety, and Health (ES&H). The Process Improvement Team identified all assessment requirements in the DOE ES&H Orders that were effective as of November 1994 and determined that 10 CFR 830.120 and DOE Order 5700.6C contain adequate requirements to ensure assessments would be performed effectively. In addition, the team decided that due to the significant change in the number and type of requirements a guide should be developed to retain necessary information from canceled Orders and convey current trends in assessment methodology. The work of the Process Improvement Team has yielded several positive outcomes. Over 400 requirements rooted in more than 40 Orders have been replaced by the 2 found in 10 CFR 830.120 and DOE Order 5700.6C. Factors which contribute to "stovepipe" assessment programs have been deleted. Flexibility to plan and implement assessment programs that provide value to an organization has been increased. Each organization's ability to respond to its' customer(s) needs is greatly enhanced. Redundant and excessive assessments are no longer mandated or implied. Organizations can focus assessment resources on those activities that pose the greatest risks and stand to benefit the most from improvement opportunities uncovered. CANCELE D CANCELE D DOE G 414.1-1 August 1996 iii ACKNOWLEDGEMENTS The Department wishes to acknowledge and thank the following team members and additional contributors. Without their expertise and dedication, the effort could not have succeeded. Process Improvement Team Members Gustave E. Danielson, Jr., Team Leader, Office of Nuclear Safety Policy and Standards W. Earl Carnes, Office of Nuclear Safety Policy and Standards Rick Cutshaw, Bechtel Nevada, Incorporated Marie Hirzel, Chicago Operations Office Rudolph Hirzel, Chicago Operations Office Mike Karmis, Chicago Operations Office Frank Russo, Office of Oversight, Planning and Analysis Eric Swanson, EG&G Rocky Flats, Incorporated

Additional Contributors Margy Beckmeyer, Westinghouse Savannah River Company Paul Chimah, Office of Engineering Operations Support Amy Ecclesine, Los Alamos National Laboratory Denise Hawkins, Oak Ridge Institute for Science and Education Gene Langston, Science Applications International Corporation John Lee, Oakland Operations Office Mark McGovern, Digital Systems Research, Incorporated John Palmer, Lawrence Livermore National Laboratory Keith Rademacher, Rust Geotech Raymond Reinhart, Fermco Scott Rogers, Office of Environmental Management Dennis Ryder, Battelle-Pacific Northwest National Laboratories Marvin Shear, Brookhaven National Laboratory Steven Stein, Brookhaven National Laboratory Gary Winner, Argonne National Laboratory CANCELE D CANCELE D August 1996 DOE G 414.1-1 iv CONTENTS PARAGRAPH PAGE FOREWORD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii ACKNOWLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii I. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 II. APPLICATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 III. GENERAL INFORMATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3.1 Drivers and Benefits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3.2 Graded Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 IV. GENERAL INFORMATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.1 Purpose of Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 4.2 Types of Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4.3 Organizational Activity Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.3.1 Process Level Assessments . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.3.2 System Level Assessments . . . . . . . . . . . . . . . . . . . . . . . . . 6 4.3.3 Program Level Assessments . . . . . . . . . . . . . . . . . . . . . . . . 7 V. GUIDELINES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.1 Assessment Personnel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 5.2 Assessment Program Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 5.2.1 Assessment Program Planning . . . . . . . . . . . . . . . . . . . . . . 10 5.2.2 Management Assessment Planning . . . . . . . . . . . . . . . . . . . 11 5.2.3 Independent Assessment Planning . . . . . . . . . . . . . . . . . . . 11 5.2.4 Planning Updates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.3 Assessment Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.4 Assessment Agendas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.5 Performance Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 5.6 Assessment Planning Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.7 Assessment Conduct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.7.1 The Pre-Assessment Meeting . . . . . . . . . . . . . . . . . . . . . . . 15 5.7.2 Assessment Conduct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.7.3 Assessment Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.7.3.1 Document Review . . . . . . . . . . . . . . . . . . . . . . . . . . 17 5.7.3.2 Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 5.7.3.3 Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 5.7.3.4 Inspection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 5.7.3.5 Performance Testing . . . . . . . . . . . . . . . . . . . . . . . . 18 5.7.4 The Post-Assessment Meeting . . . . . . . . . . . . . . . . . . . . . . . 19 CANCELE D CANCELE D DOE G 414.1-1 August 1996 v CONTENTS PARAGRAPH PAGE 5.7.5 Assessment Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.7.5.1 Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . 20 5.7.5.2 Observation Section . . . . . . . . . . . . . . . . . . . . . . . . 20 5.7.5.3 Results Section. . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.7.5.4 Attachments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5.7.6 Corrective Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 5.7.7 Follow-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 APPENDIX A - Consensus Standards and References . . . . . . . . . . . . . . . . . . A-1 APPENDIX B - Assessment Functional Areas . . . . . . . . . . . . . . . . . . . . . . . . . B-1 APPENDIX C - Tools for Assessment Planning and Conduct . . . . . . . . . . . . . . C-1 APPENDIX D - Independent Assessment Planning . . . . . . . . . . . . . . . . . . . . . D-1 CANCELE D CANCELE D DOE G 414.1-1 August 1996 Refer to these documents for specific exemptions.1 The term "products and services" is used in a generic sense and includes the programs, systems, and work processes2 that support them. 1 IMPLEMENTATION GUIDE for use with 10 CFR 830.120 I. INTRODUCTION The Department of Energy (DOE) Quality Assurance (QA) rule 10 CFR 830.120, and Order 5700.6C establish requirements for DOE and its contractors to perform Management and Independent Assessments using appropriate standards wherever applicable . This Guide provides information concerning the1 establishment and implementation of effective assessment processes. Assessments are an important step in the Plan-Do-Check-Act cycle. As such, they can add value to products and services by linking management and the conduct of2 work to meaningful improvement actions. To enable management to take meaningful actions, an assessment must embody the following principles. • Workers and managers are involved in the assessment process to ensure results contribute to improved performance of the programs, systems, and work processes. • Managers receive timely, objective feedback on the effectiveness of policies, requirements, and standards, and whether the organization complies with them. • The coordination and integration of the assessment process is maximized and the work being assessed is disrupted as little as possible. CANCELE D CANCELE D August 1996 DOE G 414.1-1 Successful implementation of the assessment requirements will be enhanced by embracing the philosophy described3 in DOE/HR-0066, "Total Quality Management Implementation Guidelines."

2 • The culture seeks continuous quality improvement and protects people and3 the environment from harm. II. APPLICATION All DOE products/services (and the programs, systems, and processes that deliver them) can be assessed over their entire life cycle using this Guide. Environment, Safety and Health (ES&H) and Safeguards and Security activities are also considered products/services and are subject to the assessment requirements (see Appendix C). Technical standards and methods that DOE finds acceptable for meeting the requirements of 10 CFR Part 830.120, and DOE 5700.6C are referenced in this guide. This guide also expands upon the Implementation Guide for use with 10 CFR Part 830.120, G-830.120 Rev. 0, April 1994. Assessments conducted in accordance with this Guide and appropriately adopted standards will satisfy the requirements of 10 CFR Part 830.120 and DOE Order 5700.6C. Alternative methods may be acceptable to DOE if the methods are demonstrated to achieve an adequate level of safety and quality. This Guide also provides a basis for determining the adequacy of Implementation Plans, Quality Assurance Programs, Integrated Safety Management Systems (reference DOE P 450.4), and Work Smart Standards (reference DOE P 450.3) prepared in response to the above requirements. III. GENERAL INFORMATION 3.1 Drivers and Benefits The success of an organization depends upon the extent to which its products and services satisfy customer requirements and expectations. Delivery of products and services occurs through the implementation of programs, systems, and processes. Responsibility for satisfying the customer lies with each member of an organization. The quality program described in 10 CFR Part 830.120 and DOE 5700.6C provides CANCELE D CANCELE D DOE G 414.1-1 August 1996 3 a results-oriented management system, that focuses on the customer and embraces continuous improvement. The assessment component of this management system builds confidence that organizations can deliver on customer expectations. Assessments also provide objective evidence of those areas in which management and workers need to improve on their ability to perform on the organizational mission and achieve managements goals. Effective assessments also prepare an organization for "third-party" voluntary or involuntary assessments performed by external governmental and non- governmental bodies. Third-party bodies typically assess for performance and/or compliance against national and international standards. Depending on their focus, voluntary third-party bodies are known as quality system registrars, laboratory accreditors, or product certifiers. An organization may have its "Quality System" registered as compliant with ISO 9001. Calibration/testing laboratories may seek accreditation to ISO Guide 25 and associated pertinent testing standards. Manufacturers can obtain "product certifications" to health, safety, and industrial standards. The emerging ISO 14000 international standards for "environmental management systems" have also spawned a registration process. DOE contractors can participate in the Voluntary Protection Program and attain recognition for excellence in safety and health management. In addition, national and State "Quality Awards" (another growth area of third-party assessment) focus on integrated management systems and customer service. Most notable are the Malcolm Baldrige National Quality Award and the Presidential Award for Quality.

Involuntary third-party assessments are conducted by regulatory or oversight bodies and most often measure compliance with regulatory requirements and standards. Familiar examples include the Nuclear Regulatory Commission, the Environmental Protection Agency, the Occupational Safety and Health Administration, the General Accounting Office, the Defense Nuclear Facilities Safety Board, and State/local health and safety agencies. Regardless of their focus or purpose, third-party bodies have at least one common interest; that is, determining whether an organization has established and implemented an effective assessment process. In the regulatory arena, a positive finding, coupled with prompt corrective actions by management can mitigate civil CANCELE D CANCELE D August 1996 DOE G 414.1-1 Quality Management Graded Approach Working Paper, Training Resources and Data Exchange (TRADE) Quality4 Management Special Interest Group, September, 1993 4 and criminal enforcement penalties, thereby reducing or eliminating monetary fines and reducing criminal sentences. Management must be aware of, and prepared to capitalize on, the diverse drivers and benefits of the assessment process when developing a program that embodies the principles described in Section I. 3.2 Graded Approach This Guide and the technical standards referenced herein should be applied using a graded approach. A survey of DOE organizations implementing DOE Order 5700.6C found most respondents "grade their applications based upon risk, using both the assessment of the probability of failure and the assessment of the consequence of failure." This Guide should be used to tailor specific4 methodologies to the activities being assessed. Within the context of this Guide, worker assessment requires the lowest level of formality and rigor. Conversely, an operational readiness review is an example of an assessment which demands a high degree of formality and rigor. IV. GENERAL INFORMATION 4.1 Purpose of Assessment Establishing and implementing an effective assessment program is an integral part of every management system. Assessments are an important feedback mechanism that provide management with the information to evaluate and improve any aspect of an organization: organizational progress in reaching strategic goals and objectives; the adequacy and implementation of management programs developed to achieve the mission; the performance capability of safety management systems ; the quality of products and services; the degree of5 CANCELE D CANCELE D DOE G 414.1-1 August 1996 The commitment to safety management systems is found in the Department of Energy Plan for the Development and5 Implementation of Integrated Safety Management (Implementation Plan for DNFSB Recommendation 95-2), dated April 18, 1996, and the associated policy P 450.4 and guidance. 5 "Management Assessment. Managers shall assess their management processes. Problems that hinder the organization from achieving its objectives shall be identified and corrected." (10 CFR 830.120) compliance with contractual and regulatory requirements. 4.2 Types of Assessment The Department's Quality Assurance rule and Order establish requirements for two types of assessment: management and independent. A third type, "worker assessment," is implied by the fundamental principle that each person is responsible for the quality of the work they perform. Taken together, these three assessment types are often referred to as "self assessments." Management and independent assessments each have a specific focus as defined by the Quality Assurance rule and Order. The following paragraphs reiterate those requirements and provide additional information regarding their focus.

Management assessments are used to look at the total picture of how well their management system meets the customer's requirements and expectations. The emphasis of management assessment is on management issues that affect performance processes such as: strategic planning; qualification; training; staffing; organizational interfaces; communication; cost control; and mission objectives. The purpose of this type of assessment is to identify management aspects of performance and make improvements through an introspective analysis to determine if the management infrastructure is properly focused on achieving desired results. CANCELE D CANCELE D August 1996 DOE G 414.1-1 6 "Independent Assessment. Independent assessments shall be planned and conducted to measure item and service quality, to measure the adequacy of work performance, and to promote improvement. The group performing independent assessments shall have sufficient authority and freedom from the line to carry out its responsibilities. Persons conducting independent assessments shall be technically knowledgeable in the areas assessed." (10 CFR 830.120) Independent assessments evaluate the performance of work processes with regard to requirements and expectations for achieving the mission and goals of the organization. The focus of independent assessments should be on the items and services produced and associated processes with the objective of improving the product/service performance and process effectiveness. (Independence is defined as not having direct responsibilities in the areas being assessed.) Worker assessments simply involve the worker routinely comparing the processes and products and services to defined expectations. This includes ensuring that the right products and services are being provided, the defined processes and procedures are being followed, and customer needs are being satisfied. 4.3 Organizational Activity Levels In order to shape a comprehensive assessment program, and one which optimizes the application of each assessment type, it may be helpful to view the organization as having three interlinked levels of activity. For the purpose of this discussion these levels are referred to as "process," "system," and "program." Where a "process" is a collection of steps or actions that yield some intermediate outcome, a "system" is made up of two or more processes which may operate independently or are interdependent and may yield a complete product or service. A "program" is the most complex level and consists of multiple interdependent "systems" which often require many interfaces to provide the desired product or service. Management and independent assessments can be applied at all three levels but will examine different aspects of them. 4.3.1 Process Level Assessments involve the examination and verification that work controls are being implemented effectively. This level of assessment is critical for ensuring that workers, the public, and the environment are protected from harm and should involve workers checking their own work. CANCELE D CANCELE D DOE G 414.1-1 August 1996 7 4.3.2 System Level Assessments focus on whether appropriate leadership and support systems are provided to enable the implementation of work processes. These assessments are performed to ensure human and material resources are being properly utilized to achieve an organization's mission and objectives. This level of assessment may range from informal daily oversight of performance to formal periodic evaluations using established protocols.

4.3.3 Program Level Assessments are used to determine whether overall organizational programs are properly established and implemented. They are appropriate for evaluating complex organizations from several perspectives. Consequently, these program assessments usually include examining the integration of the many systems (including an emphasis on environment, safety and health factors) designed to achieve organizational goals and customer expectations. The following description provides an example of the different types and levels of assessment applied to an organization's work control methods. Some of the elements within the work control system are planning, hazard identification, scheduling, performing, verifying/testing, and documenting the work performed. Process level assessments would be performed by workers and independent assessors to verify that the procedures have been complied with; the work package accurately reflects the task and associated hazards. An independent assessment at the system level could be performed to determine if all necessary elements and interfaces are addressed to assure it is capable of consistently meeting requirements and customer expectations. A management assessment of the work control system would determine the cost and resource allocation issues that impact the system. At the program level a Maintenance Management program, which relies on the work control system, would use results from the process and system level assessments as input in determining the effectiveness of the entire Maintenance program. This program assessment could be performed as either a management assessment or an independent assessment. A management assessment may focus on comparing the strategic goals for maintenance to actual performance to determine if the rewards and recognition plan targeted to improve maintenance have had the desired affect. The independent assessment could evaluate the program results against contractual and regulatory commitments. CANCELE D CANCELE D August 1996 DOE G 414.1-1 8 The following tables help to illustrate the different types of assessments applied to processes, systems, and programs. This represents one approach for implementing the requirements. Ultimately, management with support from the assessment professionals is responsible for planning the balance and application of independent and management assessments to assure they add value to the organization. TABLE 1. SENIOR MANAGEMENT PERSPECTIVE Level/Type Management Independent Worker Program C B System c B a Process B A Legend: The letters represent potential topics to be assessed. The letter size and boldness depicts the relative emphasis for each level. A Item and service quality: procedure compliance; work performance; day-to-day activity; continuous improvement. B Item and service quality: product performance and reliability meeting requirements and expectations; interface/linkage of multiple processes; process effectiveness; adequacy of policy, plans, and procedures for work performance; continuous improvement. C Management process issues for organizational achievement: mission success; strategic goals/objectives; planning, performance measures; customer satisfaction; leadership; policy deployment; qualification, training, staffing, organizational interfaces, cost control, management impact on work performance; effectiveness; continuous improvement efforts and strategy.

From the Senior Management perspective, emphasis of management assessments should be on the program level and concerned primarily with strategic issues. Data on systems performance can be rolled up from middle- and first-line management assessments and independent assessments. Examples of comprehensive assessment models at the program level include the criteria for; the Malcolm Baldrige National Quality Award, Operational Readiness Reviews, and the Voluntary Protection Program. Independent assessment emphasis is placed on system performance in support of programs and to determine its ability to deliver product and services that meet customer expectations. Independent assessments may also be used to confirm management assessment results where organizational vulnerability is high (e.g., potential regulatory penalty, ES&H significant hazard). CANCELE D CANCELE D DOE G 414.1-1 August 1996 9 TABLE 2. FIRST-LINE MANAGEMENT PERSPECTIVE Level/Type Management Independent Worker Program c B System C B a Process C B A Legend: The letters represent potential topics to be assessed. The letter size and boldness depicts the relative emphasis for each level. A Item and service quality: procedure compliance; work performance; day-to-day activity; continuous improvement. B Item and service quality: product performance and reliability meeting requirements and expectations; interface/linkage of multiple processes; process effectiveness; adequacy of policy, plans, and procedures for work performance; continuous improvement. C Management process issues for organizational achievement: mission success; strategic goals/objectives; planning, performance measures; customer satisfaction; leadership; policy deployment; qualification, training, staffing, organizational interfaces, cost control, management impact on work performance; effectiveness; continuous improvement efforts and strategy. The first-line management perspective would emphasize the capability of systems within their control and processes that support them. To assure that these systems contribute to program goals, managers must evaluate system performance based on these goals. The independent assessment focus would not change from Table 1. Workers are relied upon to verify that the process delivers item and service quality and contributes to the system. V. GUIDELINES 5.1 Assessment Personnel Assessment personnel facilitate continuous process improvement by identifying ways in which programs, systems, and processes can be improved and by providing information to management and process owners (i.e., workers). The assessor should be able to collect performance data through interviews, document reviews, observation, and inspection, communicate results effectively both orally and in writing, and understand the basics of personal interaction and behavior. CANCELE D CANCELE D August 1996 DOE G 414.1-1 10 Both management and independent assessments must be accomplished by trained and qualified individuals who are knowledgeable of the program, system, or process being assessed. An effective assessment may be accomplished through use of an assessment team with combined skills and experiences. Training for assessors should address the policies and procedures of the assessing organization. To enhance assessment performance and capability, new assessment personnel should participate in on-the-job training where they work with a qualified, experienced assessor prior to being considered fully trained or receiving a required qualification. Further guidance on assessor training and qualification is provided in DOE Orders, guides, and the standards in Appendix A.

5.2 Assessment Program Planning 5.2.1 Assessment Program Planning Assessment programs should be developed to the level of rigor and detail required to assure adequate review of programs, systems, and processes. An assessment program is a guide for the overall process and ensures assessments are conducted in a cost effective, efficient manner. Items which are considered essential for a comprehensive assessment program include: • Assessment scheduling, planning approach, and logic (including how independent and management assessments are balanced). • How performance criteria are determined/developed. • Recognition and use of third-party assessment results (accreditation, certifications, registrations, and regulatory). • Assessment ethics and behaviors. • Qualifications and training of assessment team personnel. • Protocols for conduct, including interfaces and meetings. • The format/review of assessment plans and agendas. • How concerns (findings, observations) are reported, especially imminent danger issues. • The follow-up and verification of concerns. • Assessment records that will be retained, retention periods, and protection. CANCELE D CANCELE D DOE G 414.1-1 August 1996 11 5.2.2 Management Assessment Planning Management assessments must be planned in a systematic manner by the individual managers to address all areas under their responsibility, and to focus on those areas presenting the greatest risk for failure, potential for improvement, or areas that have not been covered by an independent assessment. Senior management should retain overall responsibility for the planning and performance of management assessments. Planning for management assessments should be flexible in terms of detailed scheduling, allowing for the process to be used whenever deemed appropriate. Planning of individual assessments should consider the following: • Descriptive title or name of the assessment area; • Brief description of the area or activity to be evaluated; • Identification of the team leader and team members; • Schedule for the start and completion of the assessment, including issuance of the assessment report; and • Other information related to the actual assessment evaluation (e.g. performance objectives, management systems, resource availability, efficiency measures, effectiveness measures, etc.). 5.2.3 Independent Assessment Planning The purpose of assessment planning is to assure that the assessments are value- added, risk-based, comprehensive, and have the least amount of disruption to the program, systems, or work process being assessed. The level of planning will vary significantly depending on scope, breadth, and complexity of the system or process being assessed. The planning process should provide for input from the assessed organizations and consideration of their customers and stakeholders. Specific budget requirements and required resources and support should be identified as early in the planning process as possible. Administrative issues, such as the need for any review documents before the actual assessment visit and considerations for travel and clearances should be addressed. It is important to remember that each assessment needs to characterize a program, system, or process during a limited time. Effective planning will ensure this occurs. CANCELE D CANCELE D August 1996 DOE G 414.1-1 12

Assessments should be conducted at a frequency commensurate with the hazards, status, and importance of the program, system, or work process and should be focused on worker health and safety, public health and safety, environmental protection, community concern, compliance and liability, and business efficiency/productivity. Complexity, reliability, risk, and economic considerations must also be considered when planning and scheduling assessments. The application of a graded approach using a risk-based decision-making process will ensure that resources are applied in a manner that provides the greatest benefit to the assessed organization and their customers. There are many scheduling and planning approaches to satisfying these requirements. Appendix D provides one approach to planning and scheduling independent assessments using an integrated, risk based approach. 5.2.4 Planning Updates Assessment planning should be periodically reviewed and modified as new information on the facility or organization is obtained. An assessment that finds good performance should be used as justification to reduce the frequency and depth of future assessments. Areas of poor performance should receive increased attention, especially if there are indications that management is unable to correct identified problems, since recurring and cumulative deficiencies, even in a low hazard operation, may decrease the likelihood of its achieving its mission. 5.3 Assessment Integration Multiple layers of assessment of a single organization or facility do not add value and disrupt the organization from accomplishing its mission. Therefore, routine communication and trust among the various levels of assessment bodies is essential in coordinating plans. DOE Operations/Field Offices are the logical choice for assuming this coordinating role for assessments of contractor performance. This task involves identification of overlapping and redundant assessments to reduce their negative impact on work performance. Once identified, assessments can be coordinated and consolidated in several ways. • Change assessment scope to prevent two assessment organizations from performing the same type of assessment on the same subject. CANCELE D CANCELE D DOE G 414.1-1 August 1996 13 • Combine separate assessment teams to evaluate the subject in a single visit. • Cancel an assessment based on a mutual recognition agreement to share results of the assessment. Where significant redundancies exist, as when many contractors perform supplier quality assessments of analytical laboratory services, recognition and use of accredited third-party assessment results and shared information from other contractors should be considered. Each of these methods requires staff time to develop, implement, and manage. However, the result can be substantial savings in resources while enhancing the ability of the assessing organization to cover a greater number of activities. 5.4 Assessment Agendas The use and detail of an assessment agenda will vary depending on what is being assessed and whether the assessment is a management assessment or an independent assessment. Agendas are used to scope and plan individual assessments and should include input not only from the assessed organization but from their customers and stakeholders as well. A documented assessment agenda not only allows for the communication of expectations to the assessed organization, but also allows the assessment team to focus its activities more effectively. The scope of the assessment should be defined in terms related to the assessed organization's mission and goals so the focus and value of the assessment will be clearly understood. The extent of detail included in the assessment agenda should be commensurate with the protocols of both the assessed and assessing organizations. The assessment agenda should include the following items.

• Team members and their qualifications, organization, and interfaces. • A description of the assessment scope and performance criteria. • Dates of the assessment. • Schedule of assessment meetings - pre-assessment, daily, and post- assessment. • List of documents to be provided the assessment team when it arrives. • Request for office space, phone lines, and other administrative support as required. • Request for a site safety briefing. CANCELE D CANCELE D August 1996 DOE G 414.1-1 14 • Request for points of contact for each functional area. The organization being assessed should be contacted and provided an assessment agenda as soon as possible before the assessment (unless regulations or contracts specify other contact protocols). For management assessments, the agenda will usually not involve as much information as that listed above because, as an internal assessment, requests for documents, office space, etc., would not be required. In addition, notification may be less formal than that used for independent assessments. 5.5 Performance Criteria Assessments seek to ensure that performance expectations as defined by management and process owners are being met. The assessor should clearly understand the basis for performance in a program, system, or process. Requirements for performance as dictated by the scope of the assessment must be researched and understood. These requirements are found in the following source documents. The performance information provides insight into organizational performance. • Source Documents Federal and State Regulatory Requirements Appropriate Codes and Standards Contract Requirements DOE Orders, Manuals, and Notices Implementation Plans Implementation Procedures Facility Safety Documents Policy and Mission Statements DOE Approved Work Smart Standards Sets Standards/Requirements Identification Documents (S/RID) Plans and Programs • Performance Information Reports from Outside Regulators Facility Operations Reports Performance Reviews CANCELE D CANCELE D DOE G 414.1-1 August 1996 15 Previous Assessment Reports Internal Inspections, Reviews, and Reports Corrective Action Plans and Status Reports Concerns and Occurrence Reports Performance Indicators Requirements contained in these documents are selected based upon impact on the assessed organization's mission and the relationship to the scope of the assessment. From selected requirements, objective statements (performance measures) are developed for determining whether or not a program, system, and/or process is working. From these measures, the specific performance criteria (based on written programs, DOE Orders, rules, etc.) are developed and tools selected for conducting the appraisal. In developing performance criteria, assessment personnel cannot reinterpret or redefine requirements specified in the source documents. 5.6 Assessment Planning Tools Assessment planning tools vary in format, content, and level of detail but all have one thing in common - they help focus the assessor on the mission and objectives of the program, system, or process being assessed. Application of planning tools prior to an assessment ensures more effective use of time and ensures the focus of the appraisal is identified and maintained during the course of the assessment. Assessment planning tools are often used to relate the performance criteria to the established assessment scope and may include lists of interview questions, major elements of programs, or detailed process work steps. Similar to a road map, each tool is used to remind the assessor of where he/she is going and the items likely to be encountered along the way. Planning tools are extremely useful when the assessment basis is complex, or the requirements come from multiple sources. Typical planning tools include matrices, flow-charts, cause-effect diagrams, tree diagrams, checklists, and information systems. (See Appendix C for examples and further discussion of these tools).

5.7 Assessment Conduct 5.7.1 The Pre-Assessment Meeting CANCELE D CANCELE D August 1996 DOE G 414.1-1 16 A pre-assessment meeting is the appropriate place to "set the stage" for the performance of a positive and productive independent assessment. (Such a meeting may not be necessary for a management assessment.) This meeting is held between personnel from the assessing organization and the organization being assessed and usually takes place at the assessed organization's location/facility. The purpose of this meeting is to discuss the scope of the planned assessment and to establish interfaces between the two organizations. This meeting is also used to establish how concerns involving imminent danger or regulatory noncompliance will be communicated. Protocols to be followed during the assessment should be clarified during the meeting, which usually includes the following. • The purpose and scope of the assessment, including authority for conducting the assessment. • The assessed organization's mission, program, systems, and processes. • Length of the assessment. • The source documents and performance information that form the basis for the performance criteria to be used. • Identification of knowledgeable individuals from the assessed organization as points of contact for each assessor. • Any restrictions for the collection and/or disposal of assessment notes/records by the assessors. • Logistics, including work area, working hours, lunch hours, etc. • The time and place of the post-assessment meeting. • The time and location of daily meetings. 5.7.2 Assessment Conduct The assessment should be conducted in accordance with established requirements developed by the assessing organization. Any agendas or specific protocols established during the pre-assessment meeting are used to ensure that the assessment is conducted effectively and safely. Assessors should keep their points of contact informed of their activities to preclude surprises during the post- assessment conference. This may include requests for additional assistance or the communication of concerns that require immediate action on the part of the CANCELE D CANCELE D DOE G 414.1-1 August 1996 17 assessed organization. Timely communication will allow the assessed organization to verify the accuracy of observations and provide relevant facts and background on the issues. If a team is used to perform the assessment, daily meetings may be useful to ensure continuity and overall focus. Daily meetings provide team leaders with information on completion status of the assessment plan and identify issues requiring additional action (e.g., clearances, access, requests for personnel or material, and impasse resolution). These meetings are also the setting for informing other team members of issues that may be of interest in their assigned scope or for integrating data gathered by the various assessors. The meetings should be brief so that they do not reduce the team members' time with the processes and people they are to assess. It is important that sufficient information is gathered during the assessment to determine whether an activity meets the performance criteria established. The assessor must be able to clearly state the criterion impacted by the activity and whether identified results also impact the mission/goals of the organization. To accomplish this, the assessor may deviate from the assessment agenda to determine the extent and significance of an issue. Deviations that affect the assessor's ability to complete the assessment agenda must be made known to the team leader and the assessed organization.

5.7.3 Assessment Techniques Effective assessments use a combination of tools and techniques to maximize the productivity of assessment time and resources. Such assessment techniques include document reviews, interviews, observation, inspection, and performance testing. Use of the planning tools discussed in Attachment 3 also allows for more complex analysis and systematic coverage of the areas being assessed. In using these techniques, the assessor should not forget that the objective is to verify accomplishment of an organization's mission. To save time, the assessor should gather only data and information relevant to overall program performance and the achievement of that program's objective. In using any of these techniques, assessors should maintain good records of the assessment results. These may include personal notes or other information to CANCELE D CANCELE D August 1996 DOE G 414.1-1 18 support the assessment. These records are useful in writing the report and any associated findings and recommendations and will become invaluable if questions arise during the report review process. All classified notes should be disposed of properly in accordance with established and agreed upon procedures. A discussion of each of the techniques is provided below: 5.7.3.1 Document Review is used extensively during an assessment to substantiate the information obtained during interviews and observation. During the course of an assessment, questions may arise concerning what is heard and seen. The review of documents (including logs, procedures, work orders, and other data) provides a method for answering these questions and validating the assessment results. The drawback of document review is that the accuracy of the records cannot be ascertained by review alone. This technique must be combined with interviews, observation, and/or inspection to complete the picture of performance. Records and documents must be selected carefully to ensure that they adequately characterize the program, system, or process being assessed. 5.7.3.2 Interviews provide a means to verify the results of observation, document review, and inspection. In addition, interviews allow the responsible person to explain and clarify those results. The interview helps to eliminate misunderstandings about program implementation and provides a dialogue between the assessor and the organization being assessed, who can explain apparent conflicts or recent changes, and describe the functional organization and program expectations. Tools developed during assessment planning are used to prepare for the interview. Assessors should also prepare questions in advance to keep the interview focused. 5.7.3.3 Observation, the viewing of actual work activities, is often considered the most effective technique for determining whether performance is adequate. Assessors should understand the effect their presence has on the person being observed and convey an attitude that is helpful, constructive, positive, and unbiased. The primary goal during observation is to obtain the most complete picture possible of the performance, which should then be put into perspective relative to the overall program, system, or process. Before drawing final conclusions, the assessor should verify the results through at least one other technique. CANCELE D CANCELE D DOE G 414.1-1 August 1996 19

5.7.3.4 Inspection is performed to verify the adequacy and condition of physical facilities, systems, equipment, and components. Usually, inspections are used to obtain additional information concerning other items evaluated during the assessment, such as equipment labelling, configuration control, the status of system line-ups, adequacy of construction, or proper material storage. Inspection may also be performed to gain information and data in preparation for interviews and/or work observation. While on these inspections, the assessor must heed all security and safety requirements. It is always a good practice to be accompanied by someone familiar with the facility. 5.7.3.5 Performance Testing is used to observe the response of personnel or equipment by creating a specific situation and noting performance. This technique is especially useful when activities of interest would not normally occur during an assessment visit. It is also used when timeliness and appropriateness of the response is critical to an organization such as safeguards and security, emergency response, etc. 5.7.4 The Post-Assessment Meeting Both independent and management assessment programs can gain value from a post-assessment meeting. This meeting is used primarily to present the assessment summary and provide the assessed organization an opportunity to verify the factual accuracy of assessment results. To facilitate this, assessors should be prepared to provide detailed supporting information for those results. This meeting also offers an opportunity for the assessed organization to present its management position and any plans for addressing the results. Reasonable time should be allowed to discuss any concerns, but this meeting should not be used to argue the assessment agenda or methodology, which should have been discussed at the pre-assessment meeting. 5.7.5 Assessment Reporting Independent assessment programs and, less formally, management assessment programs, call for the development of an assessment report as a vehicle to communicate the issues identified during CANCELE D CANCELE D August 1996 DOE G 414.1-1 20 an assessment. The assessment report may be informal or formal depending on the level of assessment performed. The report must provide a clear picture of the results in terms of the programs, systems, and processes assessed. The report should be clear and easy to understand and should include only facts that directly relate to assessment observations and results. It should include sufficient information to enable the assessed organization to develop and implement appropriate improvement plans and check the report for accuracy (if such a check was not done during the assessment). Every effort should be made to ensure assessment reports are concise, accurate, and understandable. In preparing the report, authors should remember that many people who will read the report have had no active role in the assessment and the report may be their only source of information regarding its conduct and results. For example, summary information may be clearer or more easily understood if presented graphically. Specific report formats may vary considerably from one organization to the next. In developing a report format, the assessment organization should solicit input from report recipients to ensure it meets their needs. An independent assessment report usually includes the following sections. A management assessment report may be less formal and may only require the executive summary.

5.7.5.1 Executive Summary. This summary should be a brief, stand-alone document. It should describe the programs, systems, and processes assessed and the overall assessment results. The Executive Summary should describe strengths and weaknesses affecting the assessed organization's mission so that meaningful action can be taken for improvement. 5.7.5.2 Observation Section. Each part of this section should focus on the established assessment scope and the identified organization mission; otherwise, the recipient of the report will question why a specific area or activity was assessed. The observation section should include general background on the assessment including team members, the scope of the assessment, the methodology used, a summary of the assessment basis and source documents, and a conclusion regarding results. Additionally, each area assessed should be discussed in detail, including specific performance criteria used and summaries of interviews, documents reviewed, observations, and inspections. The summaries CANCELE D CANCELE D DOE G 414.1-1 August 1996 21 contained in this section should support the specific items discussed under the results section. 5.7.5.3 Results Section. This section should include specific problem areas or deficiencies, areas for improvement, or noteworthy practices identified during the assessment. In addition, this section should highlight any recurring problems as indicators of ineffective corrective action by the assessed organization. Each item listed should discuss the basis in sufficient detail to enable further analysis and action to be taken by the responsible organization. Any required post-assessment actions by the assessed organization should also be included. 5.7.5.4 Attachments. Items to be considered for attachment to the assessment report include the assessment agenda, a list of persons contacted, a list of documents reviewed, performance criteria, and the tools used to perform the assessment. Team leaders have overall responsibility for report preparation and approval. The assessment team should have an opportunity to sign the completed report. At a minimum, the final report should be distributed to the management of both the assessed and assessing organizations. Distribution to other organizations (e.g., Defense Nuclear Facilities Safety Board, regulators, etc.) should be defined during the planning phase. The value of an assessment is found in the improvement opportunities it identifies, and the value of assessment information typically diminishes over time. Therefore, the best time to release a report is immediately after the post-assessment meeting, which allows the assessed organization to begin improvement actions right away, yielding the maximum return from those actions. 5.7.6 Corrective Action Management responsible for the activities assessed are also responsible for the development of effective corrective action of the problem areas or deficiencies discovered during the assessment. As a minimum, the corrective action should address: CANCELE D CANCELE D August 1996 DOE G 414.1-1 22 • measures to correct each deficiency; • identification of all root causes for significant deficiencies; • determination of the existence of similar deficiencies; • corrective actions to preclude recurrence of like or similar deficiencies; • assignment of corrective action responsibility; and • completion dates for each corrective action.

For independent assessments, the proposed corrective action should be reviewed for concurrence by the assessment team leader. This will help ensure that the planned actions will be effective in resolving the problem areas and deficiencies reported by the assessment team. 5.7.7 Follow-up After a reasonable period of time has elapsed, follow-up activities should be performed to verify the effectiveness of the corrective action and how it was implemented. There are several ways to verify the implementation of corrective action, including: • a reassessment of the deficient areas; • review of new or revised quality-affecting documents such as manuals, procedures, training records, etc.; • verification during the next scheduled assessment; and • verification by conducting a surveillance covering the areas of concern. The key thing to remember when verifying corrective action implementation is that verification is necessary. A solution to a problem may look good on paper but may not be readily implementable. The failure to adequately identify all root causes will most likely result in a recurrence of the deficiency. Therefore, an appropriate amount of follow-up is necessary to assure the effectiveness of the corrective action process and to reestablish confidence in the item/service assessed. CANCELE D CANCELE D DOE G 414.1-1 August 1996 A-1 APPENDIX A - Consensus Standards and References Consensus Standards The following consensus standards provide additional methods and discussion for implementing the guidance contained herein. A single standard may not fully implement all elements of the requirements. The principles, recommended approaches, and applications contained in these documents must be used in conjunction with 10 CFR 830.120 and DOE 5700.6C to develop and implement assessment processes that meet the requirements. 1. ANSI/ASQC E4-1994, Quality Systems Requirements for Environmental Programs, Part A, Section 2.9, "Quality Assessment and Response", American Society for Quality Control. 2. ANSI/ASQC Q9001-1994, Quality Systems-Model for Quality Assessment in Design/Development, Production, Installation and Servicing, Section 4.17, "Internal Quality Audits", American Society for Quality Control (American National Standard equivalent to ISO 9001). 3. ANSI/ASQC Q10011-1-1994, Guidelines for Auditing Quality Systems- Auditing. 4. ANSI/ASQC Q10011-2-1994, Guidelines for Auditing Quality Systems- Qualification Criteria for Quality Systems Auditors. 5. ANSI/ASQC Q10011-3-1994, Guidelines for Auditing Quality Systems- Management of Audit Programs. 6. ASME NQA-1-1994, Quality Assurance Program Requirements for Nuclear Facility Applications, Basic Section 18, "Audits"; Supplement 18S-1, "Supplementary Requirements for Audits", Appendix 18A-1, "Nonmandatory Guidance on Audits", and Supplement 2S-3, "Supplementary Requirements for the Qualification of Quality Assurance Program Audit Personal", American Society of Mechanical Engineers. CANCELE D CANCELE D August 1996 DOE G 414.1-1 A-2 7. ANS 15.8, Quality Assurance Program Requirements for Research Reactors, American Nuclear Society, 1996. 8. DOE-EM-STD-5505-96, dated 1996, Operations Assessments. 9. DOE-ER-STD-6001-92, dated June 1992, Implementation Guide for Quality Assurance Programs for Basic and Applied Research, Part II.C, Criteria 9 and 10, "Management Assessment" and "Independent Assessment." 10. DOE-STD-3006-95, dated November 1995, Planning and Conducting Operational Readiness Reviews (ORR).

11. DOE/RW/0333P, dated December 18, 1992, Quality Assurance Requirements and Description for the Civilian Radioactive Waste Management Program, Section 18.0, "Audits." 12. ISO 14001.1, Environmental Management Systems - Specification with Guidance for Use, 1996. 13. ISO 14010.1, Guidelines for Environmental Auditing - General Principles, 1996. 14. ISO 14011/1.1, Guidelines for Environmental Auditing - Audit Procedures- part 1: Auditing of Environmental Management Systems, 1996. 15. ISO 14012.1, Guidelines for Environmental Auditing - Qualification Criteria for Environmental Auditors, 1996. 16. International Atomic Energy Agency, 50-SG-QA10, Safety Guide - Assessment, 1996. References The following references provide additional information concerning assessments. CANCELE D CANCELE D DOE G 414.1-1 August 1996 A-3 1. Implementation Guide for 830.120, Quality Assurance Requirements, Section IV, Criteria 9 and 10, "Management Assessment" and "Independent Assessment." 2. DOE/EH-0135, dated June 1990, Performance Objectives and Criteria for Technical Safety Appraisals at Department of Energy Facilities and Sites. 3. DOE/EH-0299, Volume 1, Environment, Safety and health Progress Assessment Manual, Section 2, "Assessment Process." 4. DOE/EH-0497, dated July 1995, Voluntary Protection Program, Part IV: Onsite Review Handbook. 5. NUREG/CR-5151, dated February 1989, Performance-Based Inspections. 6. DOE Student Training Manual, Management Assessment Training, Office of Nuclear Safety Policy and Standards, February 10, 1993. 7. US Environmental Protection Agency, EPA QA/G-3, Guidance for Preparing, Conducting, and Reporting the Results of Management Systems Reviews, 1995 draft. 8. US Air Force Manual, Environmental Compliance Assessment and Management Program, June 1994. 9. DOE Defense Programs/Environmental Restoration and Waste Management memorandum, "Issuance of Defense Nuclear Facilities Safety Board Recommendation 90-2 Program Implementation Instructions," dated September 14, 1994. 10. ASQC Energy Division, Quality Surveillance Guidelines and Quality Surveillance Handbook, ASQC Quality Press, 1992. 11. ASQC Energy Division, Quality Systems Auditor Training Handbook, ASQC Quality Press, 1986 CANCELE D CANCELE D August 1996 DOE G 414.1-1 A-4 12. Michael Coffman, Occupational Safety and Health Auditing, Institute of Internal Auditors, 1991. 13. David McNamee, Audit Excellence: Best Practices and TQM, Institute of Internal Auditors, 1993. 14. Hubert Glover, Environmental Auditing: Risk Assessment Guidelines, Institute of Internal Auditors, 1993. 15. Pam Anderson and Bill Wortman, The Quality Auditor Primer, Quality Council of Indiana, 1992. 16. Dorsey Talley, Management Audits for Excellence, ASQC Quality Press, 1988. 17. Paul Wilson and Richard Pearson, Performance-Based Assessments, ASQC Quality Press, 1995. 18. Hazel R. O'Leary, Distribution of Evaluations and Assessments to the Defense Nuclear Facilities Safety Board, September 7, 1994. Additional technical standards and reference documents used by DOE elements or contractors should be sent to Gustave Danielson at the address in the foreword of this Guide. Please send, as a minimum, the title, author/publisher, and revision date of these standards/documents.CANCELE D CANCELE D DOE G 414.1-1 August 1996 B-1 APPENDIX B - Assessment Functional Areas

Essential environment, safety, and health (ES&H) and safeguards and security functional areas/activities to be evaluated by a comprehensive assessment program are listed below. The list represents basic ES&H requirements, processes, and programs found in regulations, DOE policy, and Orders. This attachment should not be interpreted as limiting application of assessments to these functional areas that ensure ES&H protection, nor should it be interpreted as minimizing the importance of assessing product/service quality and organizational performance. It does, however, illustrate the wide variety of ES&H programs, systems, and processes currently implemented by DOE and its contractors. Underlying these ES&H functional areas is a complex mix of people, hardware, software, and resources all impacting on management and performance of the activities. Interplay of the elements in the mix must be considered to create an effective assessment program that is responsive to customer needs. Accelerator Safety Accident, Incidents, and Unusual Occurrences Aviation Safety Conduct of Operations Construction Safety Criticality Control Emergency Preparedness (NPH) Employee Concerns System Environmental Protection and NEPA Compliance Equipment Modifications Experimental Programs Explosives Safety Facility Operations Fire Protection Firearms Safety Industrial Hygiene Maintenance Management Motor Carrier/Vehicle Safety Nuclear Facility Safety Packaging and Transportation Performance Measures and Indicators Quality Assurance (Programs, Training/Qualification, Quality Improvement, Documents and Records, Work Processes, Design, Procurement, Inspection and Acceptance Testing, Independent and Management Assessment) Radiation Protection Reactor Safety Safeguards and Security Safety Management Systems Safety Analysis Documentation (e.g., SARs and hazard analysis) Standards/Requirements Identification Document(s) Startup and Restart Suspect/Counterfeit Items Technical Safety Requirements Training of Nuclear Facility Personnel Unreviewed Safety Questions CANCELE D CANCELE D August 1996 DOE G 414.1-1 B-2 Occupational Medicine Occurrence Reporting/Trending Worker Protection/Industrial Safety Work Smart Standards Process/Set CANCELE D CANCELE D DOE G 414.1-1 August 1996 C-1 APPENDIX C - Tools for Assessment Planning and Conduct Matrices Matrices are two-dimensional tables showing the relationship between two sets of information. They can be used to show the logical connecting points between performance criteria and implementing actions, or between required actions and personnel responsible for those actions. In this way, matrices are used to determine what actions and/or personnel have the greatest impact on an organization's mission. Matrices are especially useful as a way to focus assessment time and organize assessment conduct. Example 1 Program Development Deficiency Tracking Training Work Control Documents & Records Retention Assessment Director X X Ops Office X X X Ops Support X X X Tech Support X X X X Admin X X Example 2 Administration Chemistry Biology Materials Building Services Engineering Industrial Hygiene A A A A Rad Protection B B B Fire Protection C C C C Industrial Safety A A A Environmental C C C C Personnel Training B A B C Conduct of Ops C C C Quality Assurance A C A C A = 1st Assessment B = 2nd Assessment C = 3rd Assessment

In Example 1, the matrix is used to help the assessor plan the assessment by identifying organizational responsibilities for the different assessment areas. This type of matrix is used to maximize use of assessment time during the site visit. Example 2 is a much broader use of the matrix, which allows the assessor to do the long range planning necessary for ensuring proper application of the assessment program. In this example, the various assessment areas (Y axis) are CANCELE D CANCELE D August 1996 DOE G 414.1-1 C-2 applied against the different organizations to be assessed. Flowcharts Flowcharts illustrate the steps or activities in a process. They provide an excellent tool for examining how various steps in a process are related to each other and whether or not each subsequent activity is receiving what it needs from the previous one. Flowcharts are used to help the assessor understand how a function is being implemented based on written programs and procedures. Flow charts also illustrate reporting relationships and indicate if the handoff of information or materials is adequate. They are especially useful for locating process bottlenecks, which may hinder the organization's mission. Example In this example, the flow chart diagrams the steps in the design process, which helps identify critical areas and determine if an individual step affects the design process output. In addition, this flow chart may allow the assessor to divide the design assessment between different visits while ensuring overall coverage. CANCELE D CANCELE D DOE G 414.1-1 August 1996 C-3 Cause and Effect Diagrams Cause and Effect Diagrams illustrate the relationship between a known "effect" or outcome and all the "causes" or contributors influencing it. The effect being examined may represent either a wanted or unwanted outcome. The cause and effect diagram is used when the outcome of a process/program is known but the contributors need to be evaluated further. These diagrams are especially useful when those contributors stem from different sources across the organization being assessed. Example In this example, the assessor uses the cause and effect diagram to identify all the program elements that should be in place to prevent worker exposure. This tool can be used in two ways by the assessor: 1) to verify the effectiveness of individual elements, thereby verifying that the program is working; and 2) to look at exposure records and use the diagram to help pinpoint the source of programmatic weaknesses. CANCELE D CANCELE D August 1996 DOE G 414.1-1 C-4 Tree Diagrams Tree Diagrams are used to map out systematically in increasing detail the full range of controls and tasks needed to achieve a primary goal. They can also be used to map out the barriers needed to prevent an unwanted event (called 'causal analysis' or 'barrier analysis' trees). Tree diagrams are used by the assessor to verify whether all planned activities are in place to support an overall program's objective. They are especially useful for helping the assessor focus on the big picture; i.e., the overall goal of the program, with its supporting subgoals. Example

In this example, the tree diagram provides a map of the elements needed to support an effective training and qualification program. Using the diagram, the assessor can plan and focus the assessment to ensure that the appropriate activities are being performed and to evaluate the training organization's effectiveness overall. As this tree diagram is used for the assessment, the elements must be continually "rolled-up." This means the Capability/ Proficiency Verified element should be assessed to determine its impact on Qualification, which must be assessed to ensure it supports overall Training and Qualification. CANCELE D CANCELE D DOE G 414.1-1 August 1996 C-5 Checklists Checklists are listings of assessment objectives and performance criteria. They usually include a column for the requirements (or a reference to the requirements) and a column for recording assessment observations/evidence. Checklists are especially useful for organizing assessment time by providing a means to list appraisal objectives sequentially. They may also be structured in a form that can easily be converted into assessment report text. Example Lab/Appraisal # Date: Page 1 of Results Reference Criteria Sat Unsat Comments NL-QAM 1. Is monitoring and data collection equipment calibration? NL-QAM 2. Is equipment calibration traceable to nationally recognized standards? NL-QAP- 5.1 3. Is equipment calibration performed using approved instructions? NL-QAP- 5.1 4. Are calibration records maintained for each piece of equipment? NL-QAP- 5.1 5. Is a use log maintained? In this example, the checklist is used to list the primary elements of a laboratory's calibration program. The basis or source of the criteria element is included in the first column to provide a path back to the requirements document(s). The "comments" column provides a place for the assessor to record additional observations as they are discovered during the assessment, which helps to ensure that important data is not lost. CANCELE D CANCELE D August 1996 DOE G 414.1-1 C-6 Information Systems Information systems comprise a wide range of different forms and formats. In their simplest form they may include the weekly and monthly reports of laboratory or organizational performance which are used to alert the assessment organization of potential assessment areas. In more complicated forms, these systems may include computerized databases which link performance to specific performance objectives or track actions to resolve programmatic weaknesses. In either case, information systems are an important tool for the assessor by providing them the data needed to focus assessment activities. Example Lost-Time Accident Monthly Summary Date Type Area Work Procedure Work Crew Days Lost 5/3 Sprain Bldg 12 CAP-101 Mech 4 5/5 Sprain Bldg 5 MAP-2-12 Elec 5 5/12 Burn Area 8 PMP-1-4 Mech 2 5/15 Abrasion Area 10 PMP-3-7 Grnds 3 5/23 Burn Bldg 12 CAP-103 Elec 1 5/25 Sprain Admin Bldg N/A N/A 1 5/29 Cut Bldg 5 MAP-2-17 Elec 1 In the above example, information on lost-time injuries is displayed in both tabular and graphical form. This information can be used to focus the assessment on either the location of the injuries or the work procedures involved to identify any weaknesses in the accident prevention program. CANCELE D CANCELE D DOE G 414.1-1 August 1996 D-1

APPENDIX D - Independent Assessment Planning There are many scheduling and planning approaches to satisfying requirements for integration, grading, and use of a risk-based approach. The following steps provide one approach to detailed planning for independent assessments. This detailed level of planning should be performed on a regular periodic basis, such as annually. • Identify Assessment Areas Assessments should be managed to achieve specific objectives. Programs, systems, and processes (including functional and organizational areas) and their associated performance objectives and measurements should be reviewed to determine their appropriateness and applicability. After the determination has been made, information as discussed in the following sections should be compiled and maintained to manage the assessments and their objectives. • Evaluate the Consequences and Probability (Risk) Evaluate the program, systems, and processes to identify the various factors and elements that could result in a failure to achieve the mission or the objective's success. Factors to consider include (but are not limited to): - Worker health and safety (injuries, deaths, exposures); - Public health and safety (injuries, deaths, exposures); - Community concerns; - Regulatory non-compliance and liability; - Business efficiency/productivity; - Complexity of the involved processes; - Reliability of the engineering and administrative controls; - Skills and experience of the personnel involved; - Maturity of the program, system, or process (developed mature techniques/processes versus state-of-the-art or developmental/pilot technology); - Changes that may affect performance (including regulatory); - Life cycle phase (new, mid-life, of close out of activity); - Organizational experience with the program, system, or activity; - Economic costs (uncertainty); - Schedule/commitment, or milestones failure; - Performance measures/indicators, trending downward; and - Likelihood of discovery by other means (e.g. external assessments, management assessments, startup readiness reviews, investigations). Each of the consequences should be evaluated for severity and likelihood of occurrence (this is a rough estimate, using professional judgement). The use of a multi-disciplined team for this process is encouraged. CANCELE D CANCELE D August 1996 DOE G 414.1-1 D-2 • Prioritize the Assessment Activities Rank the programs, systems, and processes risks based on the consequences and likelihood. Those with high risk or having multiple lower risks should be ranked higher. This process may identify specific areas that cross-cut several programs, activities or organizations, allowing a single assessment to address a risk and benefit several organizations. • Identify Areas for Further Evaluation List the programs, systems, or processes and the areas of risk, then further evaluate other factors, such as: - Time since the last internal independent assessment; - Time since other assessments (external, management, process improvement teams, investigations, etc.); - Opportunities to perform the assessment in conjunction with other organizations (internal or external); - Work schedules (will a lower-ranked program start or complete before a higher ranked program or activity); - Other scheduled assessments (management assessments, process improvement teams, etc. that would be expected to address the area); - Availability of assessment personnel, including technical personnel to

perform the assessment; and - Certifications, registrations, or other scheduled activities that would be expected to evaluate the program, system, or process. • Initiate the Assessment Plan Develop an assessment plan for the specific assessment including the following elements: - Program, system, process, organization, and/or activity to be assessed (the scope of the assessment); - Objectives of the assessment; - Assessment drivers (regulatory, contractual agreements, performance objectives, internal procedures, etc. that will be used); - Assessment team members, including the lead, supporting assessors, and the technical experts (if appropriate); and - Assessment schedule with start and end dates (final planning, notification, kickoff meetings, preparation, investigation, closeout meeting, and report issuance). The assessment plan should establish the depth, scope, and breadth of the assessment. It will provide a tool for scheduling and information exchange for both the assessment team and the management of the assessed organizations. CANCELE D CANCELE D DOE G 414.1-1 August 1996 D-3 • Allocation of Resources The resources for performing assessments are limited, and seldom allow for performing all assessments. Likewise, the benefits of performing assessments on low risk areas is marginal. As a result, a realistic estimate of the resources available, including their scheduled availability should be developed. Additional factors such as the availability of personnel independent of the areas to be assessed, budget constraints, management or customer requests, and response/follow-up to previous external assessments should be considered. • Assessment Schedule and Plan Using the information developed in the prior steps, an assessment schedule and plan can be developed. The schedule and plan should reflect the areas of greatest risk and the reasonable allocation of resources. Assessments that fall "below the line" should be retained as "targets of opportunity" to be performed if resources become available, or if one of the planned assessments changes in risk or schedule. The assessment schedule and plan is a tool that allows management and the customers to understand the basis for the assessments and justifies the allocated resources. - Maintenance of the Assessment Schedule and Plan Assessment planning should be periodically reviewed and modified as new information on the facility or organization is obtained that changes the estimated risks, or reflects changes in available resources. This review should occur at regular intervals, such as monthly or quarterly. The review can be used to finalize schedules, team members, etc. for the next period. Assessment areas that have increased in risk can be moved up in the schedule, while others can be moved down. In some cases, assessments that were "below the line" should be moved up to reflect changes that have occurred since the original planning and ranking was performed. The results of assessments which identify good performance should be used to reduce the frequency and depth of future assessments. Areas of poor performance should receive increased attention especially if there are indications that management is unable to correct identified problems. This is because recurring and cumulative deficiencies, even in a low hazard operation, may decrease the likelihood of achieving its mission. - Frequency Given the likelihood of some or all of these factors being present and the wide variety of DOE activities, it is impossible to define the "right" assessment frequency in this Guide. Too few assessments will not keep pace with the changes occurring in the program, system, or process. Too many assessments will distract organizational resources from focused attention on the safe conduct of work and mission accomplishment. Therefore, it is ultimately the responsibility of management, guided by assessment professionals, to determine the appropriate CANCELE D CANCELE D August 1996 DOE G 414.1-1 D-4 mix of assessments (independent, management, and worker) that will meet customer requirements and help assure mission success. CANCELE D CANCELE D