![Page 1: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/1.jpg)
Agenda
Moderator: Betsy Grim, Director Quality Assurance, OPP
Part 1: Quality Basics
1:30 – 2:30 Louis Blume, GLNPO QAM
Quality Basics and Stages of Quality
2:30 – 2:45 Kevin Bolger, Region 5 QAM
Regulations Requiring Quality
2:45 – 3:00 Marcia Damato, GLNPO QA Team
Quality Implementation for grants
3:00 – 3:30 Break
![Page 2: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/2.jpg)
Agenda
Moderator: Betsy Grim, Director Quality Assurance, OPP
Part 2: Integrating and Documenting Quality in Project Management
Tom Huges, QAO, NHEERL and Cheryl Itkin, QAO, NCEA
3:30 – 3:50 Overview of “other” data
3:50 – 4:10 Documenting “other” data
4:10 – 4:20 Break
4:20 – 5:00 Hands on exercise
![Page 3: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/3.jpg)
Agenda
Moderator: Betsy Grim, Director Quality Assurance, OPP
3:30 – 4:30 Louis Blume, GLNPO QAM Quality Basics
2:30 – 2:45 Kevin Bolger, Region 5 QAM Regulations Requiring Quality
2:45 – 3:00 Marcia Damato, GLNPO QA Team Quality Implementation for grants
3:00 – 3:30 Break
![Page 4: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/4.jpg)
Quality Management Basics Quality Management Basics and Advanced Stages of and Advanced Stages of
Quality SystemsQuality Systems
Louis Blume, U.S. EPA, Louis Blume, U.S. EPA,
Great Lakes National Program Office, Great Lakes National Program Office, Project Level QA Training for EPA QA ManagersProject Level QA Training for EPA QA Managers
May 14, 2009May 14, 2009
San Antonio, TexasSan Antonio, Texas
![Page 5: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/5.jpg)
Louis Blume’s QA BackgroundLouis Blume’s QA Background
• QA Manager, US EPA – GLNPO, 1995 – present
• Chief of Planning, US EPA, Region 5, 1990 – 1992
• ORD Regional Scientist, US EPA, Chicago, IL, 1989 – 1990
• Soil Scientist, US EPA Office of Research and Development, Las Vegas, Nevada. 1985-1989
Relevant AccomplishmentsRelevant Accomplishments• Co-chaired EPA’s National Workshop on Graded Approaches for QA
• Presented QA Symposia in Canada and Russia
• QA Manager: National Acid Deposition Soil Survey, Direct/Delayed Response Project, US EPA
• M.S. Soil Science,Taxonomy / Morphology – University of Georgia
• B.S. Forestry – Michigan Tech University
![Page 6: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/6.jpg)
We are here today to talk about
the advantages of a
Quality System,Versus a Quality Management
Plan.
![Page 7: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/7.jpg)
Benefits of a Quality SystemBenefits of a Quality System
• Provides an effective management process that assures fact based environmental decisions
• Applies Quality principles to non-laboratory oriented data collection activities
– policy making, modeling, pollution prevention, economic assessments
• Allows flexibility on how you implement data collection activities
– can result in significant cost savings!
• Provides your organization a management structure that promotes continuous improvement
![Page 8: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/8.jpg)
EPA Order CIO 2105.0: EPA Order CIO 2105.0: Major PointsMajor Points
• Defines requirements and implementation for quality systems which support U.S. EPA environmental programs directly or through extramural agreements
• Defines and broadens the scope of “environmental data” to include all input information to decision processes.
• Identifies QA roles and responsibilities for agency managers and staff
![Page 9: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/9.jpg)
U.S. EPA Order CIO 2105.0:U.S. EPA Order CIO 2105.0:Policy and Program Requirements for the mandatory Agency-Wide Quality System
• Establishes policy and program requirements for the conduct of QA, for all environmentally-related measurements performed by or on behalf of U.S. EPA
![Page 10: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/10.jpg)
Definition of Environmental DataDefinition of Environmental Data
“Any measurements or information that describe environmental processes, location, or conditions; ecological or health effects and consequences; or the performance of environmental technology. For EPA, environmental data include information collected directly from measurements, produced from models, and compiled from other sources such as data bases or the literature.”
EPA Order 5360.1 A2, May 2000
![Page 11: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/11.jpg)
Quality SystemQuality System
• A structured, documented Management System describing an organization’s Policies, Policies, Objectives, Principles, Authorities, Objectives, Principles, Authorities, Responsibilities, Accountabilities, and Responsibilities, Accountabilities, and Implementation PlanImplementation Plan to ensure Quality in its Work Processes, Products and Services
• Provides an organization’s framework through: Planning, Implementing, Planning, Implementing, Assessing, Improving, DocumentingAssessing, Improving, Documenting
![Page 12: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/12.jpg)
Quality Management Systems Quality Management Systems can/should be based on a can/should be based on a Graded ApproachGraded Approach
Quality system requirements are commensurate with:– Importance of work– Availability of resources– Unique needs of organization– Consequences of potential decision errors
![Page 13: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/13.jpg)
GLNPO’s Graded Approach to
Quality System DocumentationQuality System DocumentationReplaced four-tiered category approach with new project categories:
1. State agencies2. Tribal grants3. Consortium grants4. Cluster grants5. Existing data/modeling6. Habitat/ecosystem restoration7. Ambient monitoring and research demonstration8. Volunteer monitoring9. Sediment assessment10. Pollution prevention and environmental education11. Repeating projects of similar scope
1 to 4: QMP 1 to 4: QMP usually requiredusually required
5 to 11: QAPP 5 to 11: QAPP usually requiredusually required
![Page 14: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/14.jpg)
Quality Review vs. Peer Review Quality Review vs. Peer Review
• Quality Reviews of QAPPs focus on:– the process– the chance for success– technical details and validity of results, from a
reproducibility standpoint
• Peer Reviews are primarily driven by the discipline
![Page 15: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/15.jpg)
Peer Review and Information Peer Review and Information Quality GuidelinesQuality Guidelines
GLNPO believes that an effective quality system ensures
– Implementation of Agency peer review policy (EPA document #100-B-00-001)
– http://www.epa.gov/osa/spc/htm/2peerrev.htm
– Adherence to OMB’s federal Information Quality Guidelines or IQGs (67 FR 8451)
– http://www.epa.gov/quality/informationguidelines/– http://www.whitehouse.gov/omb/inforeg/
agency_info_quality_links.html
![Page 16: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/16.jpg)
What Does OMB Direct Agencies What Does OMB Direct Agencies Do to comply with the IQGs?Do to comply with the IQGs?
• Adopt a basic standard of quality for disseminated information
• Apply the principles of information quality to each step of EPA’s development of information- creation, collection, maintenance &
dissemination
• Enable affected persons to seek and obtain corrections to "Information* disseminated” to the public by filing a Request for Correction (RFC)
*Information, dissemination, distribution, *Information, dissemination, distribution, and influential information are further defined at and influential information are further defined at
http://www.epa.gov/quality/informationguidelines/http://www.epa.gov/quality/informationguidelines/
![Page 17: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/17.jpg)
What is What is QualityQuality??
• Objectivity– Presentation: is accurate, clear, complete
and unbiased (transparency)– Substance: accurate, reliable, and unbiased
..based on sound statistical and research methods
• Utility– Usefulness of the information to its intended
users, including the public• Integrity
– Protection from unauthorized access or revision
![Page 18: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/18.jpg)
Quality Control
QAPP, SOPProject
Quality Assurance
QAPP, SPP (DQO)Project
Quality System
QMP
QMPOrganization/Program
Quality SystemPolicy/Regulations
![Page 19: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/19.jpg)
Specifications for Non-EPA Specifications for Non-EPA OrganizationsOrganizations
• EPA Requirements for Quality Management Plans (QA/R-2) EPA/240/B-01/002 March 2001
• Guidance for Developing Quality Systems for Environmental Programs (QA/G-1)EPA/240/R-02/008 November 2002
• EPA Requirements for QA Project Plans (QA/R5) EPA/240/B-01/003 March 2001
• Guidance for Quality Assurance Project Plans (QA/G-5) EPA/240/R-02/009 December 2002
![Page 20: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/20.jpg)
QMPs vs. QAPPsQMPs vs. QAPPs
• Quality Management Plans reflect activities and policies common to all projects.
• Quality Assurance Project Plans reflect specific projects.
QMPQMP
QAPPQAPP QAPPQAPP
![Page 21: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/21.jpg)
Quality Management Plan (QMP)Quality Management Plan (QMP)
The QMP describes quality requirements and procedures for activities that support the collection, use, and communication of environmental data and information, including:
– Project Planning– Grants, contracts and interagency agreements– Documents and records– Technical Tasks– Information Technology– QA Assessments/Audits and corrective action responses
See EPA Requirements for Quality Management Plans (EPA QA/R-2) http://www.epa.gov/quality/qa-docs/r2-final.pdf
![Page 22: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/22.jpg)
Quality Assurance Project Plans Quality Assurance Project Plans (QAPPs)(QAPPs)
• Mandatory document for any data collection activity
• Describes how data and information will be collected, analyzed and assessed
• Describes the technical details and procedures to be used in the collection of data
See EPA Requirements for Quality Assurance Project Plans (EPA QA/R-5)
http://www.epa.gov/quality/qs-docs/r5-final.pdf
![Page 23: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/23.jpg)
Required Elements in a QMPRequired Elements in a QMP
• Management and Organization• Quality System and Description• Personnel Qualifications and Training• Procurement of Items and Services• Documentation and Records• Computer Hardware and Software• Planning• Implementation of Work Processes• Assessment and Response• Quality Improvement
![Page 24: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/24.jpg)
Required Elements in a QAPPRequired Elements in a QAPP
Guidance Assurance for Project Plans, EPA QA/G-5•Project Management
– Elements A1-A9
•Measurement/Data Acquisition– Elements B1-B10
•Assessment/Oversight– Elements C1-C2
•Data Validation and Usability– Elements D1-D3
![Page 25: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/25.jpg)
Elements Requirements
PROJECT MANAGEMENT
A1 Title and Approval Sheet Title and approval sheet.
A2 Table of Contents Document control format.
A3 Distribution List Distribution list for the QAPP revisions and final guidance.
A4 Project/Task Organization Identify individuals or organizations participating in the project and discuss their roles, responsibilities and organization.
A5 Problem Definition/ Background 1) State the specific problem to be solved or the decision to be made. 2) Identify the decision maker and the principal customer for the results.
A6 Project/Task Description 1) Hypothesis test, 2) expected measurements, 3) ARARs or other appropriate standards, 4) assessment tools (technical audits), 5) work schedule and required reports.
A7 Data Quality Objectives for Measurement Data
Decision(s), population parameter of interest, action level, summary statistics and acceptable limits on decision errors. Also, scope of the project (domain or geographical locale).
A8 Special Training Requirements/Certification
Identify special training that personnel will need.
A9 Documentation and Record Itemize the information and records that must be included in a data report package, including report format and requirements for storage, etc.
MEASUREMENT/DATA ACQUISITION
B1 Sampling Process Designs (Experimental Design)
Outline the experimental design, including sampling design and rationale, sampling frequencies, matrices, and measurement parameter of interest.
B2 Sampling Methods Requirements Sample collection method and approach.
B3 Sample Handling and Custody Requirements
Describe the provisions for sample labeling, shipment, chain-of-custody forms, procedures for transferring and maintaining custody of samples.
B4 Analytical Methods Requirements Identify analytical method(s) and equipment for the study, including method performance requirements.
B5 Quality Control Requirements Describe routine (real-time) QC procedures that should be associated with each sampling and measurement technique. List required QC checks and corrective action procedures.
Required Elements in a QAPPRequired Elements in a QAPP
![Page 26: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/26.jpg)
Elements Requirements
MEASUREMENT/DATA ACQUISITION (con’t)
B6 Instrument/Equipment Testing Inspection and Maintenance Requirements
Discuss how inspection and acceptance testing, including the use of QC samples, must be performed to ensure their intended use as specified by the design.
B7 Instrument Calibration and Frequency Identify tools, gauges and instruments, and other sampling or measurement devices that need calibration. Describe how the calibration should be done.
B8 Inspection/Acceptance Requirements for Supplies and Consumables
Define how and by whom the sampling supplies and other consumables will be accepted for use in the project.
B9 Data Acquisition Requirements (Non-direct Measurements)
Define the criteria for the use of non- measurement data such as data that come from databases or literature.
B10 Data Management Outline the data management scheme including the path and storage of the data and the data record-keeping system. Identify all data handling equipment and procedures that will be used to process, compile, and analyze the data.
ASSESSMENT/OVERSIGHT
C1 Assessments and Response Actions Describe the assessment activities needed for this project. These may include DQA, PE, TSA, MSR/ PR/RR
C2 Reports to Management Identify the frequency, content and distribution of reports issued to keep management informed.
DATA VALIDATION AND USABILITY
D1 Data Review, Validation, and Verification Requirements
State the criteria used to accept or reject the data based on quality.
D2 Validation and Verification Methods Describe the process to be used for validating and verifying data, including the chain-of-custody for data throughout the lifetime of the project.
D3 Reconciliation With Data Quality Objectives
Describe how results will be evaluated to determine if DQOs have been satisfied.
GLNPO Added Note: The above elements are minimum requirements for a QAPP, and therefore, should be used during the QAPP development.
Required Elements in a QAPPRequired Elements in a QAPP
![Page 27: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/27.jpg)
Comparison of Required Elements Comparison of Required Elements in a QAPP versus a QMPin a QAPP versus a QMP
Elements of a Quality Assurance Project Plan (QAPP)
Elements of a Quality Management Plan (QMP)
Group A: Project Management ElementsGroup B: Data Generation and Acquisition ElementsGroup C: Assessment and Oversight ElementsGroup D: Data Validation and Usability Elements
1. Management and Organization2. Quality System and Description3. Personnel Qualifications and Training4. Procurement of Items and Services5. Documentation and Records6. Computer Hardware and Software7. Planning8. Implementation of Work Processes9. Assessment and Response10. Quality Improvement
![Page 28: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/28.jpg)
EPA’s Quality SystemEPA’s Quality System
EPA Organizations• EPA Order 5360.1
• EPA Order 5360 (Quality Manual)
• Contracts Management Manual (EPA Order 1900)
• ANSI/ASQC E4-1994
Non-EPA Organizations• 48 CFR 46 (FAR 46) for contractors
• 40 CFR 30, 31, and 35 for assistance agreement recipients
• ANSI/ASQC E4-1994
![Page 29: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/29.jpg)
Authorizing Requirements Authorizing Requirements
• ANSI/ASQC E4-1994/2004 is a national consensus standard designed specifically for quality systems applied to environmental data collection and environmental technology programs.
• Relates directly to ISO 9000 certification and over 14000 approved organizations
![Page 30: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/30.jpg)
Quality System Requirements for Quality System Requirements for Extramural AgreementsExtramural Agreements
• Conformance to ANSI/ASQC E4-1994/2004 requires:
Documentation must be sufficient to describe the quality system
![Page 31: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/31.jpg)
• ANSI/ASQC E4, Specifications and Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology Programs
– Provides the basis for planning, implementing, documenting, and assessing a quality system
– Standard divided into 3 parts
Part A: Management Systems Part B: Environmental Data Part C: Environmental Technology
– Standard available from ASQ (www.asq.org)
American National Consensus American National Consensus StandardStandard
![Page 32: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/32.jpg)
• Conform with ANSI/ASQC E4-1994/2004
• Document the organization's quality system (QMP) which should be approved prior to initiating environmental work
• Document the application of QA and QC activities to activity-specific efforts (QAPP) which should be approved prior to initiating environmental data collection
• Verify that products/services meet specifications
Basic Quality System Requirements Basic Quality System Requirements for Solicitations and Contractsfor Solicitations and Contracts
![Page 33: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/33.jpg)
Is it Acquisition or Assistance?Is it Acquisition or Assistance?
Acquisition vs. Assistance EPA Order 5700.1
The basic difference between acquisition and assistance is the relationship between the parties.
• If an activity is funded to meet an internal EPA need, e.g., prepare a specific report to Congress or purchase furnishings for EPA staff, then an acquisition (contractual) relationship exists and EPA's direct procurement requirements (e.g., the Federal Acquisition Regulations) apply.
• If the activity is funded to primarily benefit someone other than the Federal government
![Page 34: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/34.jpg)
QA Review Form:
• Required for all solicitations and contracts, work assignments, delivery orders, task orders, and for modifications that involve a significant change to the Statement of Work.
• Documents the necessary quality-related activities to be performed by the contractor.
• May document the quality-related activities to be performed by EPA.
• Standardizes how EPA Contracting Officer's Representatives and QA Managers communicate quality specifications to EPA Contracting Officers to ensure inclusion of the specifications in standard contract clauses.
Contract-Specific Quality System Contract-Specific Quality System ProceduresProcedures
![Page 35: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/35.jpg)
Quality Requirements for Solicitations Quality Requirements for Solicitations and Contracts and Contracts
Objectives:• Identify Authorizing Requirements - where
to find specifics and updates
• Describe requirements and responsibilities that ensure QA in Contracts
![Page 36: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/36.jpg)
Federal GovernmentFederal Government
Federal Acquisition Requirements (FAR)• Regulation - FAR 46.202-4 (February 1999)
– Applies to all Federal Agencies– Allows Federal Agencies to:
• Select a higher-level quality standard and• Tailor this standard to meet Agency-specific needs
![Page 37: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/37.jpg)
• The type and content of the contract quality requirements depends on the nature of the work. It can range from inspection at the time of acceptance to requiring that the contractor implement a comprehensive program (quality system) for controlling quality.
• Contract quality requirements fall into four general categories, depending on the extent of quality assurance needed by the Government.
Contract Quality RequirementsContract Quality Requirements
![Page 38: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/38.jpg)
Higher-level contract quality requirements. For complex or critical items or when the technical requirements of the contract require --
(1) Control of such things as work operations,in-process controls, and inspection; or
(2) Attention to such factors as organization,planning, work instructions, documentation control,and advanced metrology.
(FAR 46.202-4)
Contract Quality RequirementsContract Quality Requirements
![Page 39: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/39.jpg)
• EPA's higher-level quality standard is usually American National Standard ANSI/ASQC E4-1994, "Specifications and Guidelines for Quality Systems for Environmental Data Collection and Environmental Technology Programs."
• Chapters 7 and 46 of EPA's Contracts Management Manual (CMM) contains EPA-specific policy and procedures:– Contract Clause– Step-by-Step Directions for COs and CORs– QA Review Form
EPA’s Adoption of FAR 46.202-4EPA’s Adoption of FAR 46.202-4
![Page 40: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/40.jpg)
Not based on cost but on the type of work to be performed
Covered activities are the same as the activities covered by the EPA Quality System
Applies to simplified acquisitions except those under a purchase order (which should not be used for covered activities)
Application of Contract RequirementsApplication of Contract Requirements
![Page 41: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/41.jpg)
The QA Review FormThe QA Review Form
The QARF provides information: • From the Contracting Officer's Representative to inform the
Contracting Officer what quality requirements should be included in a solicitation
• Enhance QA Manager oversight of the quality-related activities
QARF must be completed for: • Solicitations
• Contract Statements of Work (SOW)
• SOW for work assignments/delivery orders/task orders
• Modifications to existing work assignments/delivery orders/task orders that change the SOW
• Technical directives and actions that do not affect the work performed by the contractor (for example, incremental funding or time extensions) do not require a QARF.
![Page 42: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/42.jpg)
Complete the QARFComplete the QARF
• Check "Yes" and complete the rest of the QARF if project involves: – collection, generation, use, and/or reporting of environmental
data;– design, construction, and/or operation of environmental
technologies;– development and/or use of models; or – other activities that need QA or QC requirements as
identified in your organization's QMP.
• If the work does not involve any of these activities, check "No," skip Quality Related Requirements and complete Signatures.
• Estimate the percentage of costs or level-of-effort allocated to the activities identified.
![Page 43: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/43.jpg)
Stages of Quality System Stages of Quality System ImplementationImplementation
• Quality programs are not implemented with the stroke of a pen upon the approval of a Quality Management Plan
• Functional quality programs do not just happen - they evolve, typically after QMP approval
![Page 44: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/44.jpg)
Value of Defining Stages Value of Defining Stages of Implementationof Implementation
• Provides a metric to measure success• Emphasizes the fact that good programs take
time and continuously improve• Illustrates a quality continuum• Establishes realistic expectations
1 2 3 4
Quality InfrastructureQuality Infrastructure
![Page 45: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/45.jpg)
Stages of Quality System Stages of Quality System Implementation MatrixImplementation Matrix
• Columns = Degree of Implementation– Stage 1: 0-25%– Stage 2: 25-50%– Stage 3: 50-75%– Stage 4: 75-100%
• Rows = Characteristics, Actions, Attitudes, and Keys to Success
See handout orhttp://www.epa.gov/glnpo/quality/stagesofquality
![Page 46: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/46.jpg)
Stages of QualityStages of Quality
11 22 3 3 4 4 BargainingBargaining True Value Added True Value Added
DepressionDepression Cost Savings Cost Savings
Baby StepsBaby Steps
Denial Denial Reluctance Compliance Reluctance Compliance Nirvana Nirvana**
TrainingTraining
** Ideal condition of perfect harmony and peace Ideal condition of perfect harmony and peace
1 – 5 years1 – 5 years
AcceptanceAcceptance
![Page 47: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/47.jpg)
Kubler-RossKubler-Ross11
Stages of grief:
• Denial (No lab analysis in my project!)• Anger (I can not afford to write a QA plan, you write
the plan for me)• Bargaining (Not much guidance! They will publish the
report!, They have a QMP!, They did a QA plan a year ago, they’re world renowned!)
• Depression (I don’t care anymore)• Acceptance (Okay I see the value let’s get together
and talk about the key parts!)
1. Kubler-Ross, Elisabeth. 1969. On Death and Dying. ISBN 0-684-83938-5, Reprinted 1997, Simon & Schuster, Inc.
![Page 48: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/48.jpg)
Stage 1: 0-25%Stage 1: 0-25%Characteristics • External pressure forces development of QS
• One person appointed to QA
• No formal infrastructure for training, review, assessment and inventory
Actions • Appoint additional QA personnel, through management edict, try to harness enthusiastic people showing initiative
• Artful Dodgers (Hide from QM)
• Argue that project is not technical or no data, no sampling
Attitudes • Management views quality as outside their primary focus
• Minimal understanding throughout organization, seen as an insurance policy
• Staff have narrow view of when quality is needed
Keys to Success • Develop generic QMP (not too prescriptive)
• Encourage broader ownership across the office
• Try to document existing processes that relate to Quality (e.g., workload planning, expenditures)
• Avoid using top down logic for selling Quality, vs explanation of the benefits
![Page 49: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/49.jpg)
Stage 2: 25-50%Stage 2: 25-50%Characteristics • QMP approved
• QAPPs developed for some projects
• Polarization of QM and POs
Actions • QA staff identify delinquencies & try to fill gaps
• Training initiated, typically introductory
• Good opportunity for external MSRs
• Develop inventory of projects/expenditures
• Emphasize value of QA
Attitudes • Most see QA as bureaucratic exercise
• Difficult exchanges between QA staff & POs
• Problem: “How will my QM fix this?”
Keys to Success • Management takes ownership
• Develop inventory, capture quality during award phase, build rapport with grants, contract staff
• Build on positive behavior & ignore nay-sayers
![Page 50: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/50.jpg)
QA = Quality AssistanceQA = Quality AssistanceFor For Enterprise AssuranceEnterprise Assurance
We succeed as a team, or we We succeed as a team, or we fail as a team.fail as a team.
![Page 51: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/51.jpg)
Stage 3: 50-75%Stage 3: 50-75%
Characteristics • QMP approved and partially implemented
• QMs involved in management meetings
• Management begins to ask QA questions
Actions • POs employ systematic planning for all projects
• QA staff involved in project planning
• Inventory of projects 100% implemented
Attitudes • Most staff believe QA provides value
• QM feels like part of the team and not tattle-tale
• Problem: “How will we fix this?”
• Management becomes enlightened by Quality status (answers to questions)
Keys to Success • QA staff must stay involved at project-level
• Recognize and reward QA successes
• Orient limited QA $ to high priorities
![Page 52: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/52.jpg)
Stage 4: 75-100%Stage 4: 75-100%Characteristics • Quality system is comprehensive
• QA is a component of daily activities for all staff
• Peer review & info quality key parts of QS
• Managers are actively involved and well-trained
• Office is perceived positively by external clients
Actions • Use QA training & experience in hiring criteria
• Staff use “we” terms instead of “you” terms
• Continually re-evaluate, QM provides data assessments that relate to office-wide goals
Attitudes • Staff seek out QA personnel for assistance
• Staff are empowered to improve quality
• Staff reveal QA concerns - know they’ll be heard
Keys to Success • QM integral part of project development
• PO seen as enforcer and not QM
• Hire people with positive QA attitudes
• Quality system relates to organizational goals
![Page 53: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/53.jpg)
ManagementManagement Questions Questions
Why?Why?
When a manager asks about the When a manager asks about the state of quality in their organization, state of quality in their organization, they emphasize its importance and they emphasize its importance and drive the improvement.drive the improvement.
![Page 54: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/54.jpg)
What holds managers back?What holds managers back?
• Fear of additional resource demands• Narrow view of quality (e.g. focus on lab
data), not seen as their function• Not realizing the management tools
associated with the quality process• They do not have battle scars from poor
quality
![Page 55: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/55.jpg)
Stage 2 Questions Stage 2 Questions
• How many active projects do we support?
• What percent collect environmental information?
• How many of these projects have approved quality documentation?
Focus on InventoryFocus on Inventory
![Page 56: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/56.jpg)
Stage 3 QuestionsStage 3 Questions
• How many projects have been assessed to evaluate key quality concerns and quality implementation?
• Are we focusing quality resources on the most important office decisions?
• Are we prioritizing resources to areas of greatest uncertainty?
• Is this uncertainty relevant to the decision to be made?
Focus on ImplementationFocus on Implementation
![Page 57: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/57.jpg)
Stage 4 QuestionsStage 4 Questions
• Have true environmental outcomes been addressed?
• Have we discussed how these quality issues affect the decision?
• Is the final product disseminated, consistent with IQG and Peer Review (reproducible)?
• Have we discussed recommendations for improvement?
Focus on ReflectionFocus on Reflection
![Page 58: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/58.jpg)
Advanced Stages of Quality SystemsAdvanced Stages of Quality Systems
• Are we there yet?
• If not what can we do to move forward?
![Page 59: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/59.jpg)
Advanced Stages of Quality SystemsAdvanced Stages of Quality Systems
• Align quality systems with management priorities– Government Performance Results Act
(GPRA)– Fulfilling the mission– Information Quality Guidelines
![Page 60: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/60.jpg)
Advanced Stages of Quality SystemsAdvanced Stages of Quality Systems
• Align key program functions toward GPRA goals– In aligning and coordinating different components must have
awareness of how pieces fit together – This awareness will come from the new policy order
• Use the quality system (i.e., the quality framework) to align and coordinate towards the same goals (GPRA goals)
• Educate and involve all components of the office– Define how quality is part of day-to-day activities– Show value added aspects of the quality system– Management, budget office, funding office, clerical,
administrative, and information management staff all need to be onboard
![Page 61: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/61.jpg)
Advanced Stages of Quality SystemsAdvanced Stages of Quality Systems
• Focus on Mission Critical Functions (Enterprise Assurance™)– Where can an effective quality
management system have the greatest impact within the office?
– Don’t sweat the small stuff (unless you have a few low hanging apples)
![Page 62: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/62.jpg)
Advanced Stages of Quality Advanced Stages of Quality Systems: Quality FrameworkSystems: Quality Framework
• All mission critical programs may not need a formal QMP per EPA/QAR2 if a quality process exists that addresses common quality practices for various standards:– Personnel Qualifications and Training,– Assessment and Response,– Documentation and Records,– Systematic Planning, etc.,– Procurement of Key Products and Services, – Continuous Improvement, etc.
![Page 63: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/63.jpg)
Advanced Stages of Quality SystemsAdvanced Stages of Quality Systems
• This focus on inventory must be addressed at all levels of influence– Headquarters must ask Regions, who must ask
States/Tribes, who must ask NGOs or local entities
• What is the status of quality (i.e., what is baseline or inventory)? Of the ___$ we provide what % is covered by sufficient quality?
• How is quality implemented?• Provides basis for consistency across programs
![Page 64: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/64.jpg)
Advanced Stages of Quality SystemsAdvanced Stages of Quality Systems
• Discuss quality issues as part of National Funding Guidance
• Discuss quality implementation and infrastructure when Regional Program Managers conduct program reviews with States and Tribes
Follow the money!
![Page 65: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/65.jpg)
Advanced Stages of Quality SystemsAdvanced Stages of Quality Systems
• For grants, interagency agreements, and contracts– We need to address how these extramural
activities meet Performance Goals as defined in our Agency’s Strategic Plan (i.e., define desired outcome, as opposed to a scope that tells them how to implement quality)
– Delegated programs that implement successful quality systems improve overall effectiveness and ease the burden of the overseer
![Page 66: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/66.jpg)
Advanced Stages of Quality SystemsAdvanced Stages of Quality Systems
• The environmental community needs to be able to answer baseline/inventory questions, before we can prepare a quality implementation scorecard– How many active projects are there that collect environmental
information?– How many of them have approved quality documentation?– How many have evaluated the quality of the final decisions?
• If we cannot answer these questions for all projects, we should at least answer the questions that relate to our Strategic Plan/ GPRA goals
![Page 67: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/67.jpg)
Next Steps?????Next Steps?????
• Help our managers to answer baseline quality questions. Utilize the annual QAARWP?
• Utilize the baseline data to evaluate if our quality tools are contributing effectively to Agency goals
• Identify the gaps and orient our limited quality resources
![Page 68: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/68.jpg)
Next Steps????? Next Steps?????
• Help our managers understand the quality framework and reduce confusion how various new initiatives fit together to achieve our mission
• Do whatever we can to empower stakeholders with quality tools processes and resources
![Page 69: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/69.jpg)
Key Points in any Quality System*Key Points in any Quality System*
1. The quality system is not optional.
2. All staff in the office should have a responsibility for the quality of their work and of the organization, and have identifiable roles in the quality system.
*acquired from OW’s Ten Commandments for QA (2004 QMP)
![Page 70: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/70.jpg)
Key Points in any Quality System*Key Points in any Quality System*
3. There is an individual identified within each organizational unit in the Office whose quality system activities are independent of the line management structure.
4. Quality is a critical responsibility of all levels of management within the Office and all management personnel have identifiable roles in the quality system. Managers are responsible for ensuring the allocation Managers are responsible for ensuring the allocation of funding for quality management activities, including of funding for quality management activities, including intramural, extramural, and travel funds, as well as intramural, extramural, and travel funds, as well as funding for personnel and quality-related training.funding for personnel and quality-related training.
![Page 71: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/71.jpg)
Key Points in any Quality System*Key Points in any Quality System*
5. Quality can only be achieved through systematic planning, assessment, and corrective action.
6. The importance of the project, the risk of a decision error, the schedule for completion, and the available resources are used to establish the level of quality management applied to a given activity. These considerations must be addressed and documented during the planning phase of the activity.
![Page 72: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/72.jpg)
Key Points in any Quality System*Key Points in any Quality System*
7. The quality of any environmental data or information used by the Office must be assessed (known) and documented, regardless of the source.
8. All environmental decisions made by the Office must be evaluated relative to the quality of the underlying data and information and these evaluations must be documented.
![Page 73: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/73.jpg)
Suggestions for Enhancement of Suggestions for Enhancement of Our Quality ProgramsOur Quality Programs
• Provide quality support to states through annual funding guidance. State funding is much more dynamic and politically driven than federal funding
• Encourage Regions/States to negotiate Quality System resources through the Environmental Performance Partnership Agreement (ENPPA)
![Page 74: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/74.jpg)
Suggestions for Enhancement of Suggestions for Enhancement of Our Quality ProgramsOur Quality Programs
• Discuss Implementation of our Quality System at Regional/State/Tribal levels alleviating Regional ambiguity and inflexibility of quality process
• Allow delegated programs to use the Quality System to provide more efficient utilization of limited resources
![Page 75: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/75.jpg)
Suggestions for Enhancement of Suggestions for Enhancement of Our Quality ProgramsOur Quality Programs
• Incorporate Peer Review and IQG programs in revisions to the QMP
• Discuss process for Regional and State/Tribal implementation of goals and quality issues in revision to the QMP
• Provide opportunities for managers to take ownership of quality system
![Page 76: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/76.jpg)
“A true functional value-added quality system is not driven by approved
documentation, but more so, by the activities implemented on a daily basis
that enhance the quality of the environmental decision.”
Great Lakes National Program Office’s QMP: http://www.epa.gov/glnpo/qmp/index.html
ConclusionConclusion
![Page 77: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/77.jpg)
Please send all comments and questions to:Louis Blume, GLNPO QM 312-353-2317; [email protected]
Great Lakes National Program Office’s QMP: http://www.epa.gov/glnpo/qmp/index.html
Soaring Towards QualitySoaring Towards Quality
![Page 78: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/78.jpg)
CONGRATULATIONS!CONGRATULATIONS!
You have successfully completed the You have successfully completed the EPA Quality in Contracts and Grants TrainingEPA Quality in Contracts and Grants Training
Please complete the course review form and be sure to mention potential “Toolbox” items
If you would like to know more about the EPA Quality System or have any questions, please contact:
• Quality Assurance Manager, or Quality System Coordinatorhttp://intranet.epa.gov/quality/documents/qa-community.doc
• Quality staff: [email protected] or 202-564-6830
• Visit the EPA QA Web site at:http://www.epa.gov/quality
![Page 79: Agenda Moderator: Betsy Grim, Director Quality Assurance, OPP](https://reader033.vdocuments.us/reader033/viewer/2022061218/54b62b294a7959fb188b4606/html5/thumbnails/79.jpg)
ResourcesResources
EPA Quality Requirements for Solicitations and Contractsintranet.epa.gov/quality/contracts.html
EPA Quality Systemwww.epa.gov/quality
Federal Acquisition Regulationswww.arnet.gov/far/(Note Part 46 is "Quality Assurance")
EPA's Acquisition Policy Informationwww.epa.gov/oam/ptod/
FAI COR Mentor Courseintranet.epa.gov/oamintra/trainingwww.faionline.com
ELECTRONIC COPIES of the QARF in Adobe Acrobat, WordPerfect, and Word at:http://intranet.epa.gov/quality/contracts/ctools.html#qarf
Your organizations QA Manager http://intranet.epa.gov/quality/documents/qa-community.doc