You are here

Quality Assurance

QUALITY ASSURANCE / QUALITY CONTROL PROCEDURES FOR HEI STUDIES

PART 1. GENERAL QUALITY ASSURANCE / QUALITY CONTROL PROCEDURES


1.1. POLICY STATEMENT

The mission of the Health Effects Institute (HEI) is to provide high-quality, impartial, relevant scientific information on the health effects of pollutants from motor vehicles and other sources in the environment. All HEI studies are expected to have adequate Quality Assurance /Quality Control (QA/QC) procedures in place. Adequate QA/QC procedures ensure that the data are collected under defined conditions as specified in relevant Standard Operating Procedures (SOPs) and data protocols, are reliable and traceable, and the analyses appropriate and reproducible. The QA/QC procedures provided apply to all HEI studies (described in Part 1). For studies that involve human subjects and some other studies with a high potential for use in regulatory decisions, HEI has additional requirements (described in Part 2).  This includes an external audit by an HEI selected audit team.

The HEI QA/QC procedures listed here are provided to all funded investigators. In addition, HEI’s QA/QC procedures are included in all RFAs published by HEI. More detailed guidance can be found at EPA’s website, for example www.epa.gov/quality. In particular, we refer to EPA’s Guidance for Quality Assurance Project Plans (pdf), EPA QA/G-5. Even though that document is designed for environmental data collection, the sections are broadly applicable to all types of data. For epidemiological data and analyses, investigators are also encouraged to review the guidelines on good epidemiological practice published by the German Society for Epidemiology. Policies regarding data are evolving rapidly and more details will be added to this QA policy as the EPA requirements or community practices change; in the meanwhile, please refer to the HEI policy on data access.


1.2. QUALITY ASSURANCE / QUALITY CONTROL COMPONENTS

A QA Plan should be submitted to HEI at the same time as the Year 1, 5-month progress report (or sooner), following a structured form (www.healtheffects.org/system/files/hei-form-for-qa-plan-sept2019.docx). The QA plan should be comprehensive and in sufficient detail for an independent researcher to replicate and verify the study. The QA plan should be revised and kept up to date as the study progresses.

The QA plan will be reviewed by the HEI Research Committee, HEI staff and the HEI selected audit team, if applicable.  During the course of the study, or the review of the final report, the HEI Research or Review Committee, HEI staff and the HEI selected audit team may require submissions of updated QA plans, and request modifications.  

The Principal Investigator (PI) and his/her institution have the primary responsibility to prepare, update and adhere to the QA plan.  

The QA plan shall include the following items: 

  • Signature field, attesting that the QA/QC plan is updated and signed-off by the PI
  • Brief Description of Study Aims and Design and Data Flow
  • List of Standard Operating Procedures (SOPs) and Data Protocols
  • Quality Control Procedures for Data Collection
  • Data Processing Procedures, Data Linkages and Data Analyses
  • Data and Records Management
  • List of Qualified Personnel

Form for submitting a QA plan to HEI

I. Brief Description of Study Aims and Design and Data Flow. The QA plan should give a brief description of study aims and design and refer to the study protocol, added as an appendix, for details. HEI expects all studies to be conducted according to a written study protocol. A study protocol defines the study objectives, study design and methodologies to be used. The original Project Plan submitted with the HEI application may serve as an initial study protocol. The protocol may be amended as necessary to accommodate changes in study design/approach. The amended version must be a stand-alone document and not refer to the previous version of the protocol.  A version history page should be part of the protocol that shows the version number, the date the new protocol is effective from, a listing of key changes made, and the signature of the PI.  Older versions of the protocol should be retained for traceability.

The QA plan should also contain a flow chart that shows the data flow from raw (i.e. “as collected”) data to the final processed (i.e. “as used in the final statistical analyses”) data. The data collection and processing steps should be expanded in Sections II through V below.

II. List of Standard Operating Procedures (SOPs) and Data Protocols. For HEI studies — whether they generate new data, and/or make use of previously collected data — SOPs and data protocols need to be in place for all critical procedures. SOPs will be used to document routine and repetitive procedures in the laboratory or field for which variability must be minimized. In case SOPs do not exist, they should be developed by individuals knowledgeable of the specific procedures. SOPs will describe what, when, where, how, and why in a stepwise manner. They will be sufficiently complete and detailed to ensure that the data collected are of appropriate quality. 

Data protocols also include standardized procedures for processing and cleanup of the raw data, and other repetitive data steps. Data protocols should also be developed when making use of previously collected data and/or leveraging other ongoing studies, in addition to listing the SOPs used initially.

SOPs and data protocols will be uniquely identified and dated, and updated as needed. Copies of all current SOPs and data protocols should be readily available for reference by the study team, as well as by a third party. All SOPs and data protocols that have been superseded will be maintained in a historical file.

III. Quality Control Procedures for Data Collection. Data collection includes, but is not limited to, field or laboratory measurements, chemical analysis, health data collection, survey, or obtaining previously collected data and/or leveraging other ongoing studies.  For each type, describe quality control checks to be taken, acceptance criteria, and corrective actions. For critical routine procedures refer to the SOPs and data protocols for details.

IV. Data Processing Procedures, Data Linkages and Data Analyses. Data processing includes all manipulations performed on raw (i.e. “as collected”) data; how the data are treated, cleaned, transformed, linked, and statistically analyzed (as shown in the flow chart in Section I). For each step, describe quality control checks to be taken, acceptance criteria, and corrective actions. For critical routine procedures, refer to the SOPs and data protocols for details. This section should also include all code and metadata as well as any software, program or tool development and associated quality checks, if applicable. 

Data analysis plans should be summarized in the QA plan, and refer to the study protocol for details. Modifications to the data analysis plans, specific statistical procedures and statistical code should be added to the QA plan and protocol as the study progresses. In addition, a code book or data dictionary for all the variables in the statistical models should be developed.

V. Data and Records Management.  Describe how you will manage, store, and preserve documentation, records, data, code, and metadata from their generation through the archival process. Identify the individual(s) responsible for these tasks. Discuss methods for managing data versions and tracking. Describe data security, data confidentiality, data access, and data transfer processes. Describe hardware and software to be used. In the case of written records, procedures should include ways to ensure the integrity of the data and/or the documentation such as dating and signing original entries, and documenting, dating and signing changes or edits with the reason for the change.

VI. List of Qualified Personnel. Qualified personnel will need to conduct the proposed research. List the name, position, role in the project and qualifications for key personnel.  Refer to biographical sketches in an appendix for details. Also list training procedures for personnel for the conduct of the study. Also describe plans to cover all of these tasks in case personnel leave the study team.

vintage car broken down with detached wheel

 

PART 2. SPECIAL QA/QC PROCEDURES

HEI uses third-party quality assurance (QA) procedures for studies that involve human subjects and other studies with a high potential for use in regulatory decisions. HEI will inform the PI after approval of the study whether the Special QA procedures will apply to his/her study. HEI reserves the right to conduct a QA audit of an HEI study during the course of the study or the review of the final report.

The special procedures augment the QA/QC procedures applied to all HEI studies (through staff and Committee Oversight) and provide additional assurance that the data are collected under defined conditions as specified in a written protocol and Standard Operating Procedures (SOPs), are reliable and traceable, and the analyses appropriate and reproducible.


2.1 THIRD-PARTY QA OVERSIGHT

HEI will generally engage one or more qualified individuals to serve as Quality Assurance consultant(s) for the project. This individual will report to HEI’s Director of Science and be responsible for overseeing the implementation of HEI’s special QA/QC procedures.

Typically, the HEI special QA oversight entails two audits by the QA auditors, one during the course of the study and one of the data presented by the investigators in the HEI Final Report. Generally, the QA audit during the course of the study is conducted on-site and the QA audit of the final report is conducted remotely.

The audits are performed using the audit framework presented in the US Environmental Protection Agency’s Guidance on Technical Audits and Related Assessment for Environmental Data Operations (EPA QA/G-7 2000, available at www.epa.gov/sites/production/files/2015-07/documents/g7-final.pdf). The auditors’ review will be from a QA perspective and will not focus on the technical design aspects, although it is acknowledged that there is some overlap.


2.2. ELEMENTS OF A QA AUDIT DURING THE STUDY

The key elements of a QA audit include:

  1. Opening Meeting with the audit team, the PI, and key project personnel.
  2. Observation of the project activities performed by the personnel who regularly perform such activities.
  3. Examining study documentation, such as the QA plan, progress reports, code book or data dictionary, calibration readouts, process data readouts, sample logs, custody papers, instrument logs, printouts from data spreadsheets, and maintenance notebooks.
  4. Interviews with the project personnel to verify the results of observation and to clarify issues noted during document review.
  5. Objective Evidence Compilation, such as model outputs and source code documentation. This includes an audit of the data trail. It may include live demonstrations of the data transformation processes by investigators, and/or snapshots of data processing activities, depending on whether there are restrictions on accessing the data (e.g. due to confidentiality).  
  6. Closing Meeting, during which the QA consultant provides a verbal summary to the Principal Investigator of significant findings that need to be addressed.
  7. QA Audit Report. The audit team prepares a “Business Confidential” report of the audit. The report shall detail the nature of the audit, significant findings, and any requirements for corrective action(s). The audit report shall be provided to the HEI Director of Science, who will then transmit it to the HEI project manager for transmission to and discussion with the PI. If corrective action is required, the PI will ensure that such action is taken and will transmit the relevant documentation to the HEI project manager, who will send it for additional review to the audit team. All copies of the audit report are to be marked as “Business Confidential” and are to be destroyed after use or maintained in a file separate from other records of the project. These audit reports are only to be released to people directly involved in management of the projects. The audit team shall maintain a log of all audits indicating for each audit: the date conducted, participating personnel, and the nature of the audit.


2.3. ELEMENTS OF A QA AUDIT OF THE FINAL REPORT

During the QA audit of the Investigator’s final report, the auditors will review the draft final report to ensure that the data and results reported accurately represent the data collected and processed. The auditors will ensure that the methods used are well documented and clearly explained in terms of assumptions, model development, and validation procedures. The auditors will also ensure that the interpretation of results and the conclusions are consistent with the data shown in figures and tables and cited in the text. The auditors will also verify that the records of deviations from the study and QA plans and SOPs are included in the study records. The auditors may request the metadata and statistical code, and samples of raw (or aggregated) data to determine data traceability and to verify whether the model results are consistent with the information summarized in the final report. The level of aggregation employed in the raw data used for the data traceability may be determined and limited by the privacy and confidentiality restrictions in place.

Similar to the “Business Confidential” audit report during the study, the auditors will provide a final QA Audit report, and will suggest recommendations for the investigators to implement as part of finalizing their HEI research report. The audit team will provide a final QA statement, which will be printed in the final published report.


2.4. TIMING OF QA AUDIT

While the exact timing of the audits varies across studies, the followed guidelines should be followed when defining the general plan and scope of the QA oversight for a study:

A. Audits during the course of the research period

Typically an on-site audit is conducted at the end of Year 1 or during Year 2 to ensure that data collection is done according to the protocol, the data collected are traceable, and a data management plan is in place. If problems are encountered and not addressed adequately, a follow-up visit may be needed.

B. Audit of the final report

Unless there are specific reasons to expedite the review of a final report, the timing of the final report QA audit will be decided during the first discussion of the draft final report by the Review Committee. The following guidelines will be followed:

  1. If the Review Committee thinks that the draft final report does not require additional analyses, then a QA audit of the draft report should be scheduled immediately after the first discussion by the Review Committee, so the investigators can address all issues raised by the auditors in the revised report.
  2. If the Review Committee thinks that the draft final report requires substantive changes and/or reanalysis of the data, the QA audit should be conducted on the revised final report, as soon as it is received by HEI.
  3. Regardless of the timing of the final report audit, the auditors should always be provided with the final “accepted” version of the report and asked to review it before issuing the final QA Statement, which will be printed in the final published report.


DOWNLOADS

Form for submitting a QA plan to HEI (word file)

Quality Assurance and Quality Control procedures (pdf of the text on this page)