Correspondence Volume 122 | Issue 3 | March 2014
Environ Health Perspect; DOI:10.1289/ehp.1307727
Instruments for Assessing Risk of Bias and Other Methodological Criteria of Animal Studies: Omission of Well-Established Methods
Nancy B. Beck,1 Richard A. Becker,1* Alan Boobis,2* Dean Fergusson,3* John R. Fowle III,4* Julie Goodman,5* Sebastian Hoffmann,6* Manoj Lalu,7* Marcel Leist,8*and Martin L. Stephens9*
All authors contributed equally and are listed in alphabetical order.
*Members of the Evidence-Based Toxicology Collaboration (EBTC), an initiative of scientists in academia, industry, and government who are interested in promoting evidence-based approaches to strengthen decision making in the safety sciences (see http://www.ebtox.com).
Citation: Beck NB, Becker RA, Boobis A, Fergusson D, Fowle JR III, Goodman J, Hoffmann S, Lalu M, Leist M, Stephens ML. 2014. Instruments for assessing risk of bias and other methodological criteria of animal studies: omission of well-established methods. Environ Health Perspect 122:A66–A67; http://dx.doi.org/10.1289/ehp.1307727
The authors had complete control over the design, conduct, interpretation, and reporting of the analyses included in this letter. The contents are solely the responsibility of the authors and do not necessarily reflect the official opinions or policies of the authors’ employers or clients.
None of the authors received specific financial support or honorarium as compensation for developing this letter. Several authors are members of the Evidence-Based Toxicology Collaboration (EBTC), and M.L. Stephens and S. Hoffmann serve as the secretariats for the North American and European EBTC Steering Committees, respectively, for which they are compensated for their time. The EBTC’s overall aims are to improve toxicological decision making, facilitate the modernization of the toxicological toolbox, and reinvigorate the safety sciences (see http://www.ebtox.com). S. Hoffmann, J.R. Fowle III, and J. Goodman are consultants and have worked on a range of toxicity and risk assessment issues for a wide variety of clients. R.A. Becker and N.B. Beck are employed by the American Chemistry Council, a trade association of chemical manufacturers. A. Boobis, D. Fergusson, M. Lalu, and M. Leist are employed by institutes of higher education. In the past 3 years, A. Boobis and M. Leist have worked on a range of toxicity and risk assessment issues for a number of clients; this has included some consultancies.
Published: 1 March 2014
Related EHP Correspondence
In response to the systematic review by Krauth et al. (2013) of instruments for assessing animal toxicology studies for risk of bias and other aspects of quality, we propose the need for a broader perspective when appraising—and hopefully improving—such studies.
Krauth et al. (2013) reviewed 30 instruments, 4 of which were designed for environmental toxicology studies used to evaluate human and ecological health hazards. The authors noted that these instruments were derived from preclinical pharmaceutical research in animal models. Many of these instruments focus on efficacy and not toxicity, and—as acknowledged by the authors—they may have limited potential application in environmental health research because they often have criteria that are not relevant to hazard and risk assessments.
Based on these 30 instruments, Krauth et al. concluded that a limited number of risk of bias assessment criteria have been empirically tested for animal research, including randomization, concealment of allocation, blinding, and accounting for all animals. However, the authors did not discuss which elements of risk of bias criteria have been empirically tested, nor did they discuss how they were tested, leaving the reader with no information on their reliability or usefulness.
We would like to bring the readers’ attention to several other important publications in environmental chemical health hazard assessment that are pertinent to this topic (Ågerstrand et al. 2011; Hulzebos et al. 2010; Schneider et al. 2009), along with a U.S. Environmental Protection Agency (EPA) approach developed under the High Production Volume Challenge (U.S. EPA 1999b) as well as relevant and potentially eligible guidance developed by the U.S. EPA (1999a) and the Food and Drug Administration (FDA 2003). In addition, the majority of the procedures specified in Good Laboratory Practices and regulatory in vivo toxicity test guidelines (e.g., U.S. EPA 2013; Organisation for Economic Co-operation and Development 1999) were specifically developed to minimize systematic errors, assure high quality data and produce scientifically reliable studies.
These additional publications describe design, conduct, and reporting criteria that form the basis of the methodologies employed globally to assure quality and reliability of in vivo toxicological investigations for regulatory assessment of human and ecological health hazards. Because the application of systematic review and related evidence-based approaches in toxicology is still in its infancy, it is especially important at this time to recognize the contributions of these publications.
The omission of these publications by Krauth et al. could have major science policy implications. The National Toxicology Program (NTP) (whose parent organization, the National Institute of Environmental Health Sciences, funded the research of Krauth et al.) has begun relying on Krauth et al. (2013) to identify elements of risk of bias in evaluating animal studies of environmental agents as part of its systematic reviews for assessing health effects (NTP 2013a, 2013b). The reliance on criteria that have not been transparently empirically tested instead of well-established methodological criteria developed by authoritative national and international organizations could result in biased systematic reviews that ultimately lead to regulations or classifications not supported by the science.
We suggest that further work is warranted in pulling together published perspectives on how to evaluate study quality in animal toxicology studies. Issues in appraising such studies for evaluating environmental hazards to humans and wildlife go well beyond those of human clinical trials, and would benefit from collaboration of experts in animal toxicology with experts in human clinical trials of medical interventions and human epidemiology.
In the original version of this article, information for Alan Boobis and Marcel Leist was omitted from the Competing Financial Interests declaration. The complete declaration is as follows:
None of the authors received specific financial support or honorarium as compensation for developing this letter. Several authors are members of the Evidence-Based Toxicology Collaboration (EBTC), and M.L. Stephens and S. Hoffmann serve as the secretariats for the North American and European EBTC Steering Committees, respectively, for which they are compensated for their time. The EBTC’s overall aims are to improve toxicological decision making, facilitate the modernization of the toxicological toolbox, and reinvigorate the safety sciences (see http://www.ebtox. com). S. Hoffmann, J.R. Fowle III, and J. Goodman are consultants and have worked on a range of toxicity and risk assessment issues for a wide variety of clients. R.A. Becker and N.B. Beck are employed by the American Chemistry Council, a trade association of chemical manufacturers. A. Boobis, D. Fergusson, M. Lalu, and M. Leist are employed by institutes of higher education. In the past 3 years, A. Boobis and M. Leist have worked on a range of toxicity and risk assessment issues for a number of clients; this has included some consultancies.
The authors regret the error.
Ågerstrand M, Breitholtz M, Rudén C. 2011. Comparison of four different methods for reliability evaluation of ecotoxicity data: a case study of non-standard test data used in environmental risk assessments of pharmaceutical substances. Environ Sci Eur 23:17; doi: 10.1186/2190-4715-23-17.
FDA (Food and Drug Administration). 2003. General Guidelines for Designing and Conducting Toxicity Studies. In: Guidance for Industry and Other Stakeholders, Toxicological Principles for the Safety Assessment of Food Ingredients, Redbook 2000. Available: http://www.fda.gov/Food/GuidanceRegulation/GuidanceDocumentsRegulatoryInformation/IngredientsAdditivesGRASPackaging/ucm078315.htm [accessed 15 October 2013].
Krauth D, Woodruff TJ, Bero L. 2013. Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review. Environ Health Perspect 121:985–992; doi: 10.1289/ehp.1206389.
NTP (National Toxicology Program). 2013a. Appendix 2: Risk of Bias Guidance for BPA Exposure and Obesity Protocol. Available: http://ntp.niehs.nih.gov/NTP/OHAT/EvaluationProcess/Appendix2BPA_Draft.pdf [accessed 13 February 2014].
NTP (National Toxicology Program). 2013b. Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments. Available: http://ntp.niehs.nih.gov/NTP/OHAT/EvaluationProcess/DraftOHATApproach_February2013.pdf [accessed 15 October 2013].
Organisation for Economic Co-operation and Development. 1998. OECD Series on Principles of Good Laboratory Practice and Compliance Monitoring, No 1: OECD Principles on Good Laboratory Practice. ENV/MC/CHEM(98)17. Paris:OECD. Available: http://search.oecd.org/officialdocuments/displaydocumentpdf/?doclanguage=en&cote=env/mc/chem(98)17 [accessed 13 February 2014].
Schneider K, Schwarz M, Burkholder I, Kopp-Schneider A, Edler L, Kinsner-Ovaskainen A, et al. 2009. “ToxRTool”, a new tool to assess the reliability of toxicological data. Toxicol Lett 189(2):138–144.
U.S. EPA (U.S. Environmental Protection Agency). 1999a. Auditing General Toxicology Studies. Available: http://www.epa.gov/compliance/resources/policies/monitoring/fifra/sop/glp-da-09.pdf [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 1999b. Determining the Adequacy of Existing Data. Available: http://www.epa.gov/hpv/pubs/general/datadfin.htm [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 2013. OCSPP Harmonized Test Guidelines. Series 870: Health Effects Test Guidelines. Available: http://www.epa.gov/ocspp/pubs/frs/publications/Test_Guidelines/series870.htm [accessed 15 October 2013].
ISEE 2015 Abstracts Now AvailableEHP is pleased to present the abstracts for the 2015 annual conference of the International Society for Environmental Epidemiology (ISEE), “Addressing Environmental Health Inequalities,” being held 30 August–3 September 2015 in São Paulo, Brazil.
New Editor-in-ChiefWe are pleased to announce that Sally Perreault Darney has been selected as the new Editor-in-Chief of EHP. Sally comes to EHP from the U.S. Environmental Protection Agency, where she most recently co-led a large research project focused on assessing health disparities in vulnerable groups and providing healthy environments for children. Learn more about Sally and her vision for the journal in the September issue of EHP.
CEHN August 2015 Article of the Month“Ambient Heat and Sudden Infant Death: A Case-Crossover Study Spanning 30 years in Montreal, Canada” (DOI:10.1289/ehp.1307960) has been selected by the Children’s Environmental Health Network (CEHN) as its August 2015 Article of the Month. These CEHN summaries discuss the potential policy implications of current children’s environmental health research.
Sign Up to Receive E-mail Alerts
Recent Advance Publications
Prenatal Ambient Air Pollution, Placental Mitochondrial DNA Content, and Birth Weight in the INMA (Spain) and ENVIRONAGE (Belgium) Birth Cohorts
Endocrine-Disrupting Chemicals and Oil and Natural Gas Operations: Potential Environmental Contamination and Recommendations to Assess Complex Environmental Mixtures
Bisphenol A, Bisphenol S, and 4-Hydroxyphenyl 4-Isoprooxyphenylsulfone (BPSIP) in Urine and Blood of Cashiers
Prenatal Phthalate Exposures and Childhood Fat Mass in a New York City Cohort
Effects of in Utero Exposure to Arsenic during the Second Half of Gestation on Reproductive End Points and Metabolic Parameters in Female CD-1 Mice