Correspondence Volume 122 | Issue 3 | March 2014
Environ Health Perspect; DOI:10.1289/ehp.1307727
Instruments for Assessing Risk of Bias and Other Methodological Criteria of Animal Studies: Omission of Well-Established Methods
Nancy B. Beck,1 Richard A. Becker,1* Alan Boobis,2* Dean Fergusson,3* John R. Fowle III,4* Julie Goodman,5* Sebastian Hoffmann,6* Manoj Lalu,7* Marcel Leist,8*and Martin L. Stephens9*
All authors contributed equally and are listed in alphabetical order.
*Members of the Evidence-Based Toxicology Collaboration (EBTC), an initiative of scientists in academia, industry, and government who are interested in promoting evidence-based approaches to strengthen decision making in the safety sciences (see http://www.ebtox.com).
Citation: Beck NB, Becker RA, Boobis A, Fergusson D, Fowle JR III, Goodman J, Hoffmann S, Lalu M, Leist M, Stephens ML. 2014. Instruments for assessing risk of bias and other methodological criteria of animal studies: omission of well-established methods. Environ Health Perspect 122:A66–A67; http://dx.doi.org/10.1289/ehp.1307727
The authors had complete control over the design, conduct, interpretation, and reporting of the analyses included in this letter. The contents are solely the responsibility of the authors and do not necessarily reflect the official opinions or policies of the authors’ employers or clients.
None of the authors received specific financial support or honorarium as compensation for developing this letter. Several authors are members of the Evidence-Based Toxicology Collaboration (EBTC), and M.L. Stephens and S. Hoffmann serve as the secretariats for the North American and European EBTC Steering Committees, respectively, for which they are compensated for their time. The EBTC’s overall aims are to improve toxicological decision making, facilitate the modernization of the toxicological toolbox, and reinvigorate the safety sciences (see http://www.ebtox.com). S. Hoffmann, J.R. Fowle III, and J. Goodman are consultants and have worked on a range of toxicity and risk assessment issues for a wide variety of clients. R.A. Becker and N.B. Beck are employed by the American Chemistry Council, a trade association of chemical manufacturers. A. Boobis, D. Fergusson, M. Lalu, and M. Leist are employed by institutes of higher education. In the past 3 years, A. Boobis and M. Leist have worked on a range of toxicity and risk assessment issues for a number of clients; this has included some consultancies.
Published: 1 March 2014
Related EHP Correspondence
In response to the systematic review by Krauth et al. (2013) of instruments for assessing animal toxicology studies for risk of bias and other aspects of quality, we propose the need for a broader perspective when appraising—and hopefully improving—such studies.
Krauth et al. (2013) reviewed 30 instruments, 4 of which were designed for environmental toxicology studies used to evaluate human and ecological health hazards. The authors noted that these instruments were derived from preclinical pharmaceutical research in animal models. Many of these instruments focus on efficacy and not toxicity, and—as acknowledged by the authors—they may have limited potential application in environmental health research because they often have criteria that are not relevant to hazard and risk assessments.
Based on these 30 instruments, Krauth et al. concluded that a limited number of risk of bias assessment criteria have been empirically tested for animal research, including randomization, concealment of allocation, blinding, and accounting for all animals. However, the authors did not discuss which elements of risk of bias criteria have been empirically tested, nor did they discuss how they were tested, leaving the reader with no information on their reliability or usefulness.
We would like to bring the readers’ attention to several other important publications in environmental chemical health hazard assessment that are pertinent to this topic (Ågerstrand et al. 2011; Hulzebos et al. 2010; Schneider et al. 2009), along with a U.S. Environmental Protection Agency (EPA) approach developed under the High Production Volume Challenge (U.S. EPA 1999b) as well as relevant and potentially eligible guidance developed by the U.S. EPA (1999a) and the Food and Drug Administration (FDA 2003). In addition, the majority of the procedures specified in Good Laboratory Practices and regulatory in vivo toxicity test guidelines (e.g., U.S. EPA 2013; Organisation for Economic Co-operation and Development 1999) were specifically developed to minimize systematic errors, assure high quality data and produce scientifically reliable studies.
These additional publications describe design, conduct, and reporting criteria that form the basis of the methodologies employed globally to assure quality and reliability of in vivo toxicological investigations for regulatory assessment of human and ecological health hazards. Because the application of systematic review and related evidence-based approaches in toxicology is still in its infancy, it is especially important at this time to recognize the contributions of these publications.
The omission of these publications by Krauth et al. could have major science policy implications. The National Toxicology Program (NTP) (whose parent organization, the National Institute of Environmental Health Sciences, funded the research of Krauth et al.) has begun relying on Krauth et al. (2013) to identify elements of risk of bias in evaluating animal studies of environmental agents as part of its systematic reviews for assessing health effects (NTP 2013a, 2013b). The reliance on criteria that have not been transparently empirically tested instead of well-established methodological criteria developed by authoritative national and international organizations could result in biased systematic reviews that ultimately lead to regulations or classifications not supported by the science.
We suggest that further work is warranted in pulling together published perspectives on how to evaluate study quality in animal toxicology studies. Issues in appraising such studies for evaluating environmental hazards to humans and wildlife go well beyond those of human clinical trials, and would benefit from collaboration of experts in animal toxicology with experts in human clinical trials of medical interventions and human epidemiology.
In the original version of this article, information for Alan Boobis and Marcel Leist was omitted from the Competing Financial Interests declaration. The complete declaration is as follows:
None of the authors received specific financial support or honorarium as compensation for developing this letter. Several authors are members of the Evidence-Based Toxicology Collaboration (EBTC), and M.L. Stephens and S. Hoffmann serve as the secretariats for the North American and European EBTC Steering Committees, respectively, for which they are compensated for their time. The EBTC’s overall aims are to improve toxicological decision making, facilitate the modernization of the toxicological toolbox, and reinvigorate the safety sciences (see http://www.ebtox. com). S. Hoffmann, J.R. Fowle III, and J. Goodman are consultants and have worked on a range of toxicity and risk assessment issues for a wide variety of clients. R.A. Becker and N.B. Beck are employed by the American Chemistry Council, a trade association of chemical manufacturers. A. Boobis, D. Fergusson, M. Lalu, and M. Leist are employed by institutes of higher education. In the past 3 years, A. Boobis and M. Leist have worked on a range of toxicity and risk assessment issues for a number of clients; this has included some consultancies.
The authors regret the error.
Ågerstrand M, Breitholtz M, Rudén C. 2011. Comparison of four different methods for reliability evaluation of ecotoxicity data: a case study of non-standard test data used in environmental risk assessments of pharmaceutical substances. Environ Sci Eur 23:17; doi: 10.1186/2190-4715-23-17.
FDA (Food and Drug Administration). 2003. General Guidelines for Designing and Conducting Toxicity Studies. In: Guidance for Industry and Other Stakeholders, Toxicological Principles for the Safety Assessment of Food Ingredients, Redbook 2000. Available: http://www.fda.gov/Food/GuidanceRegulation/GuidanceDocumentsRegulatoryInformation/IngredientsAdditivesGRASPackaging/ucm078315.htm [accessed 15 October 2013].
Krauth D, Woodruff TJ, Bero L. 2013. Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review. Environ Health Perspect 121:985–992; doi: 10.1289/ehp.1206389.
NTP (National Toxicology Program). 2013a. Appendix 2: Risk of Bias Guidance for BPA Exposure and Obesity Protocol. Available: http://ntp.niehs.nih.gov/NTP/OHAT/EvaluationProcess/Appendix2BPA_Draft.pdf [accessed 13 February 2014].
NTP (National Toxicology Program). 2013b. Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments. Available: http://ntp.niehs.nih.gov/NTP/OHAT/EvaluationProcess/DraftOHATApproach_February2013.pdf [accessed 15 October 2013].
Organisation for Economic Co-operation and Development. 1998. OECD Series on Principles of Good Laboratory Practice and Compliance Monitoring, No 1: OECD Principles on Good Laboratory Practice. ENV/MC/CHEM(98)17. Paris:OECD. Available: http://search.oecd.org/officialdocuments/displaydocumentpdf/?doclanguage=en&cote=env/mc/chem(98)17 [accessed 13 February 2014].
Schneider K, Schwarz M, Burkholder I, Kopp-Schneider A, Edler L, Kinsner-Ovaskainen A, et al. 2009. “ToxRTool”, a new tool to assess the reliability of toxicological data. Toxicol Lett 189(2):138–144.
U.S. EPA (U.S. Environmental Protection Agency). 1999a. Auditing General Toxicology Studies. Available: http://www.epa.gov/compliance/resources/policies/monitoring/fifra/sop/glp-da-09.pdf [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 1999b. Determining the Adequacy of Existing Data. Available: http://www.epa.gov/hpv/pubs/general/datadfin.htm [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 2013. OCSPP Harmonized Test Guidelines. Series 870: Health Effects Test Guidelines. Available: http://www.epa.gov/ocspp/pubs/frs/publications/Test_Guidelines/series870.htm [accessed 15 October 2013].
New Search FunctionWe're pleased to present our improved search page. Try it, and let us know what you think!
CEHN September 2014 Article of the Month“Prenatal Polybrominated Diphenyl Ether Exposures and Neurodevelopment in U.S. Children through 5 Years of Age: The HOME Study” (Environ Health Perspect; DOI:10.1289/ehp.1307562) has been selected by the Children’s Environmental Health Network (CEHN) as its September 2014 Article of the Month. These CEHN summaries discuss the potential policy implications of current children’s environmental health research.
Register Today for PPTOX IVEnvironmental Stressors in Disease and Implications for Human Health
October 26–29, 2014
Boston Marriott Long Wharf Hotel
Join the Endocrine Society for the fourth international summit of Prenatal Programming and Toxicity (PPTOX), a premiere conference series dedicated to cutting-edge discussion of environmental hazards during early life and long-term consequences.* Housing deadline: Monday, October 6
For more information, visit the Endocrine Society website.
Sign Up to Receive E-mail Alerts
Recent Advance Publications
Characteristics of Walkable Built Environments and BMI z-Scores in Children: Evidence from a Large Electronic Health Record Database
Asthma in Inner-City Children at 5-11 Years of Age and Prenatal Exposure to Phthalates: The Columbia Center for Children’s Environmental Health Cohort
Comparative Assessment of the Effects of Climate Change on Heat- and Cold-Related Mortality in the United Kingdom and Australia
Proximity to Natural Gas Wells and Reported Health Status: Results of a Household Survey in Washington County, Pennsylvania
Green and Blue Spaces and Behavioral Development in Barcelona Schoolchildren: The BREATHE Project
Aerial Application of Mancozeb and Urinary Ethylene Thiourea (ETU) Concentrations among Pregnant Women in Costa Rica: The Infants’ Environmental Health Study (ISA)
Pyrethroid Pesticide Exposure and Parental Report of Learning Disability and Attention Deficit/Hyperactivity Disorder in U.S. Children: NHANES 1999–2002
- AdvPubl: Built environments and child body mass index http://t.co/IYIG3xraxB
- EHPNoonNews: Mesothelioma risk endures decades after asbestos exposure http://t.co/3O9QFntpcY @snwa
- EHPNoonNews: Pharmaceuticals from treated wastewater found in shallow groundwater http://t.co/ppLfNVFnOY @usgs
- EHPNoonNews: Reactions to tattoo inks can mimic skin cancer http://t.co/HMFTAmxE79 @washingtonpost
- EHPNoonNews: Tracking epigenetic “memory” from cell to cell and across generations http://t.co/liFYinpBTX @ucsc