Correspondence Volume 122 | Issue 3 | March 2014
Environ Health Perspect; DOI:10.1289/ehp.1307727
Instruments for Assessing Risk of Bias and Other Methodological Criteria of Animal Studies: Omission of Well-Established Methods
Nancy B. Beck,1 Richard A. Becker,1* Alan Boobis,2* Dean Fergusson,3* John R. Fowle III,4* Julie Goodman,5* Sebastian Hoffmann,6* Manoj Lalu,7* Marcel Leist,8*and Martin L. Stephens9*
All authors contributed equally and are listed in alphabetical order.
*Members of the Evidence-Based Toxicology Collaboration (EBTC), an initiative of scientists in academia, industry, and government who are interested in promoting evidence-based approaches to strengthen decision making in the safety sciences (see http://www.ebtox.com).
Citation: Beck NB, Becker RA, Boobis A, Fergusson D, Fowle JR III, Goodman J, Hoffmann S, Lalu M, Leist M, Stephens ML. 2014. Instruments for assessing risk of bias and other methodological criteria of animal studies: omission of well-established methods. Environ Health Perspect 122:A66–A67; http://dx.doi.org/10.1289/ehp.1307727
The authors had complete control over the design, conduct, interpretation, and reporting of the analyses included in this letter. The contents are solely the responsibility of the authors and do not necessarily reflect the official opinions or policies of the authors’ employers or clients.
None of the authors received specific financial support or honorarium as compensation for developing this letter. Several authors are members of the Evidence-Based Toxicology Collaboration (EBTC), and M.L. Stephens and S. Hoffmann serve as the secretariats for the North American and European EBTC Steering Committees, respectively, for which they are compensated for their time. The EBTC’s overall aims are to improve toxicological decision making, facilitate the modernization of the toxicological toolbox, and reinvigorate the safety sciences (see http://www.ebtox.com). S. Hoffmann, J.R. Fowle III, and J. Goodman are consultants and have worked on a range of toxicity and risk assessment issues for a wide variety of clients. R.A. Becker and N.B. Beck are employed by the American Chemistry Council, a trade association of chemical manufacturers. A. Boobis, D. Fergusson, M. Lalu, and M. Leist are employed by institutes of higher education.
Published: 1 March 2014
Related EHP Correspondence
In response to the systematic review by Krauth et al. (2013) of instruments for assessing animal toxicology studies for risk of bias and other aspects of quality, we propose the need for a broader perspective when appraising—and hopefully improving—such studies.
Krauth et al. (2013) reviewed 30 instruments, 4 of which were designed for environmental toxicology studies used to evaluate human and ecological health hazards. The authors noted that these instruments were derived from preclinical pharmaceutical research in animal models. Many of these instruments focus on efficacy and not toxicity, and—as acknowledged by the authors—they may have limited potential application in environmental health research because they often have criteria that are not relevant to hazard and risk assessments.
Based on these 30 instruments, Krauth et al. concluded that a limited number of risk of bias assessment criteria have been empirically tested for animal research, including randomization, concealment of allocation, blinding, and accounting for all animals. However, the authors did not discuss which elements of risk of bias criteria have been empirically tested, nor did they discuss how they were tested, leaving the reader with no information on their reliability or usefulness.
We would like to bring the readers’ attention to several other important publications in environmental chemical health hazard assessment that are pertinent to this topic (Ågerstrand et al. 2011; Hulzebos et al. 2010; Schneider et al. 2009), along with a U.S. Environmental Protection Agency (EPA) approach developed under the High Production Volume Challenge (U.S. EPA 1999b) as well as relevant and potentially eligible guidance developed by the U.S. EPA (1999a) and the Food and Drug Administration (FDA 2003). In addition, the majority of the procedures specified in Good Laboratory Practices and regulatory in vivo toxicity test guidelines (e.g., U.S. EPA 2013; Organisation for Economic Co-operation and Development 1999) were specifically developed to minimize systematic errors, assure high quality data and produce scientifically reliable studies.
These additional publications describe design, conduct, and reporting criteria that form the basis of the methodologies employed globally to assure quality and reliability of in vivo toxicological investigations for regulatory assessment of human and ecological health hazards. Because the application of systematic review and related evidence-based approaches in toxicology is still in its infancy, it is especially important at this time to recognize the contributions of these publications.
The omission of these publications by Krauth et al. could have major science policy implications. The National Toxicology Program (NTP) (whose parent organization, the National Institute of Environmental Health Sciences, funded the research of Krauth et al.) has begun relying on Krauth et al. (2013) to identify elements of risk of bias in evaluating animal studies of environmental agents as part of its systematic reviews for assessing health effects (NTP 2013a, 2013b). The reliance on criteria that have not been transparently empirically tested instead of well-established methodological criteria developed by authoritative national and international organizations could result in biased systematic reviews that ultimately lead to regulations or classifications not supported by the science.
We suggest that further work is warranted in pulling together published perspectives on how to evaluate study quality in animal toxicology studies. Issues in appraising such studies for evaluating environmental hazards to humans and wildlife go well beyond those of human clinical trials, and would benefit from collaboration of experts in animal toxicology with experts in human clinical trials of medical interventions and human epidemiology.
Ågerstrand M, Breitholtz M, Rudén C. 2011. Comparison of four different methods for reliability evaluation of ecotoxicity data: a case study of non-standard test data used in environmental risk assessments of pharmaceutical substances. Environ Sci Eur 23:17; doi: 10.1186/2190-4715-23-17.
FDA (Food and Drug Administration). 2003. General Guidelines for Designing and Conducting Toxicity Studies. In: Guidance for Industry and Other Stakeholders, Toxicological Principles for the Safety Assessment of Food Ingredients, Redbook 2000. Available: http://www.fda.gov/Food/GuidanceRegulation/GuidanceDocumentsRegulatoryInformation/IngredientsAdditivesGRASPackaging/ucm078315.htm [accessed 15 October 2013].
Krauth D, Woodruff TJ, Bero L. 2013. Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review. Environ Health Perspect 121:985–992; doi: 10.1289/ehp.1206389.
NTP (National Toxicology Program). 2013a. Appendix 2: Risk of Bias Guidance for BPA Exposure and Obesity Protocol. Available: http://ntp.niehs.nih.gov/NTP/OHAT/EvaluationProcess/Appendix2BPA_Draft.pdf [accessed 13 February 2014].
NTP (National Toxicology Program). 2013b. Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments. Available: http://ntp.niehs.nih.gov/NTP/OHAT/EvaluationProcess/DraftOHATApproach_February2013.pdf [accessed 15 October 2013].
Organisation for Economic Co-operation and Development. 1998. OECD Series on Principles of Good Laboratory Practice and Compliance Monitoring, No 1: OECD Principles on Good Laboratory Practice. ENV/MC/CHEM(98)17. Paris:OECD. Available: http://search.oecd.org/officialdocuments/displaydocumentpdf/?doclanguage=en&cote=env/mc/chem(98)17 [accessed 13 February 2014].
Schneider K, Schwarz M, Burkholder I, Kopp-Schneider A, Edler L, Kinsner-Ovaskainen A, et al. 2009. “ToxRTool”, a new tool to assess the reliability of toxicological data. Toxicol Lett 189(2):138–144.
U.S. EPA (U.S. Environmental Protection Agency). 1999a. Auditing General Toxicology Studies. Available: http://www.epa.gov/compliance/resources/policies/monitoring/fifra/sop/glp-da-09.pdf [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 1999b. Determining the Adequacy of Existing Data. Available: http://www.epa.gov/hpv/pubs/general/datadfin.htm [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 2013. OCSPP Harmonized Test Guidelines. Series 870: Health Effects Test Guidelines. Available: http://www.epa.gov/ocspp/pubs/frs/publications/Test_Guidelines/series870.htm [accessed 15 October 2013].
New Impact Factor
We are pleased to announce our new impact factor of 7.03 and—for the first time—our inclusion in the category of Toxicology. Thank you to our authors and readers, who continue to make EHP a leading journal in Environmental Sciences, in Public, Environmental, and Occupational Health, and now in Toxicology.
EHP News Feature Wins Award
Our March 2013 cover story, “A Tale of Two Forests: Addressing Postnuclear Radiation at Chernobyl and Fukushima,” has won first place in the Outstanding Feature Story category of the Society of Environmental Journalists Awards for Reporting on the Environment. The judges called the story “an exceedingly vital and original piece of environmental feature writing.” Congratulations to authors Winifred A. Bird and Jane Braxton Little on their exciting achievement!
CEHN August 2014 Article of the Month
“Gestational Exposure to Endocrine-Disrupting Chemicals and Reciprocal Social, Repetitive and Stereotypic Behaviors in 4- and 5-Year-Old Children: The HOME Study” (Environ Health Perspect; DOI:10.1289/ehp.1307261) has been selected by the Children’s Environmental Health Network (CEHN) as its August 2014 Article of the Month. These CEHN summaries discuss the potential policy implications of current children’s environmental health research.
Register Today for PPTOX IVEnvironmental Stressors in Disease and Implications for Human Health
October 26–29, 2014
Boston Marriott Long Wharf Hotel
Join the Endocrine Society for the fourth international summit of Prenatal Programming and Toxicity (PPTOX), a premiere conference series dedicated to cutting-edge discussion of environmental hazards during early life and long-term consequences.Key deadlines:
* Abstract submission: Tuesday, September 2
* Housing deadline: Monday, October 6
For more information, visit the Endocrine Society website.
Sign Up to Receive E-mail Alerts
Recent Advance Publications
- Effect of Exposure to Atmospheric Ultrafine Particles on Production of Free Fatty Acids and Lipid Metabolites in the Mouse Small Intestine
- Risks and Benefits of Green Spaces for Children: A Cross-Sectional Study of Associations with Sedentary Behavior, Obesity, Asthma, and Allergy
- Genistein Disrupts Glucocorticoid Receptor Signaling in Human Uterine Endometrial Ishikawa Cells
- Modification of the Association between PM10 and Lung Function Decline by Cadherin 13 Polymorphisms in the SAPALDIA Cohort: A Genome-Wide Interaction Analysis
- Mitochondria, Energetics, Epigenetics, and Cellular Responses to Stress
- Incense Use and Cardiovascular Mortality among Chinese in Singapore: The Singapore Chinese Health Study
- Perfluorochemicals and Human Semen Quality: The LIFE Study
- AdvPubl: Atmospheric ultrafine particles and intestinal lipid metabolism http://t.co/flqRKKLMep
- Trending EHP news this month: Nanosilver: weighing the risks and benefits http://t.co/6oSYgL8MNO
- Trending EHP research this month: The societal costs and benefits of commuter bicycling http://t.co/yGlks7cL5l
- EHPNoonNews: Big Food to divulge chemical info http://t.co/jaCI6PWQRx @washingtonpost EHP lookback http://t.co/1eRkamlSkF
- EHPNoonNews: EPA identifies health concern from chemical used in paint strippers http://t.co/yMIisVom4Y @epa