Correspondence March 2014 | Volume 122 | Issue 3
Instruments for Assessing Risk of Bias and Other Methodological Criteria of Animal Studies: Omission of Well-Established Methods
Nancy B. Beck,1 Richard A. Becker,1* Alan Boobis,2* Dean Fergusson,3* John R. Fowle III,4* Julie Goodman,5* Sebastian Hoffmann,6* Manoj Lalu,7* Marcel Leist,8*and Martin L. Stephens9*
All authors contributed equally and are listed in alphabetical order.
*Members of the Evidence-Based Toxicology Collaboration (EBTC), an initiative of scientists in academia, industry, and government who are interested in promoting evidence-based approaches to strengthen decision making in the safety sciences (see http://www.ebtox.com).
Citation: Beck NB, Becker RA, Boobis A, Fergusson D, Fowle JR III, Goodman J, Hoffmann S, Lalu M, Leist M, Stephens ML. 2014. Instruments for assessing risk of bias and other methodological criteria of animal studies: omission of well-established methods. Environ Health Perspect 122:A66–A67; http://dx.doi.org/10.1289/ehp.1307727
The authors had complete control over the design, conduct, interpretation, and reporting of the analyses included in this letter. The contents are solely the responsibility of the authors and do not necessarily reflect the official opinions or policies of the authors’ employers or clients.
None of the authors received specific financial support or honorarium as compensation for developing this letter. Several authors are members of the Evidence-Based Toxicology Collaboration (EBTC), and M.L. Stephens and S. Hoffmann serve as the secretariats for the North American and European EBTC Steering Committees, respectively, for which they are compensated for their time. The EBTC’s overall aims are to improve toxicological decision making, facilitate the modernization of the toxicological toolbox, and reinvigorate the safety sciences (see http://www.ebtox.com). S. Hoffmann, J.R. Fowle III, and J. Goodman are consultants and have worked on a range of toxicity and risk assessment issues for a wide variety of clients. R.A. Becker and N.B. Beck are employed by the American Chemistry Council, a trade association of chemical manufacturers. A. Boobis, D. Fergusson, M. Lalu, and M. Leist are employed by institutes of higher education. In the past 3 years, A. Boobis and M. Leist have worked on a range of toxicity and risk assessment issues for a number of clients; this has included some consultancies.
Published: 1 March 2014
Related EHP Correspondence
In response to the systematic review by Krauth et al. (2013) of instruments for assessing animal toxicology studies for risk of bias and other aspects of quality, we propose the need for a broader perspective when appraising—and hopefully improving—such studies.
Krauth et al. (2013) reviewed 30 instruments, 4 of which were designed for environmental toxicology studies used to evaluate human and ecological health hazards. The authors noted that these instruments were derived from preclinical pharmaceutical research in animal models. Many of these instruments focus on efficacy and not toxicity, and—as acknowledged by the authors—they may have limited potential application in environmental health research because they often have criteria that are not relevant to hazard and risk assessments.
Based on these 30 instruments, Krauth et al. concluded that a limited number of risk of bias assessment criteria have been empirically tested for animal research, including randomization, concealment of allocation, blinding, and accounting for all animals. However, the authors did not discuss which elements of risk of bias criteria have been empirically tested, nor did they discuss how they were tested, leaving the reader with no information on their reliability or usefulness.
We would like to bring the readers’ attention to several other important publications in environmental chemical health hazard assessment that are pertinent to this topic (Ågerstrand et al. 2011; Hulzebos et al. 2010; Schneider et al. 2009), along with a U.S. Environmental Protection Agency (EPA) approach developed under the High Production Volume Challenge (U.S. EPA 1999b) as well as relevant and potentially eligible guidance developed by the U.S. EPA (1999a) and the Food and Drug Administration (FDA 2003). In addition, the majority of the procedures specified in Good Laboratory Practices and regulatory in vivo toxicity test guidelines (e.g., U.S. EPA 2013; Organisation for Economic Co-operation and Development 1999) were specifically developed to minimize systematic errors, assure high quality data and produce scientifically reliable studies.
These additional publications describe design, conduct, and reporting criteria that form the basis of the methodologies employed globally to assure quality and reliability of in vivo toxicological investigations for regulatory assessment of human and ecological health hazards. Because the application of systematic review and related evidence-based approaches in toxicology is still in its infancy, it is especially important at this time to recognize the contributions of these publications.
The omission of these publications by Krauth et al. could have major science policy implications. The National Toxicology Program (NTP) (whose parent organization, the National Institute of Environmental Health Sciences, funded the research of Krauth et al.) has begun relying on Krauth et al. (2013) to identify elements of risk of bias in evaluating animal studies of environmental agents as part of its systematic reviews for assessing health effects (NTP 2013a, 2013b). The reliance on criteria that have not been transparently empirically tested instead of well-established methodological criteria developed by authoritative national and international organizations could result in biased systematic reviews that ultimately lead to regulations or classifications not supported by the science.
We suggest that further work is warranted in pulling together published perspectives on how to evaluate study quality in animal toxicology studies. Issues in appraising such studies for evaluating environmental hazards to humans and wildlife go well beyond those of human clinical trials, and would benefit from collaboration of experts in animal toxicology with experts in human clinical trials of medical interventions and human epidemiology.
In the original version of this article, information for Alan Boobis and Marcel Leist was omitted from the Competing Financial Interests declaration. The complete declaration is as follows:
None of the authors received specific financial support or honorarium as compensation for developing this letter. Several authors are members of the Evidence-Based Toxicology Collaboration (EBTC), and M.L. Stephens and S. Hoffmann serve as the secretariats for the North American and European EBTC Steering Committees, respectively, for which they are compensated for their time. The EBTC’s overall aims are to improve toxicological decision making, facilitate the modernization of the toxicological toolbox, and reinvigorate the safety sciences (see http://www.ebtox. com). S. Hoffmann, J.R. Fowle III, and J. Goodman are consultants and have worked on a range of toxicity and risk assessment issues for a wide variety of clients. R.A. Becker and N.B. Beck are employed by the American Chemistry Council, a trade association of chemical manufacturers. A. Boobis, D. Fergusson, M. Lalu, and M. Leist are employed by institutes of higher education. In the past 3 years, A. Boobis and M. Leist have worked on a range of toxicity and risk assessment issues for a number of clients; this has included some consultancies.
The authors regret the error.
Ågerstrand M, Breitholtz M, Rudén C. 2011. Comparison of four different methods for reliability evaluation of ecotoxicity data: a case study of non-standard test data used in environmental risk assessments of pharmaceutical substances. Environ Sci Eur 23:17; doi: 10.1186/2190-4715-23-17.
FDA (Food and Drug Administration). 2003. General Guidelines for Designing and Conducting Toxicity Studies. In: Guidance for Industry and Other Stakeholders, Toxicological Principles for the Safety Assessment of Food Ingredients, Redbook 2000. Available: http://www.fda.gov/Food/GuidanceRegulation/GuidanceDocumentsRegulatoryInformation/IngredientsAdditivesGRASPackaging/ucm078315.htm [accessed 15 October 2013].
Krauth D, Woodruff TJ, Bero L. 2013. Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review. Environ Health Perspect 121:985–992; doi: 10.1289/ehp.1206389.
NTP (National Toxicology Program). 2013a. Appendix 2: Risk of Bias Guidance for BPA Exposure and Obesity Protocol. Available: http://ntp.niehs.nih.gov/NTP/OHAT/EvaluationProcess/Appendix2BPA_Draft.pdf [accessed 13 February 2014].
NTP (National Toxicology Program). 2013b. Draft OHAT Approach for Systematic Review and Evidence Integration for Literature-based Health Assessments. Available: http://ntp.niehs.nih.gov/NTP/OHAT/EvaluationProcess/DraftOHATApproach_February2013.pdf [accessed 15 October 2013].
Organisation for Economic Co-operation and Development. 1998. OECD Series on Principles of Good Laboratory Practice and Compliance Monitoring, No 1: OECD Principles on Good Laboratory Practice. ENV/MC/CHEM(98)17. Paris:OECD. Available: http://search.oecd.org/officialdocuments/displaydocumentpdf/?doclanguage=en&cote=env/mc/chem(98)17 [accessed 13 February 2014].
Schneider K, Schwarz M, Burkholder I, Kopp-Schneider A, Edler L, Kinsner-Ovaskainen A, et al. 2009. “ToxRTool”, a new tool to assess the reliability of toxicological data. Toxicol Lett 189(2):138–144.
U.S. EPA (U.S. Environmental Protection Agency). 1999a. Auditing General Toxicology Studies. Available: http://www.epa.gov/compliance/resources/policies/monitoring/fifra/sop/glp-da-09.pdf [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 1999b. Determining the Adequacy of Existing Data. Available: http://www.epa.gov/hpv/pubs/general/datadfin.htm [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 2013. OCSPP Harmonized Test Guidelines. Series 870: Health Effects Test Guidelines. Available: http://www.epa.gov/ocspp/pubs/frs/publications/Test_Guidelines/series870.htm [accessed 15 October 2013].
EHP is pleased to announce that “The Prevalence of Antibiotic-Resistant Staphylococcus aureus Nasal Carriage among Industrial Hog Operation Workers, Community Residents, and Children Living in Their Households: North Carolina, USA,” published in EHP in the April 2017 issue, has been selected by the Children’s Environmental Health Network (CEHN) as its October 2017 Article of the Month. (more…)
EHP is pleased to announce that “A Difference-in-Differences Approach to Assess the Effect of a Heat Action Plan on Heat-Related Mortality, and Differences in Effectiveness According to Sex, Age, and Socioeconomic Status (Montreal, Quebec)” has been awarded Honorable Mention as one of the Best Environmental Epidemiology Papers published in 2016. (more…)