Correspondence Volume 122 | Issue 3 | March 2014
Environ Health Perspect; DOI:10.1289/ehp.1307727R
Instruments for Assessing Risk of Bias and Other Methodological Criteria: Krauth et al. Respond
David Krauth,1 Tracey J. Woodruff,2,3 Lisa Bero1,4
Citation: Krauth D, Woodruff TJ, Bero L. 2014. Instruments for assessing risk of bias and other methodological criteria: Krauth et al. respond. Environ Health Perspect 122:A67; http://dx.doi.org/10.1289/ehp.1307727R
The authors declare they have no actual or potential competing financial interests.
Published: 1 March 2014
Related EHP Correspondence
Beck et al. criticize our systematic review (Krauth et al. 2013) because we included instruments derived from preclinical animal research. Assessment instruments developed for preclinical animal models have criteria that are relevant to hazard and risk assessment because risk of bias in animal studies is not dependent on the data stream or the question being asked, but on the design of the study. Many instruments that have been developed (including those for evaluating animal toxicology studies) have criteria that have not been shown to bias research outcomes (see Supplemental Material, Table S1, of Krauth et al. 2013).
Furthermore, Table 1 of our paper (Krauth et al. 2013) lists the criteria found in most instruments we identified. In the “Discussion,” we described the empirical evidence supporting the use of some of these criteria and cited the relevant references with the empirical data. By empirical evidence, we mean that a criterion (e.g., randomization) has been shown to be associated with overestimation or underestimation of effect (this could be an efficacy or harm outcome).
Beck et al. note several publications in environmental chemical health hazard assessment [Ågerstrand et al. 2011; Food and Drug Administration (FDA) 2003; Hulzebos et al. 2010; Organisation for Economic Co-operation and Development (OECD) 1998; Schneider et al. 2009; U.S. Environmental Protection Agency (EPA) 1999a, 1999b, 2013]. All of these publications, except OECD (1998), were identified in our search; however, they did not meet the a priori inclusion criteria for our systematic review. As noted in our “Methods” (Krauth et al. 2013), we included the earliest publication of an instrument when it was used in subsequent reports. The article by Ågerstrand et al. (2011) was based on four earlier published papers (i.e., Durda and Preziosi 2000; Hobbs et al. 2005; Klimisch et al. 1997; Schneider et al. 2009). We cited three of these in our review, but excluded Schneider et al. (2009) because it appeared to be a description of software that could be used to operationalize the Klimisch criteria. After reviewing the criteria described by Schneider et al. (2009) in their supplemental file, we found no unique additional criteria that were not already included in our Table 1 and Supplemental Material, Table S1. The reports from the U.S. EPA (1999a, 1999b) and FDA (2003) were neither indexed in Medline nor found in screening of bibliographies. In addition, U.S. EPA (2013) was published after we ended our study. Because we did not find the OECD document (OECD 1998), we cannot conclude whether or not it should have been included in our study.
The comment by Beck et al. that the National Toxicology Program is relying on criteria that have not been “transparently empirically tested” is not correct. In our paper (Krauth et al. 2013), we recommended the use of empirically tested criteria and we pointed out criteria that have been shown to be a risk of bias.
We caution against gathering judgments on how to assess study quality and propose that evidence should guide such evaluations. We propose an empirically based approach—as opposed to consensus-based opinion of experts—as this would provide a more unbiased evaluation of the data.
Ågerstrand M, Breitholtz M, Ruden C. 2011. Comparison of four different methods for reliability evaluation of ecotoxicity data: a case study of non-standard test data used in environmental risk assessments of pharmaceutical substances. Environ Sci Eur 23:17; doi: 10.1186/2190-4715-23-17.
FDA (Food and Drug Administration). 2003. General Guidelines for Designing and Conducting Toxicity Studies. In: Guidance for Industry and Other Stakeholders, Toxicological Principles for the Safety Assessment of Food Ingredients, Redbook 2000. Available: http://www.fda.gov/Food/GuidanceRegulation/GuidanceDocumentsRegulatoryInformation/IngredientsAdditivesGRASPackaging/ucm078315.htm [accessed 15 October 2013].
Krauth D, Woodruff TJ, Bero L. 2013. Instruments for assessing risk of bias and other methodological criteria of published animal studies: a systematic review. Environ Health Perspect 121:985–992; doi: 10.1289/ehp.1206389.
OECD (Organisation for Economic Co-operation and Development). 1998. OECD Series on Principles of Good Laboratory Practice and Compliance Monitoring, No 1: OECD Principles on Good Laboratory Practice. ENV/MC/CHEM(98)17. Paris:OECD. Available: http://search.oecd.org/officialdocuments/displaydocumentpdf/?doclanguage=en&cote=env/mc/chem(98)17 [accessed 13 February 2014].
Schneider K, Schwarz M, Burkholder I, Kopp-Schneider A, Edler L, Kinsner-Ovaskainen A, et al. 2009. “ToxRTool”, a new tool to assess the reliability of toxicological data. Toxicol Lett 189(2):138–144.
U.S. EPA (U.S. Environmental Protection Agency). 1999a. Auditing General Toxicology Studies. Available: http://www.epa.gov/compliance/resources/policies/monitoring/fifra/sop/glp-da-09.pdf [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 1999b. Determining the Adequacy of Existing Data. Available: http://www.epa.gov/hpv/pubs/general/datadfin.htm [accessed 15 October 2013].
U.S. EPA (U.S. Environmental Protection Agency). 2013. OCSPP Harmonized Test Guidelines. Series 870: Health Effects Test Guidelines. Available: http://www.epa.gov/ocspp/pubs/frs/publications/Test_Guidelines/series870.htm [accessed 15 October 2013].
New Impact Factor
We are pleased to announce our new impact factor of 7.03 and—for the first time—our inclusion in the category of Toxicology. Thank you to our authors and readers, who continue to make EHP a leading journal in Environmental Sciences, in Public, Environmental, and Occupational Health, and now in Toxicology.
EHP News Feature Wins Award
Our March 2013 cover story, “A Tale of Two Forests: Addressing Postnuclear Radiation at Chernobyl and Fukushima,” has won first place in the Outstanding Feature Story category of the Society of Environmental Journalists Awards for Reporting on the Environment. The judges called the story “an exceedingly vital and original piece of environmental feature writing.” Congratulations to authors Winifred A. Bird and Jane Braxton Little on their exciting achievement!
CEHN August 2014 Article of the Month
“Gestational Exposure to Endocrine-Disrupting Chemicals and Reciprocal Social, Repetitive and Stereotypic Behaviors in 4- and 5-Year-Old Children: The HOME Study” (Environ Health Perspect; DOI:10.1289/ehp.1307261) has been selected by the Children’s Environmental Health Network (CEHN) as its August 2014 Article of the Month. These CEHN summaries discuss the potential policy implications of current children’s environmental health research.
Register Today for PPTOX IVEnvironmental Stressors in Disease and Implications for Human Health
October 26–29, 2014
Boston Marriott Long Wharf Hotel
Join the Endocrine Society for the fourth international summit of Prenatal Programming and Toxicity (PPTOX), a premiere conference series dedicated to cutting-edge discussion of environmental hazards during early life and long-term consequences.Key deadlines:
* Abstract submission: Tuesday, September 2
* Housing deadline: Monday, October 6
For more information, visit the Endocrine Society website.
Sign Up to Receive E-mail Alerts
Recent Advance Publications
- Risks and Benefits of Green Spaces for Children: A Cross-Sectional Study of Associations with Sedentary Behavior, Obesity, Asthma, and Allergy
- Genistein Disrupts Glucocorticoid Receptor Signaling in Human Uterine Endometrial Ishikawa Cells
- Modification of the Association between PM10 and Lung Function Decline by Cadherin 13 Polymorphisms in the SAPALDIA Cohort: A Genome-Wide Interaction Analysis
- Mitochondria, Energetics, Epigenetics, and Cellular Responses to Stress
- Incense Use and Cardiovascular Mortality among Chinese in Singapore: The Singapore Chinese Health Study
- Perfluorochemicals and Human Semen Quality: The LIFE Study
- Molecular Signaling Network Motifs Provide a Mechanistic Basis for Cellular Threshold Responses
- EHPNoonNews: EPA to clean up Houston hazardous waste site http://t.co/2F9hKH0wFT @houstonchron
- EHPNoonNews: Common epigenomic fingerprint found in multiple cancers http://t.co/2o9Q3QcMWC @genbio
- EHPNoonNews: Cookstoves, highways linked to high blood pressure in China http://t.co/Vv0hb9CyBB @timesofindia
- EHPNoonNews: WHO recommendations on e-cigarettes http://t.co/V9l4U59Nem @bbcworld Related coming in Sept EHP News
- AdvPubl: Risks and benefits of green spaces for children http://t.co/T9E8166bza