Skip to main content

Bridges to Baccalaureate Summer Research Program: Evaluating What You Find

Critical Evaluation

What is evidence?

  • All research is (potentially) "evidence" and there are no "perfect" studies.
  • Is there an agenda (bias)?
    » It's doubtful that any study is totally without some kind of bias, either in the question asked, the study design, or in the author's pre-existing beliefs, not to mention the source of the research funds.
    » How bias in methodology was controlled and the significance of bias in any particular study is what's relevant.

Peer review
Peer review refers to a process whereby a scholarly work (ie, an article) is reviewed and critiqued by experts to ensure it meets some standards of acceptance before it is published. Does this process make for better science?
» Read Editorial peer review for improving the quality of reports of biomedical studies (Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000016)

Who pays for science?
Most scientific research is funded by government grants, companies doing research and development, and non-profit foundations. Because science is attempting to get at some "truth," the source of research funding shouldn't have a significant effect on the outcome of scientific research, right?
» Read Industry sponsorship and research outcome (Cochrane Database Syst Rev. 2012 Dec 12;12:MR000033).

Oops! I made a mistake (or ... was it cheating..?)
Occasionally, scientists make mistakes, and sometimes those mistakes affect the conclusions of a published article. Articles may be retracted if the mistake is significant. This is a formal process where the author or journal publishes a statement outlining the error. Sometimes, however, retraction is the result of fraud, plagiarism, or other bad acts.
» Read Retraction Watch;
» Read The continued use of retracted, invalid scientific literature (JAMA 263 (10):1420-3, 1990)

Reliability and validity
Reliable data collection: relatively free from "measurement error."
  » Is the device used to measure elapsed time in an experiment accurate?
Validity refers to how well a measure assesses what it claims to measure
  » How accurately can this animal study of drug metabolism be extrapolated to humans?

Other things to consider
Do the conclusions of the article follow the evidence that's presented?
Are opinions or notions posited as facts? (Search "As is well known..." in Google Scholar.)
Publication bias: Studies where nothing happened are less likely to get published.
CV boosting: Does this study add to the body of knowledge, or is it just something the author is doing to add to his/her list of publications?
Significance of a single study: Science is incremental. Beware of any study that's proclaimed to be a "breakthrough."

More On Evaluating

Books that can help!

Copyright © 2014-2016 The Regents of the University of California. All rights reserved. Except where otherwise noted, this work is subject to a Creative Commons Attribution-Noncommercial 4.0 License.