Skip to main content
Critically Evaluating the Literature
"History of science teaches us that scientific endeavor has often in the past wasted effort in fields with absolutely no yield of true scientific information."
(Ioannidis, 2005; see handout (.docx))
What is evidence? Things to keep in mind:
- All research is (potentially) "evidence" and there are no "perfect" studies.
- Is there an agenda (bias)?
» It's doubtful that any study of humans is totally without some kind of bias, either in the study design, or in the author's pre-existing beliefs, not to mention the source of the research funds. How bias in methodology was controlled and the significance of bias in any particular study is what's relevant.
- Is qualitative research "evidence"?
» If your goal is to understand beliefs and meanings in the group with whom you are working, then qualitative studies can be important.
Who pays for science?
Most scientific research is funded by government, companies doing research and development, and non-profit entities. Because science is attempting to get at some "truth," the source of research funding shouldn't have a significant effect on the outcome of scientific research, right?
» Read Industry sponsorship and research outcome Cochrane Database Syst Rev. 2012 Dec 12;12:MR000033).
What to consider when looking at survey or estimated data:
- Look at sample sizes and survey response rates - representative of your population? Enough responses to be valid?
- Who was surveyed? - representative of population being compared to? Include group you are interested in?
- Were the survey respondants from heterogeneous groups? Do the survey questions have a similar meaning to members of different groups?
- How was survey conducted? Via telephone? - Many people only have cell phones. Random selection or targeted group?
- What assumptions and methods were used for extrapolating the data?
- Look at definitions of characteristics - Does this match your own definitions?
- When was the data collected?
Reliability and validity
Adopted from Chapter 3, Conducting research literature reviews: from the Internet to paper, by Arlene Fink; Sage, 2014.
Reliable data collection is relatively free from "measurement error"
» Is the survey written at a reading level too high for the people completing it?
» Is the device used to measure elapsed time in an experiment accurate?
Validity refers to how well a measure assesses what it claims to measure
» If the survey is supposed to measure "quality of life," how is that concept defined?
» How accurately can this animal study of drug metabolism be extrapolated to humans?
- Who is the author?
- What else has the author written?
- In which communities and contexts does the author have expertise?
- Does the author represent a particular set of world views?
- Do they represent specific gender, sexual, racial, political, social and/or cultural orientations?
- Do they privilege some sources of authority over others?
- Do they have a formal role in a particular institution (e.g. a professor at Oxford)?
- Why was this source created?
- Does it have an economic value for the author or publisher?
- Is it an educational resource? Persuasive?
- What (research) questions does it attempt to answer?
- Does it strive to be objective?
- Does it fill any other personal, professional, or societal needs?
- Who is the intended audience?
- Is it for scholars?
- Is it for a general audience?
- Where was it published?
- Was it published in a scholarly publication, such as an academic journal?
- Who was the publisher? Was it a university press?
- Was it formally peer-reviewed?
- Does the publication have a particular editorial position?
- Is it generally thought to be a conservative or progressive outlet?
- Is the publication sponsored by any other companies or organizations? Do the sponsors have particular biases?
- Were there any apparent barriers to publication?
- Was it self-published?
- Were there outside editors or reviewers?
- Where, geographically, was it originally published, and in what language?
- In what medium?
- Was it published online or in print? Both?
- Is it a blog post? A YouTube video? A TV episode? An article from a print magazine?
- What does the medium tell you about the intended audience?
- What does the medium tell you about the purpose of the piece?
- How is it relevant to your research?
- Does it analyze the primary sources that you're researching?
- Does it cover the authors or individuals that you're researching, but different primary texts?
- Can you apply the authors' frameworks of analysis to your own research?
- What is the scope of coverage?
- Is it a general overview or an in-depth analysis?
- Does the scope match your own information needs?
- Is the time period and geographic region relevant to your research?
- When was the source first published?
- What version or edition of the source are you consulting?
- Are there differences in editions, such as new introductions or footnotes?
- If the publication is online, when was it last updated?
- What has changed in your field of study since the publication date?
- Are there any published reviews, responses or rebuttals?
- Did they cite their sources?
- If not, do you have any other means to verify the reliability of their claims?
- Who do they cite?
- Is the author affiliated with any of the authors they're citing?
- Are the cited authors part of a particular academic movement or school of thought?
- Look closely at the quotations and paraphrases from other sources:
- Did they appropriately represent the context of their cited sources?
- Did they ignore any important elements from their cited sources?
- Are they cherry-picking facts to support their own arguments?
- Did they appropriately cite ideas that were not their own?
Fact Checking Websites
Project of the Annenberg Public Policy Center of the University of Pennsylvania. Aimed at voters, they monitor TV ads, debates, speeches, interviews, and new releases from major US political figures.
The Fact Checker
Column of the Washington Post by Glenn Kessler. Looks at statements made by political figures and government officials around the world.
Created by the Tampa Bay Times newspaper. The website checks claims made by elected officials, political parties, advocacy groups, and many others.