Evaluation is about determining the quality, value, or importance of something in order to take action. It is underpinned by the systematic collection of information and evidence.
What is evidence? Things to keep in mind:
Who pays for science?
Most scientific research is funded by government, companies doing research and development, and non-profit entities/educational institutions. Because science is attempting to get at some "truth," the source of research funding shouldn't have a significant effect on the outcome of scientific research, right?
Read Industry sponsorship and research outcome Cochrane Database of Systematic Reviews 2017, Issue 2. Art. No.: MR000033).
Peer review
Peer review refers to a process whereby scholarly work (ie, an article) is reviewed and critiqued by experts to ensure it meets some standards of acceptance before it is published.
Does this process make for better science?
Read: More than 10,000 research papers were retracted in 2023 — a new record
Read: Editorial peer review for improving the quality of reports of biomedical studies (Cochrane Database of Systematic Reviews 2007, Issue 2. Art. No.: MR000016).
What gets researched? What gets published? ("Publication bias"); What (or Who) gets funded? Who gets cited?
Studies that report interventions that had no effect are less likely to get published. What does this mean in terms of the state of knowledge on a topic?
Read Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias (PLoS One. 2008 Aug 28;3(8):e3081).
There is also evidence of disparities in grant funding for research:
Read Race, Ethnicity, and NIH Research Awards (Science. 2011 Aug 19;333(6045):1015-9).
as well as disparities in recruitment for trials:
Read Racial Differences in Eligibility and Enrollment in a Smoking Cessation Clinical Trial (Health Psychol. 2011 Jan; 30(1): 40–48).
Not to mention disparities in whose research gets cited:
Read Gender Disparity in Citations in High-Impact Journal Articles (JAMA Network Open 2021;4:e2114509).
Read Non-White scientists appear on fewer editorial boards, spend more time under review, and receive fewer citations (PNAS 2023, 120(13):e2215324120).
Sometimes stuff gets researched because of "scientific inertia":
Read Adequate and anticipatory research on the potential hazards of emerging technologies: a case of myopia and inertia? (J Epidemiol Community Health 2014;68:890-895).
Is it race? or is it racism?
Race is a sociological construct, yet most articles describing racial disparities ascribe them to race, not to racism.
Read NIH must confront the use of race in science (Science 2020;369(6509):1313-1314).
See Critically Appraising for Antiracism: Identifying racial bias in published research: A guide and tool to help evaluate research literature for racism.
Oops! I made a mistake (or ... was it cheating..?)
Occasionally, researchers make mistakes, and sometimes those mistakes affect the conclusions of a published article. Articles may be retracted if the mistake is significant. This is a formal process where the author or journal publishes a statement outlining the error. Sometimes, however, retraction is the result of fraud, plagiarism, or other bad acts.
Read Retraction Watch.
Opinion or fact?
Do the conclusions of the article follow the evidence that's presented? Are opinions or notions posited as facts?
Search "As is well known..." in Google Scholar.
Read A Propaganda Index for Reviewing Problem Framing in Articles and Manuscripts: An Exploratory Study (PLoS One. 2011;6(5):e19516).
CV boosting: Does this study add to the body of knowledge, or is it just something the author is doing to add to an ever-growing list of publications?
(In)significance of a single study: Science is incremental. Beware of any study that's proclaimed to be a "breakthrough."
Read Evidence-Based Public Health (RC Brownson. New York; Oxford: Oxford University Press, 2011).
What's the question?
Compare: "Our intervention worked toward fixing Problem X" vs. "The best intervention(s) for fixing Problem X is/are:..."
What to consider when looking at survey or estimated data:
Adopted from information on the UCSF Family Health Outcomes Project web siteReliability and validity:
(Adopted from Chapter 3, Conducting research literature reviews: from the Internet to paper, by Arlene Fink; Sage, 2010).
Reliable data collection is relatively free from "measurement error":
Validity refers to how well a measure assesses what it claims to measure:
Doubt Is Their Product: How Industry's Assault on Science Threatens Your Health by
"Doubt is our product," a cigarette executive once observed, "since it is the best means of competing with the 'body of fact' that exists in the minds of the general public. It is also the means of establishing a controversy." In this eye-opening expose, David Michaels reveals how the tobacco industry's duplicitous tactics spawned a multimillion dollar industry that is dismantling public health safeguards. Product defense consultants, he argues, have increasingly skewed the scientific literature, manufactured andmagnified scientific uncertainty, and influenced policy decisions to the advantage of polluters and the manufacturers of dangerous products. To keep the public confused about the hazards posed by global warming, second-hand smoke, asbestos, lead, plastics, and many other toxic materials, industryexecutives have hired unscrupulous scientists and lobbyists to dispute scientific evidence about health risks. In doing so, they have not only delayed action on specific hazards, but they have constructed barriers to make it harder for lawmakers, government agencies, and courts to respond to futurethreats. The Orwellian strategy of dismissing research conducted by the scientific community as "junk science" and elevating science conducted by product defense specialists to "sound science" status also creates confusion about the very nature of scientific inquiry and undermines the public'sconfidence in science's ability to address public health and environmental concerns Such reckless practices have long existed, but Michaels argues that the Bush administration deepened the dysfunction by virtually handing over regulatory agencies to the very corporate powers whose products andbehavior they are charged with overseeing. In Doubt Is Their Product Michaels proves, beyond a doubt, that our regulatory system has been broken. He offers concrete, workable suggestions for how it can be restored by taking the politics out of science and ensuring that concern for public safety, rather than private profits, guides ourregulatory policy.
Rescuing Science from Politics: Regulation and the Distortion of Scientific Research by
Rescuing Science from Politics debuts chapters by the nation's leading academics in law, science, and philosophy who explore ways that the law can be abused by special interests to intrude on the way scientists conduct research. The high stakes and adversarial features of regulation create the worst possible climate for the honest production and use of science especially by those who will ultimately bear the cost of the resulting regulatory standards. Yet an in-depth exploration of the ways in which dominant interest groups distort the available science to support their positions has received little attention in the academic or popular literature. The book begins by establishing non-controversial principles of good scientific practice. These principles then serve as the benchmark against which each chapter author compares how science is misused in a specific regulatory setting and assist in isolating problems in the integration of science by the regulatory process.
Evaluating Research in Academic Journals: a practical guide to realistic evaluation by
Evaluating Research in Academic Journalsis a guide for students who are learning how to evaluate reports of empirical research published in academic journals. It breaks down the process of evaluating a journal article into easy-to-understand steps, and emphasizes the practical aspects of evaluating research - not just how to apply a list of technical terms from textbooks. The book avoids oversimplification in the evaluation process by describing the nuances that may make an article publishable even when it has serious methodological flaws. Students learn when and why certain types of flaws may be tolerated, and why evaluation should not be performed mechanically. Each chapter is organized around evaluation questions. For each question, there is a concise explanation of how to apply it in the evaluation of research reports. Numerous examples from journals in the social and behavioral sciences illustrate the application of the evaluation questions, and demonstrate actual examples of strong and weak features of published reports. Common-sense models for evaluation combined with a lack of jargon make it possible for students to start evaluating research articles the first week of class. New to this edition New chapters on: evaluating mixed methods research evaluating systematic reviews and meta-analyses program evaluation research Updated chapters and appendices that provide more comprehensive information and recent examples Full new online resources: test bank questions and PowerPoint slides for instructors, and self-test chapter quizzes, further readings and additional journal examples for students.
Finding and Evaluating Evidence: systematic reviews and evidence-based practice by
Evidence-based practice (EBP) promises to have a profound impact on social work practice, education, and scholarship, but adopting EBP depends on the availability and accessibility of evidence and on strategies to synthesize this information. Systematic reviews provide a comprehensive,unbiased method for retrieving and synthesizing relevant research. This pocket guide is a concise introduction that describes the steps required to complete a systematic review and the criteria that can be used to assess the quality of existing reviews.The authors provide straightforward information on how to define a search question that captures the problem's parameters, develop a search strategy that is transparent and comprehensive, assess the quality and credibility of existing research, and summarize the available research to support EBP insocial work. Providing coverage for both quantitative and qualitative synthesis methods, examples illustrate the steps and decisions associated with each approach to research synthesis. Key points and suggested readings and websites provide avenues for further study. This pocket guide is anexcellent introduction to EBP and systematic reviews that will be valued by social work students, practitioners, and scholars.
How to Read a Paper: the basics of evidence-based medicine and healthcare by
Required reading in many medical and healthcare institutions, How to Read a Paper is a clear and wide-ranging introduction to evidence-based medicine and healthcare, helping readers to understand its central principles, critically evaluate published data, and implement the results in practical settings. Author Trisha Greenhalgh guides readers through each fundamental step of inquiry, from searching the literature to assessing methodological quality and appraising statistics. How to Read a Paper addresses the common criticisms of evidence-based healthcare, dispelling many of its myths and misconceptions, while providing a pragmatic framework for testing the validity of healthcare literature. Now in its sixth edition, this informative text includes new and expanded discussions of study bias, political interference in published reports, medical statistics, big data and more. Offers user-friendly guidance on evidence-based healthcare that is applicable to both experienced and novice readers Authored by an internationally recognised practitioner and researcher in evidence-based healthcare and primary care Includes updated references, additional figures, improved checklists and more How to Read a Paper is an ideal resource for healthcare students, practitioners and anyone seeking an accessible introduction to evidence-based healthcare.
