What is evidence?
All research is (potentially) "evidence" and there are no "perfect" studies.
Critically evaluating what you read will help any unearth biases or methodological shortcomings that may be present.
Things to consider when evaluating research for bias:
- The question being addressed: What kind of research gets funded?
- Publication bias: Research that shows no effect tends not to get published
- Conflict of interest, author affiliation, source(s) of funding: Does the researcher (or the funder) have a vested interest in the outcome?
- Documentation and assumptions: Are all stated "facts" referenced?
- Peer review: Is the article peer-reviewed? Does it matter?
- Authority: Does the researcher have the knowledge to work in this area?
- Significance of a single study: Science is an incremental process; one study rarely "changes everything"
...Consider that how issues are framed is influenced by our assumptions and biases, and also, keep swimming upstream!
Who pays for science? Does it matter? (There is evidence that it does matter)
Research may be funded by:
- Industry/trade groups
- Private foundations/associations
Reliability and validity
Reliable data collection: relatively free from "measurement error:"
- Is the survey written at a reading level too high for the people completing it?
- If I measure something today, then measure it again tomorrow using the same scale, will it vary? Why?
Validity refers to how well a measure assesses what it claims to measure:
- If the survey is supposed to measure "quality of life," how is that concept defined? Is it measurable?
What to consider when looking at survey or estimated data:
- Look at sample sizes and survey response rates - representative of your population? Enough responses to be valid?
- Who was surveyed? - representative of population being compared to? Include group you are interested in?
- Were the survey respondents from heterogeneous groups? Do the survey questions have a similar meaning to members of different groups?
- How was survey conducted? Via telephone? - Many people only have cell phones. Random selection or targeted group?
- What assumptions and methods were used for extrapolating the data?
- Look at definitions of characteristics - Does this match your own definitions?
- When was the data collected?