Critical Appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value and relevance in a particular context. -- Amanda Burls (University of Oxford)
Trustworthiness asks: How VALID and RELIABLE is the research? Validity is the extent to which you measure what you intend to measure. A study which is sufficiently free from bias has what is known as internal validity. If a study is generalizable it has external validity.
Reliability has to do with consistency of results. Would you get the same results each time you ran the study?
Relevance is applicability to your patient.
Critical Appraisal is an evaluation of how close the research findings approximate the truth.
SECONDARY LITERATURE analyzes, synopsizes, and synthesizes the primary literature.
The advantage of secondary literature is that the information has already been appraised by others. You may need to critically appraise the study, and there are also critical appraisal checklists to help with appraisal of systematic reviews. GRADE (provided by Cochrane) is a systematic approach to rating the certainty of evidence in systematic reviews and other evidence syntheses.
PRIMARY LITERATURE reports original scientific research.
If you have not been able to find valid & reliable, pre-appraised information then you will need to appraise the primary literature. Below is an outline of the different sections of a research paper where you will look for answers. Use a critical appraisal checklist to help with this process.
Look at the introduction or methods section to discover a sentence which states the question investigated in the paper. Does it address all the elements of your question, or only portions?
The methods section should allow you to find the publication type, a description of the patients included and excluded from the study, and the number of patients studied. The greater the similarity between your patient and the patients studied, the more likely the results apply to your patient. The greater the number of patients studied, the more likely the results of the study will be significant.
The results section of the study should provide numbers so you may make some simple calculations. The calculations will add to your conclusions about the validity of the results of the study.
The discussion may contain notes about the limitations of the study, interpretation of the results of the study, ideas about areas of future study.
There are two forms of error that can cause the results of a study to be false. Random Error is analogous to the idea that if you flip a coin 10 times you may likely end up with a result that is not 5 heads and 5 tails. If you flip a coin 1,000 times the results are likely to be very close to 500/500. Systematic Error (also called Bias) is due to some flaw in the design, implementation, or analysis of a study. This is why carefully reading the methods section and understanding the likelihood of various biases in some study designs is important.
See also: Greenhalgh, Trisha. How to Read a Paper: The Basics of Evidence-Based Medicine. Fifth edition., John Wiley & Sons, Incorporated, 2014.
Levels of evidence & evidence pyramids provide an algorithm to assess which study designs are most appropriate to answer specific questions. Experimental studies (RCTs) are near the top of the pyramid because the study methodology (randomization and blinding) reduces the potential for bias. However, relying on a hierarchy of study design without critical appraisal of the study's methodology and design quality can be misleading.
Critical appraisal will determine whether the evidence is as close to the truth as we can get, or if further research or refinements to study design are needed.
Note: A poorly executed and biased systematic review is not better than a good randomized trial or cohort study!
Critical appraisal checklists provide a consistent methodology for assessing articles. When assessing an article for a journal club, or otherwise, these checklists should be in your tool box of favorites.
Other Considerations:
The Critical Appraisal Checklists and Tools are centered on the various study design types (cohort, case-controlled, RCTs, Systematic Reviews, etc.) and address the possible biases or flaws inherent in each.
Pre-appraised summary evidence does not have an appraisal checklist per se, but depends on the transparency of the working groups or editors to adequately describe the methodology used to assess the evidence, and your own knowledge of the hierarchies of evidence cited.
If practice recommendations henge upon a single RCT that has not been explicitly appraised or graded, you may wish to further explore and appraise that evidence yourself.
Now that you have appraised the evidence, continue on to Step 4: Apply the evidence to the care of your patient; & Step 5: Assess the effectiveness of care based on this evidence.