For education and revision only. Not individual medical advice; check current NICE/BNF and local guidance before clinical use. Clinical safety and terms.
MAKINDO MEDICINE · Statistics & Evidence-Based Medicine203 / 268
Evidence-based medicine: Study design, measures of effect, bias and critical appraisal
📊 Study design, measures of effect, bias and critical appraisal. Evidence-based medicine integrates the best research evidence with clinical expertise and patient values. To use evidence well, you need to recognise study designs, calculate and interpret effect measures (RR, OR, ARR, NNT, HR), spot bias and confounding, and appraise trials, systematic reviews and guidelines. These topics appear in almost every MRCP Part 1 paper and in the UKMLA content map (clinical research and governance). Your four questions for any paper: Is the study design right for the question? Are the results valid (bias, confounding, chance)? What are the results, in absolute terms? Do they apply to my patient?
The EBM Process and PICO
The five steps (5 As):Ask an answerable question → Acquire the evidence → Appraise it → Apply it to the patient → Assess the outcome.
PICO:Population/patient, Intervention (or exposure), Comparison, Outcome (sometimes PICOT - time, or PICOS - study design).
Example: In adults with AF aged over 80 (P), does apixaban (I) compared with no anticoagulation (C) reduce stroke without excess major bleeding (O)?
Sources: pre-appraised resources (NICE, CKS, Cochrane, BMJ Best Practice) before primary databases (PubMed/MEDLINE, Embase).
For questions about treatment effectiveness, the traditional hierarchy (highest first): systematic reviews/meta-analyses of RCTs → individual RCTs → cohort studies → case–control studies → case series/reports → expert opinion.
Randomisation - the only method that balances both known and unknown confounders between groups.
Allocation concealment - the person enrolling patients cannot know or influence the next allocation (e.g. central web-based randomisation); prevents selection bias. Different from blinding.
Blinding (masking) - participants, clinicians, outcome assessors and analysts unaware of allocation; prevents performance and detection bias. Not always possible (surgery) - use blinded outcome assessment and objective outcomes.
Variants: block randomisation (balance numbers), stratified randomisation (balance key factors), cluster RCTs (randomise practices or wards - avoids contamination), crossover trials (each patient receives both treatments in sequence - for stable chronic conditions; needs a washout), factorial trials (two interventions tested at once), pragmatic trials (real-world effectiveness) versus explanatory trials (efficacy under ideal conditions), adaptive/platform trials (e.g. RECOVERY).
Intention-to-treat (ITT) analysis - analyse participants in the groups to which they were randomised regardless of what they received; preserves randomisation and reflects real-world effectiveness. Per-protocol - only those who completed treatment as allocated; can exaggerate benefit, but is needed alongside ITT in non-inferiority trials.
Phases of drug development: phase I (safety, dose, pharmacokinetics in small groups, often healthy volunteers); phase II (efficacy and side effects in patients); phase III (large RCTs versus standard care - licensing); phase IV (post-marketing surveillance).
Reporting: CONSORT (flow diagram, ITT, allocation concealment); registration before recruitment (ISRCTN, ClinicalTrials.gov).
Design: a group defined by exposure is followed over time to compare incidence of outcomes (e.g. smokers versus non-smokers → lung cancer). Prospective (follow forward) or retrospective/historical (using existing records).
Strengths: establishes temporal sequence; can study multiple outcomes from one exposure; calculates incidence and relative risk; good for prognosis and rare exposures.
Weaknesses: expensive and slow; loss to follow-up (attrition bias); confounding; inefficient for rare diseases.
Famous UK cohorts: British Doctors Study (smoking - Doll and Hill), Framingham (US, CVD risk), Whitehall II (social gradient in health), UK Biobank, 1946 birth cohort.
Design: start with people who have the outcome (cases) and without it (controls), and look backwards at past exposure.
Strengths: quick, cheap, efficient for rare diseases and diseases with long latency; can study multiple exposures.
Weaknesses:recall bias (cases remember exposures differently), selection of controls (should come from the same population as cases), cannot calculate incidence or relative risk directly - gives the odds ratio.
Matching controls to cases (age, sex) controls for those factors but they can no longer be studied as risk factors; overmatching reduces efficiency.
Nested case–control - cases and controls drawn from within a cohort (exposure data recorded before the outcome).
Classic example: Doll and Hill's 1950 smoking and lung cancer study; DES and vaginal adenocarcinoma.
Cross-sectional, Ecological, Case Series and Qualitative Studies
Cross-sectional - exposure and outcome measured at one point in time; gives prevalence; cannot establish temporal sequence (chicken-and-egg); used for surveys and diagnostic accuracy studies.
Ecological - unit of analysis is a population or group (e.g. country salt intake versus stroke rates); prone to the ecological fallacy (assuming group associations apply to individuals).
Case report/series - describes unusual cases; generates hypotheses (e.g. thalidomide, early AIDS reports).
Qualitative research - interviews, focus groups, observation; explores experiences, beliefs and reasons; sampling is purposive; analysis is thematic; quality judged by credibility, transferability and reflexivity rather than statistical tests.
Mixed methods combine quantitative and qualitative approaches.
Systematic review - a pre-specified, comprehensive, reproducible search, selection, appraisal and synthesis of all studies on a question (PRISMA reporting; protocol registered on PROSPERO). Narrative reviews are not systematic and are prone to selective citation.
Meta-analysis - statistical pooling of results, weighting studies by precision (inverse variance). Displayed as a forest plot: each study's estimate (square, size proportional to weight) and CI (horizontal line), the pooled estimate (diamond), and the line of no effect (1 for ratios, 0 for differences).
Heterogeneity - variation between study results beyond chance; tested with Cochran's Q and quantified with I² (25% low, 50% moderate, 75% high). High heterogeneity → explore causes; use a random-effects model (assumes true effects vary) rather than fixed-effect.
Publication bias - positive studies are more likely to be published; assessed with a funnel plot (asymmetry suggests missing small negative studies), Egger's test; reduced by trial registration and searching grey literature.
Garbage in, garbage out - a meta-analysis of biased trials gives a precise but biased answer.
Cochrane Collaboration - the leading producer of systematic reviews; its logo is a forest plot of steroids for preterm labour.
Absolute and Relative Measures - RR, ARR, RRR, NNT
Event rates: CER (control event rate) = events in control ÷ control group; EER (experimental event rate) = events in treatment ÷ treatment group.
Measure
Formula
Example (CER 10%, EER 6%)
Relative risk (RR)
EER ÷ CER
0.06 ÷ 0.10 = 0.6
Relative risk reduction (RRR)
(CER − EER) ÷ CER = 1 − RR
40%
Absolute risk reduction (ARR)
CER − EER
4% (0.04)
Number needed to treat (NNT)
1 ÷ ARR
1 ÷ 0.04 = 25
Always report absolute measures - an RRR of 40% sounds impressive whether it reduces risk from 10% to 6% (NNT 25) or from 0.1% to 0.06% (NNT 2,500).
NNT - the number of patients who need to be treated for a stated time to prevent one additional bad outcome; always quote the time period and the comparator. Round up to the next whole number.
NNH (number needed to harm) = 1 ÷ absolute risk increase (ARI) for an adverse effect.
Baseline risk matters: the same RRR produces a larger ARR (smaller NNT) in high-risk patients - the basis of treating by absolute CVD risk (QRISK).
Attributable risk (risk difference) = risk in exposed − risk in unexposed; population attributable fraction - the proportion of disease in the population due to the exposure.
A confounder is associated with both the exposure and the outcome and is not on the causal pathway between them, distorting the apparent association.
Classic example: coffee drinking appears to cause lung cancer - but coffee drinkers are more likely to smoke (smoking is the confounder).
Control at the design stage:randomisation (known and unknown confounders), restriction (e.g. only non-smokers), matching.
Control at the analysis stage: stratification (Mantel–Haenszel), multivariable regression (adjusted ORs/HRs), propensity scores. These only adjust for measured confounders - residual confounding remains in observational studies.
Confounding by indication - in observational studies of treatments, sicker patients are more likely to receive the treatment, making it look harmful (or healthier patients make it look beneficial - healthy user effect, e.g. HRT and coronary disease in observational studies versus the WHI trial).
Effect modification (interaction) - different from confounding: the effect genuinely differs between subgroups (e.g. a drug works only in women); it should be reported, not adjusted away.
Reverse causation - the outcome causes the exposure (e.g. weight loss before cancer diagnosis making low BMI look harmful).
Three questions (CASP): Are the results valid? What are the results? Will the results help locally?
For an RCT: clearly focused question (PICO)? randomised with concealed allocation? groups similar at baseline? blinding? all participants accounted for and analysed by ITT? groups treated equally apart from the intervention? size and precision of effect (absolute measures, CIs)? important outcomes (patient-oriented, not just surrogate)? applicable to my patients (inclusion/exclusion criteria, setting)? benefits worth the harms and costs?
Surrogate outcomes (BP, HbA1c, LDL, bone density, tumour response) do not always predict patient-important outcomes (death, stroke, fracture, quality of life) - e.g. some drugs improve HbA1c or arrhythmia suppression but increase mortality (rosiglitazone, flecainide in CAST).
Composite outcomes - check which components drive the result.
Industry funding and conflicts of interest - consider their influence on design, outcome choice and reporting.
GRADE rates the certainty of evidence for each outcome as high, moderate, low or very low.
RCT evidence starts as high and is downgraded for risk of bias, inconsistency, indirectness, imprecision and publication bias. Observational evidence starts as low and can be upgraded for a large effect, a dose–response gradient, or when all plausible confounding would reduce the effect.
Strength of recommendation (strong or weak/conditional) also reflects the balance of benefits and harms, values and preferences, and resource use.
Declaration of Helsinki and Good Clinical Practice (GCP); research ethics committee approval (Health Research Authority in England).
Informed consent - voluntary, informed, with the right to withdraw without affecting care; special rules for adults lacking capacity (Mental Capacity Act - consultee) and emergency research.
Equipoise - genuine uncertainty about which treatment is better is required to justify randomisation.
Audit versus research versus service evaluation:audit measures practice against a standard (no ethics approval usually); research generates new generalisable knowledge (needs ethics approval); service evaluation describes current service without a standard.
Data protection - UK GDPR, Caldicott principles, the national data opt-out.
Research misconduct - fabrication, falsification, plagiarism; report to the institution and GMC as appropriate.
Scenario. Observational studies showed HRT users had fewer heart attacks, but the Women's Health Initiative RCT showed no benefit and some harm. Why?
Answer: Confounding (healthy user effect and socioeconomic status) - women choosing HRT were healthier. Randomisation balances unknown confounders.
Scenario. An investigator wants to study whether a rare childhood cancer is linked to parental occupational exposures. Which design?
Answer: Case–control study (rare outcome); beware recall bias - use occupational records where possible. Report odds ratios.
Scenario. A drug reduces the relative risk of hip fracture by 50%. In women at 2% 5-year risk and in women at 20% 5-year risk, what are the NNTs?
Answer: ARR 1% → NNT 100 over 5 years; ARR 10% → NNT 10. Same RRR, very different absolute benefit.
Scenario. A forest plot shows 8 trials; the diamond lies left of 1 and does not touch it; I² = 82%.
Answer: The pooled result favours treatment and is statistically significant, but there is substantial heterogeneity - explore the causes (populations, doses, risk of bias) and interpret the pooled estimate cautiously (random-effects model).
Scenario. In a trial, 20% of patients in the treatment arm stopped the drug because of side effects; the authors present a per-protocol analysis showing benefit.
Answer: Per-protocol analysis excludes those who did badly and breaks randomisation - risk of attrition bias. The ITT analysis is the primary result for a superiority trial.