Statistics: Hypothesis testing, significance, power and choosing a test

📊 Hypothesis testing, significance, power and choosing a test. Trials report p-values and confidence intervals to separate real effects from chance. Doctors need to know what a p-value does and does not mean, why studies miss real effects (type II error and low power), how multiple comparisons mislead, and which test suits which data. MRCP questions typically give a study design and ask for the correct test or the interpretation of a result.
Your four questions for any reported result: What was the null hypothesis? Could this be chance (p-value, CI)? Could the study have missed a real effect (power)? Was the test appropriate for the data and design?

---

Null Hypothesis and p-values

• Null hypothesis (H₀): there is no difference or association (e.g. drug and placebo have the same effect). The alternative hypothesis (H₁) is that there is a difference.
• The p-value is the probability of obtaining a result at least as extreme as that observed if the null hypothesis were true.
• p < 0.05 (the conventional significance level, α) → reject H₀: the result is "statistically significant".
• What p does NOT mean:
  • It is not the probability that the null hypothesis is true.
  • It is not the probability that the result is due to chance.
  • It does not measure the size or importance of an effect.
  • p = 0.06 is not "no effect"; p = 0.04 is not proof of an effect.
• Large samples make tiny, clinically irrelevant differences significant; small samples can miss large, important ones. Always look at the effect size and CI.

Type I and Type II Errors

Decision	H₀ actually true (no real effect)	H₀ actually false (real effect)
Reject H₀ ("significant")	Type I error (α) - false positive	Correct - power (1 − β)
Do not reject H₀	Correct	Type II error (β) - false negative

• α is set in advance (usually 0.05). β is usually set at 0.1–0.2, giving power of 80–90%.
• Memory aid: Type I - "I see something that is not there" (false positive); Type II - "I'm too blind to see what is there" (false negative).
• Type II errors are common in small trials - "absence of evidence is not evidence of absence".

Power and Sample Size

• Power = 1 − β = the probability of detecting a true effect of a specified size.
• Sample size depends on: the significance level (α), the desired power (1 − β), the minimum clinically important difference to detect (smaller differences need much larger samples), and the variability of the outcome (SD) or the baseline event rate (rarer events need larger samples).
• Power increases with: larger sample size, larger true effect, less variability, higher α, one-tailed testing (rarely justified).
• Allow for dropouts - inflate the sample size by the expected loss to follow-up.
• Post-hoc power calculations after a negative trial are uninformative; look at the CI instead.
• A sample size calculation must be reported in trial protocols and papers (CONSORT).

Multiple Comparisons and Data Dredging

• With 20 independent tests at α = 0.05, one "significant" result is expected by chance alone; the chance of at least one false positive is 1 − 0.95²⁰ ≈ 64%.
• Bonferroni correction: divide α by the number of tests (e.g. 10 tests → significance at p < 0.005). Simple but conservative (reduces power).
• Pre-specified primary outcome - the trial is powered for one primary outcome; secondary outcomes are supportive; post-hoc findings are hypothesis-generating only.
• Subgroup analyses - only credible if pre-specified, few in number, biologically plausible, and supported by a significant test for interaction (not just significant in one subgroup and not another). The ISIS-2 trial famously found that aspirin did not seem to help patients born under Gemini or Libra - an illustration of spurious subgroup effects.
• p-hacking / outcome switching - analysing data in many ways until p < 0.05; prevented by trial registration and published protocols.

One-tailed and Two-tailed Tests

• Two-tailed - tests for a difference in either direction; standard in clinical research.
• One-tailed - tests for a difference in one pre-specified direction only; more powerful but justified only when an effect in the other direction is impossible or irrelevant - rarely appropriate in clinical trials (a treatment can always be harmful).

Parametric versus Non-parametric

• Parametric tests (t-tests, ANOVA, Pearson correlation) assume the data (or differences) are approximately normally distributed with similar variances; they use means and SDs and are more powerful when assumptions hold.
• Non-parametric (distribution-free) tests (Mann–Whitney, Wilcoxon, Kruskal–Wallis, Spearman) use ranks; they suit skewed data, small samples, outliers and ordinal data.
• Paired versus unpaired: paired (dependent) data come from the same individuals measured twice (before/after), matched pairs, or the two eyes of one person; unpaired (independent) data come from separate groups.

Which Test? - The Decision Table

Question	Data	Parametric test	Non-parametric equivalent
Compare 2 independent groups	Continuous	Unpaired (independent) t-test	Mann–Whitney U (Wilcoxon rank-sum)
Compare 2 paired measurements	Continuous	Paired t-test	Wilcoxon signed-rank
Compare 3 or more independent groups	Continuous	One-way ANOVA	Kruskal–Wallis
Compare 3 or more repeated measures	Continuous	Repeated-measures ANOVA	Friedman
Compare proportions in independent groups	Categorical	Chi-squared (χ²)	Fisher's exact (small expected counts <5)
Compare paired proportions	Categorical (before/after)	McNemar's test	-
Association between two continuous variables	Continuous	Pearson correlation (r)	Spearman rank correlation (ρ)
Predict a continuous outcome	Mixed	Linear regression	-
Predict a binary outcome	Mixed	Logistic regression (gives odds ratios)	-
Time to event	Survival	Log-rank test; Cox proportional hazards (gives hazard ratios)	-
Agreement between two raters	Categorical	Cohen's kappa	-
Agreement between two methods	Continuous	Bland–Altman	-

Tests for Categorical Data

• Chi-squared test compares observed counts with those expected if there were no association, in a contingency table (e.g. 2 × 2).
• Assumption: expected frequency ≥5 in at least 80% of cells (and none <1); otherwise use Fisher's exact test (common with small samples or rare outcomes).
• Degrees of freedom for a contingency table = (rows − 1) × (columns − 1); a 2 × 2 table has 1.
• Chi-squared for trend - for ordered categories (e.g. dose groups).
• McNemar's test - paired binary data (e.g. the same patients tested positive/negative with two different tests).

Correlation and Regression

• Correlation coefficient (r) ranges from −1 (perfect negative) through 0 (no linear association) to +1 (perfect positive).
• Pearson's r - linear association between normally distributed variables; Spearman's ρ - monotonic association using ranks.
• r² (coefficient of determination) - the proportion of variation in one variable explained by the other (r = 0.7 → r² = 0.49 = 49%).
• Correlation does not imply causation (confounding, reverse causation, chance); it does not measure agreement; outliers distort it; a non-linear relationship (e.g. U-shaped) can give r ≈ 0.
• Linear regression - fits y = a + bx; the slope (b) is the change in outcome per unit change in predictor; multiple regression adjusts for several variables (confounders).
• Logistic regression - binary outcomes; coefficients are exponentiated to give adjusted odds ratios.
• Overfitting - too many predictors for the number of events (rule of thumb: at least 10 events per variable).

Survival Analysis

• Survival (time-to-event) data - time until death, recurrence, admission; some patients are censored (lost to follow-up or event-free at study end).
• Kaplan–Meier - estimates the proportion event-free over time, accounting for censoring; median survival is the time at which the curve crosses 50%.
• Log-rank test - compares survival curves between groups (gives a p-value, no effect size).
• Cox proportional hazards regression - estimates the hazard ratio (HR) adjusted for covariates; assumes the ratio of hazards is constant over time (proportional hazards).
• HR interpretation: HR 0.75 → 25% lower instantaneous risk of the event at any time in the treatment group; HR 1 → no difference.

Superiority, Equivalence and Non-inferiority

• Superiority trial - is A better than B? The null hypothesis is no difference.
• Equivalence trial - are A and B similar within a pre-specified margin (±Δ)? Both CI limits must lie within the margin.
• Non-inferiority trial - is A no worse than B by more than a pre-specified margin (Δ)? The CI for the difference must not cross the non-inferiority margin. Used when the new treatment is cheaper, safer or easier (e.g. DOACs versus warfarin).
• Analysis: non-inferiority should be shown in both intention-to-treat and per-protocol populations (ITT alone can bias towards similarity).
• Margin choice must be clinically justified - a generous margin can declare an inferior treatment "non-inferior".

Frequentist and Bayesian Thinking

• Frequentist statistics (p-values, CIs) ask how likely the data are, given a hypothesis.
• Bayesian statistics combine prior belief with the data to give a posterior probability that a hypothesis is true (e.g. "an 85% probability that the drug reduces mortality by at least 2%"). Increasingly used in adaptive and platform trials (e.g. REMAP-CAP).
• Clinical reasoning is Bayesian: pre-test probability + test result → post-test probability (see the Diagnostic Tests chapter).

Hypothesis Testing at a Glance

Concept	Remember
p-value	Probability of data at least this extreme if H₀ is true
Type I (α)	False positive; usually 5%
Type II (β)	False negative; small trials
Power	1 − β; usually 80–90%; set before the trial
Bonferroni	α ÷ number of tests
Normal data	t-test (2 groups), ANOVA (3+)
Skewed data	Mann–Whitney (unpaired), Wilcoxon (paired), Kruskal–Wallis (3+)
Proportions	Chi-squared; Fisher's if small; McNemar if paired
Survival	Kaplan–Meier, log-rank, Cox (HR)

Spot the Test

• Compare mean systolic BP in 200 patients randomised to drug or placebo. → Unpaired t-test.
• Compare pain scores (0–10) before and after a nerve block in the same 30 patients. → Wilcoxon signed-rank (paired, ordinal).
• Compare the proportion of patients with wound infection in two groups of 15. → Fisher's exact test.
• Compare median CRP across three antibiotic groups. → Kruskal–Wallis.
• Does BMI predict risk of developing diabetes, adjusted for age and sex? → Logistic regression (or Cox if time to diabetes).
• Association between birth weight and later systolic BP (both normally distributed). → Pearson correlation / linear regression.
• Same patients assessed with two tests (positive/negative). → McNemar's test.

Exam-style Scenarios

• Scenario. A trial of 40 patients finds no significant difference in mortality (RR 0.70, 95% CI 0.35–1.40, p = 0.31). The authors conclude the drug is ineffective.
  Answer: Wrong conclusion - the trial is underpowered (risk of type II error). The CI includes both a 65% reduction and a 40% increase; the result is inconclusive.
• Scenario. A paper tests 25 secondary outcomes and reports that one is significant at p = 0.03.
  Answer: This is likely a chance finding (multiple comparisons). With Bonferroni correction the threshold is 0.05 ÷ 25 = 0.002. Treat it as hypothesis-generating.
• Scenario. A non-inferiority trial with a margin of HR 1.30 reports HR 1.05 (95% CI 0.90–1.24). Interpret.
  Answer: The upper CI limit (1.24) is below 1.30, so the new drug is non-inferior; the CI includes 1, so superiority is not shown.
• Scenario. A large registry finds that a statin lowers LDL by an average of 0.02 mmol/L more than a comparator, p < 0.001.
  Answer: Statistically significant because of the huge sample, but clinically trivial - judge clinical importance by the effect size.

Key Points

• The p-value is the probability of the data (or more extreme) if H₀ is true - not the probability that H₀ is true.
• Small trials risk type II errors; look at the CI rather than declaring "no effect".
• Power is set before a study from α, β, the minimum important difference and variability.
• Multiple testing inflates false positives - pre-specify outcomes and be sceptical of subgroups.
• Choose tests by data type, distribution, pairing and number of groups.

Guidelines and Further Reading

• Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ 1995
• Wasserstein RL, Lazar NA. The ASA statement on p-values: context, process, and purpose (2016)
• CONSORT 2025 statement (reporting randomised trials); CONSORT extension for non-inferiority and equivalence trials
• Petrie A, Sabin C. Medical Statistics at a Glance
• ISIS-2 Collaborative Group, Lancet 1988 (astrological subgroup analysis)