How to interpret p-values and confidence intervals in research papers

Many researchers encounter a result that is labelled as statistically significant, only to find that the actual effect is too small to matter in the real world. This disconnect often happens because of a misunderstanding of how we measure uncertainty in data. When reading a paper, it is easy to focus solely on whether a result passed a certain threshold, but the nuance lies in what that threshold actually tells us about the truth.

The logic behind the p-value

A p-value is a way of asking how unusual a set of observations would be if there were actually no effect occurring. In scientific testing, we start with a baseline assumption, often called the null hypothesis, which suggests that any pattern seen in the data is just due to chance. The p-value measures the probability of seeing results at least as extreme as the ones observed, assuming that this null hypothesis is true.

If a p-value is very low, it suggests that the observed data is quite unlikely to have occurred by chance alone under the null hypothesis. This leads researchers to reject that baseline assumption. However, a low p-value does not prove that a theory is correct, nor does it measure the size of an effect. It only tells you that the data is inconsistent with a model where nothing is happening.

A common error is treating a p-value as the probability that the hypothesis is true. It is not. It is a statement about the data in relation to a specific model. A study might produce a very small p-value because the sample size was enormous, even if the actual difference between groups is so tiny that it has no practical utility. This is why looking at the p-value in isolation can be misleading.

Using confidence intervals to see the range

While p-values tell us about the likelihood of chance, confidence intervals provide a range of plausible values for the effect being studied. If a study finds that a new teaching method improves test scores, the confidence interval will give a range, such as an increase of 2 to 8 points. This range represents where the true effect likely sits, based on the data collected.

Confidence intervals are useful because they communicate two things at once: the estimated size of the effect and the precision of that estimate. A narrow interval suggests that the researchers have a very good idea of where the true value lies. A wide interval suggests a great deal of uncertainty, often because the sample size was too small or the data was very inconsistent.

When a confidence interval for a difference between two groups includes zero, it is mathematically equivalent to saying the p-value is above the standard threshold for significance. If the interval is entirely above or below zero, the result is considered statistically significant. However, the interval is often more informative because it shows the scale of the potential impact.

Distinguishing significance from importance

The most important distinction to make when reviewing research is the difference between statistical significance and practical significance. Statistical significance is a mathematical property of the data and the sample size. Practical significance is a judgement about whether the result actually changes how we act or think.

For example, a clinical trial might find that a new medication reduces blood pressure by a statistically significant amount. If the confidence interval shows the reduction is between 0.1 and 0.3 millimetres of mercury, the result is technically significant but medically useless. The effect is too small to improve a patient's health.

Conversely, a small study might show a large improvement in a metric, but because the sample size was small, the confidence interval is massive, ranging from a tiny improvement to a huge one. This result might be practically interesting, but the researchers cannot be certain about the magnitude of the effect. In this case, the lack of statistical significance is a reflection of uncertainty rather than a proof that the treatment does not work.

Evaluating the quality of evidence

To get a clear picture of a study, look for both the p-value and the confidence interval together. The p-value helps you decide if the result is worth taking seriously, while the confidence interval tells you how much that result actually matters and how much you should trust the precision of the estimate.

Always check the sample size reported in the methods section. Large samples make it much easier to achieve low p-values, which can sometimes mask the fact that the effect size is negligible. If you see a very low p-value, immediately look at the effect size and the confidence interval to see if the magnitude of the change is meaningful for your specific field or interest.

Things to try this week

• Look at a recent paper in your field and locate the confidence intervals for the primary outcomes.
• Compare the width of the confidence intervals to see if the study provides a precise estimate or a vague one.
• Check if the most significant findings in a paper actually represent a large enough change to be useful in practice.
• Note whether the authors discuss the limitations of their sample size in relation to the strength of their conclusions.