Counterfactual

“The counterfactual is the comparison we need most and can never observe directly. ”

Every causal claim contains an implicit comparison. What happened with the programme is compared with what would have happened without it. The second condition is the counterfactual. It is essential and unobservable. The same farmer cannot experience the same season both with and without training. The same forest cannot be protected and cleared under identical conditions.

Evaluation therefore estimates the missing alternative through design, data and assumptions. A before-and-after comparison is not automatically a counterfactual. Conditions change over time. Prices, rainfall, law, technology and behaviour can improve or worsen independently. The pre-intervention value is a historical baseline; it represents the past, not necessarily the future that would have occurred without action.

Random assignment constructs an expected counterfactual by creating groups similar on average before treatment. Quasi-experimental methods use matched groups, discontinuities, trends or synthetic combinations. Theory-based approaches examine mechanisms and rival explanations where statistical comparison is impossible. The quality of a counterfactual depends on comparability.

Farmers who volunteer for a programme may be more connected, motivated or resourced than non-participants. Comparing outcomes directly can attribute pre-existing differences to the intervention. Matching observed characteristics helps, but unobserved differences may remain. Spillovers complicate comparison. Control communities may learn from participants, share infrastructure or experience market effects.

The intervention then changes the counterfactual group.

This can reduce the measured difference even while overall impact grows. System interventions rarely have a clean no-action condition. A landscape programme may coordinate policies already underway. The relevant counterfactual is not nothing happened, but what the collection of actors and programmes would have produced without the added coordination. Defining the alternative requires contextual knowledge.

Ethics matter. Withholding a potentially beneficial service may be inappropriate, particularly where rights or urgent harm are involved. Designs can use phased rollout, existing thresholds, historical comparison or other approaches without denying essential support.

Counterfactuals are also used in carbon and finance. A project baseline estimates emissions or removals without the project. Small changes in assumed land use, technology or policy can materially change credited volume. Governance should prevent developers from selecting the most favourable plausible scenario. The estimate should evolve only through clear rules.

Updating a counterfactual after results are known can create moving baselines. Yet freezing an implausible scenario can overstate effects. Transparent recalibration is preferable to either opportunism or false permanence. No counterfactual is certain. Confidence depends on design, assumptions, diagnostics and sensitivity.

Reports should explain which population and period the estimate applies to and how alternative plausible counterfactuals affect the conclusion. Counterfactual design also needs to account for interference.

A programme may affect the comparison group through market prices, knowledge spillovers or displaced activity. In that case, the supposedly untreated condition is no longer untouched, and a simple difference may understate or misstate the effect. Landscape and policy interventions often require methods that recognise these wider system responses rather than assuming isolated units. Ethics matter as well.

Random assignment can provide strong evidence, but withholding a proven protection from a high-risk group may be unacceptable. Phased rollout, matched comparisons, discontinuity designs and natural experiments can sometimes create credible evidence without denying essential support. Methodological strength is one criterion; rights, feasibility and the decision at stake are others.

The best counterfactual is the strongest ethical comparison available, not the most prestigious design in the abstract. The discipline is to state the missing world explicitly. What exactly is assumed to happen without intervention, and why? Who or what provides the comparison? Which assumption carries the conclusion? The counterfactual is not a technical appendix. It is the foundation of the causal claim.

Practical application

Define the counterfactual before implementation where possible. Choose comparison methods suited to ethics, scale and context. Test baseline equivalence, trends, spillovers and selection, and use sensitivity analysis for alternative assumptions.

Keep historical baselines distinct from counterfactual scenarios. Document changes and avoid selecting comparators after seeing outcomes. Phrase causal conclusions according to counterfactual strength. Write a counterfactual protocol identifying the intervention population, comparison source, period, assumptions, contamination risks and planned robustness tests.

Explain the design in non-technical language so decision-makers can see the missing-world assumption on which the conclusion rests. Where no credible comparison exists, narrow the claim rather than disguising judgement as measurement. Report balance and overlap diagnostics where relevant, and show how conclusions change when the comparison specification changes.

Robustness is evidence about the counterfactual, not a decorative appendix.

Where spillovers are expected, define whether the estimate concerns direct participants, the wider population or total system effect; each requires a different comparison.

Why it matters

Without a credible counterfactual, organisations can claim ordinary trends, recovery from shocks or other actors' work as programme impact. Resource allocation then rewards presence rather than causal effect.

Common misconception

The pre-programme condition or a non-participant group is often assumed to be the counterfactual. Either can be useful only if it credibly represents what would have happened to the treated population without intervention.

Connections

Baseline provides a reference; Counterfactual provides an alternative causal scenario. Attribution estimates the intervention's effect relative to it. Additionality asks whether outcomes exceed it, while Contribution can be assessed where a single counterfactual is not sufficient.

A question worth asking

What is the most plausible alternative future without your intervention, and what evidence makes it more than a convenient assumption?

Selected references

OECD. 2023. Glossary of Key Terms in Evaluation and Results-Based Management for Sustainable Development, Second Edition. Gertler, P. J. et al. 2016. Impact Evaluation in Practice, Second Edition. World Bank. Rubin, D. B. 1974. Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Shadish, W. R. , Cook, T. D. and Campbell, D. T. 2002.

Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Greenhouse Gas Protocol. 2005. The GHG Protocol for Project Accounting.