Hamilton (1989) — A New Approach to the Analysis of Nonstationary Time Series and the Business Cycle
Citation. Hamilton, J. D. (1989). A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle. Econometrica, 57(2), 357–384.
One-line takeaway. Model a series as generated by parameters that switch between a few hidden states following a Markov chain, estimate everything by maximum likelihood, and infer the state probabilities with a forward filter — the template for essentially all quantitative regime analysis since.
What the paper claims
An observed series — Hamilton uses US real GNP growth — is generated by a model whose mean (and, in extensions, variance and dynamics) switches between a small number of unobserved states, where the state s_t follows a first-order Markov chain with fixed transition probabilities. Crucially, nothing is labelled: the number of states aside, the per-state parameters, the transition matrix, and the sequence of state probabilities are all inferred from the data. Applied to GNP, the two-state version recovers an “expansion” and a “contraction” state whose inferred timing lines up with NBER recession dates it was never shown.
How they show it
The engine is the Hamilton filter — a forward recursion that carries the probability of each state, predicts it one step through the transition matrix, and reweights by how well each state’s model explains the new observation — combined with maximum-likelihood estimation of the parameters (later standardised as EM). A backward pass yields smoothed state probabilities conditioned on the whole sample. The persuasive exhibit is the smoothed probability of recession plotted against NBER shading.
What I’d push on
- Smoothed ≠ real-time. The famous plots use smoothed probabilities, which condition on the entire sample and therefore contain look-ahead. A system that must act at time t has only the filtered (forward-only) probability, which is noisier and lagged. This is the single most important distinction for applied use, and the one most papers quietly blur — my proposal insists on filtered, point-in-time regime signals for exactly this reason.
- The number of states is assumed, not learned. Two is a modelling choice; too few blurs distinct regimes, too many overfits and invites label instability. Model selection (BIC, cross-validated likelihood) belongs before interpretation.
- Local optima and label switching. The likelihood is multimodal and the state labels are arbitrary, so results are not reproducible without an identification convention (I order states by variance) and sensible starts.
- Linear-Gaussian within a regime. Genuinely fat-tailed or nonlinear within-state behaviour is only approximated; the model buys regime structure, not distributional realism.
- In-sample fit vs out-of-sample value. The in-sample correspondence with NBER dates is compelling, but whether regime-switching forecasts out of sample — or merely fits — is a separate, more contested question, and precisely the one worth testing rather than assuming.
How it connects to the proposal
Hamilton is the direct ancestor of this site’s regime-aware modelling (a 2-state Gaussian HMM) and Markov-switching regression entries, where I re-implemented the Hamilton filter from scratch and verified it against a known model to twelve decimals. The proposal’s departure from Hamilton is threefold: use the filtered (real-time) state, not the smoothed one; condition decisions on it rather than merely describe it; and test whether that conditioning pays — which the Quant Lab does, finding the answer is nuanced (nothing beyond volatility for sizing, but real value for calibration and at transitions).