Main Goal: Causal Forecasting

• Causal forecasting: leverage causal information inside a forecasting model whose objective stays strictly predictive.
• The priority is to minimize total out-of-sample prediction error, with the forecasting algorithms being mostly machine-learning models.

How can causal information help in forecasting?

I currently see two main channels:

• Robustness to a shock in the DGP.

• Preventing the model from ignoring a rare but important event.

Step Goal: Improve forecasts

I started with a more modest step goal:

• Improve the Conflict Forecast model (Mueller & Rauh (2018), (2022a), (2022b), and Mueller et al. (2024)) using Ceasefire Agreements from PA-X data, Bell & Badanjak (2019).

On the causality of ceasefires

Causality vs. prediction

• Explain vs Predict.
  • Breiman (2001) splits statistical practice into data modeling (assume a stochastic model to extract information about nature) and algorithmic modeling (treat the mechanism as unknown, fit any \(f(x)\) and validate by test-set error to predict new responses).
  • Shmueli (2010), (2025) and Carlin & Moreno-Betancur (2025) reorganize this around the question, not the model, where a variable’s admissibility changes with the goal (to explain or to predict). Hence the true-model myth: a less causally faithful model can predict better than a causally correct one.
  • Also Iskhakov et al. (2020), Kleinberg et al. (2015), and Yarkoni & Westfall (2017).
• Significant ≠ predictive.
  • Lo et al. (2015) formalize the gap: significance is a distributional difference (\(f_{D=0}\neq f_{D=1}\)), prediction needs distributional separation (\(\sum_x \min\{f_{D=0},f_{D=1}\}\)). A weak but stable mean shift yields a tiny \(p\)-value, but if distributions overlap ≈90% a classifier can perform poorly.
  • Ward et al. (2010) run this test on canonical civil-war models (Fearon–Laitin, Collier–Hoeffler): highly significant variables sometimes carry very modest AUC contributions out-of-sample.

Other relevant literature

• Causal machine learning. Chernozhukov et al. (2018) is the leading methodology. Their focus is on a valid identification of the treatment effect while controlling for high-dimensional nuisance parameters. Amazing work, but it solves a slightly different problem.

• Difference-in-differences. Callaway & Sant’Anna (2021) is the main DiD tool I used for the analysis at the beginning (the plot in the motivation). Sun & Abraham (2021) is also a very interesting read, warning against the sinful use of two-way fixed-effects regressions with leads and lags. Mueller & Rauh (2024) is the DiD work I partly followed for the motivation plot.

• Matching within DiD. For the intricacies of matching when running a DiD analysis: Daw & Hatfield (2018) and Ham & Miratrix (2023) (matching when parallel trends does not hold).

Setup and notation

• \(\mathbf{Y_{t+h}}\) the target, consists on \(h\)-step ahead violence/fatalities, or aggregate measures of those.
• \(\mathbf{D_t}\) binary treatment, indicating the presence of a ceasefire agreement in period \(t\).
• \(\mathbf{X_t}\) information set, lags of \(Y\), lags of \(D\), news-topic features, etc.
• \(Y\) is forecastable from \(X\): \(\widehat{g}(X_t) \approx Y_{t+h} \forall \, h\geq0\).
• \(D\) is not predictable from \(X\): \(\widehat{f}(X_t) \not\approx D_{t+h}\;\forall \, h\geq 0\).
  • And, in fact, the best possible forecast for \(D\) is the conditional probability \(P(D_{t+h}{=}1 \mid Violence_t)\), with any other model achieving negative \(R^2\) or baseline ROC & PR AUC performance.
• Imbalance:
  • \(P(Y{=}1)\in[0.17,0.30]\) (on incidence tasks, which are binary classification tasks, not regression ones).
  • \(P(D{=}1)\approx 1/85 \approx 0.012\).

The “obvious” modelling approaches

• Add \(D_t\) as a feature to RF / LightGBM / Logistic AR.
  • Binary encoding.
  • Ordinal linearly decaying encoding.
• Give more weight to \(D_t\) in the model training process:
  • Sample weighting: Upweighting the observations with \(D_t=1\) in the loss function of the trees.
  • Feature weight: Upsampling \(D_t\) in the feature space by making it more likely to be selected for each tree.
  • First split forcing: Forcing each tree to use \(D_t\) as the first split.
• Two stage approach:
  • First, predict \(D_{t+h}\) from \(X_t\) (with \(h\geq0\)).
  • Then, use \(\widehat{D}_{t+h}\) as a feature in the main forecasting model.
  • This approach failed early on due to \(D_{t+h}\) being unpredictable by \(X_t\).

The results

In order to have a better insight into these results, I have prepared a dashboard to visualize everything.

Linear forecasting model

For an \(h\)-step outcome at country–month \((i,t)\),

\[ y_{i,t+h} = X'_{it}\,\beta_h \;+\; \tau_h\, d_{it} \;+\; \varepsilon_{i,t+h}, \]

with

• \(\mathbf{y_{i,t+h}}\): \(h\)-step ahead outcome for individual \(i\).
• \(\mathbf{X_{it}}\): information set known at \(t\) (lags of \(Y\), lags of \(D\), news text data).
• \(\mathbf{\beta_h}\): \(h\)-step ahead predictive coefficients on \(X\).
• \(\mathbf{\tau_h}\): \(h\)-step ahead ATT, assumed homogeneous across \((i,t)\).
• \(\mathbf{d_{it}\in\{0,1\}}\): contemporaneous binary treatment.
• \(\mathbf{\varepsilon_{i,t+h}}\): random shock with \(\mathbb{E}[\varepsilon_{i,t+h}] = 0\).

Conditional Mean Independence \(\mathbb{E}[\varepsilon_{i,t+h}\mid X_{it},\,d_{it}]=0.\)

Full-information forecast

Under conditional mean independence, the optimal forecast given the information set and the treatment \((X_{it},d_{it})\) is,

\[ \widehat y^{\,f}_{i,t+h} \;=\; \mathbb{E}[y_{i,t+h}\mid X_{it},d_{it}] \;=\; X'_{it}\beta_h + \tau_h\, d_{it}. \]

Restricted forecast

Now I restrict the forecast to \(X_{it}\) alone and define the population nowcaster of treatment,

\[ \widetilde d(X_{it}) \;:=\; \mathbb{E}[d_{it}\mid X_{it}]. \]

By iterated expectations, conditional mean independence implies \(\mathbb{E}[\varepsilon_{i,t+h}\mid X_{it}] = 0\), so the optimal forecast given \(X_{it}\) is,

\[ \widehat y^{\,r}_{i,t+h} \;=\; \mathbb{E}[y_{i,t+h}\mid X_{it}] \;=\; X'_{it}\beta_h + \tau_h\,\widetilde d(X_{it}). \]

MSE gain from observing \(D\):

The MSE gain from observing \(D\) on top of \(X\):

\[ \Delta\mathrm{MSE}\;:=\;\mathrm{MSE}_r-\mathrm{MSE}_f\;=\;\tau_h^2\,\mathbb{E}\!\bigl[(d_{it}-\widetilde d(X_{it}))^2\bigr]. \]

MSE decomposition analysis

\(\Delta\mathrm{MSE}\) vanishes whenever any of these three components is near zero:

• \(\mathbf{\tau_h^2}\): goes to 0 if the effect is too small (small signal).
• \(\mathbf{p(1-p)}\): quickly drops to 0 when the treatment is imbalanced.
• \(\mathbf{1-R^2_{d|X_{it}}}\): inversely proportional to the predictability of the treatment.

Allowing for OVB

If we drop conditional mean independence, then: \(\tau_h=\tau_h^{\mathrm{causal}}+bias_h\), where \(bias_h\) is the omitted-variable bias. The exact same algebra now uses the projection coefficient:

\[ \Delta\mathrm{MSE}\;=\;\bigl(\tau_h^{\mathrm{causal}}+bias_h\bigr)^{2}\cdot p(1-p)\cdot (1-R^{2}_{d|X_{it}}). \]

Nowcasting vs. Forecasting \(D\)

The decomposition uses \(R^2_{d|X_{it}}\) (nowcast), not \(R^2_{d|X_{i,t-k}}\) (forecast).

Linear regression almost never assigns \(\hat\tau = 0\)

Frisch–Waugh-Lovell residualization gives, in OLS,

\[ \hat\tau_h^{\mathrm{OLS}}\;=\;\frac{\mathrm{Cov}(\widetilde y,\widetilde d)}{\mathrm{Var}(\widetilde d)}. \]

For any non-zero \(\mathrm{Cov}(\widetilde y,\widetilde d)\) (that is, for \(X\) having any predictive capacity towards \(D\), no matter how small) the projection coefficient is not zero.

(Regression) Trees split by variance reduction

At a node \(\mathcal{N}\) holding sample \(S\), a candidate split \((j,th)\) on feature \(X_j\) at threshold \(th\) partitions \(S\) into \(S_L=\{i \in S \mid X_{ij} \leq th\}\) and \(S_R=\{i \in S \mid X_{ij} > th\}\). The impurity for regression trees is:

\[ I(S) \;=\; \frac{1}{|S|}\sum_{i\in S}(Y_i-\bar Y_S)^2 \]

The variance reduction (split-gain) is:

\[ \Delta I(j,th)\;=\;I(S)\;-\;\frac{|S_L|}{|S|}\,I(S_L)\;-\;\frac{|S_R|}{|S|}\,I(S_R) \]

The tree picks the optimal split \((j^*,th^*) = \arg\max_{(j,th) \in J \times \mathcal{T}_j} \Delta I(j,th)\), where \(J \subseteq \{1,\dots,d\}\) represents the available features to evaluate (\(J = \{1,\dots,d\}\) for standard trees, or a random subset for Random Forests), and \(\mathcal{T}_j\) is the set of all possible thresholds for feature \(j\).

What \(D\) competes against at each node

At any given split, the tree evaluates the available features and strictly chooses the one that provides the highest immediate signal (variance reduction).

Feature \(D\) is selected at a node only if:

\[ \Delta I(D, 0.5) \;>\; \max_{X_j, th} \Delta I(X_j, th) \]

(The split-gain of \(D\) must be strictly greater than the best possible split from any other feature \(X_j\).)

Why \(D\) Can Be Completely Ignored

• The “Winner-Takes-All” Rule: If \(D\) is the not the feature with the highest split-gain at every single node, it will never be selected.
• Pruning and Stopping: Trees generally need to use stopping criteria in order to avoid over-fitting. If the tree hits its stopping criteria (like max depth or early pruning) before \(D\) ever manages to be the best choice, it will be left out of the model (tree).
• Forcing splits on \(D\): We can force the trees to split first on \(D\), but in practice this lead to an overall worse performance of the model, in line with what we observed in the simulation.

Setup

States \(S_t\in\{0,1\}\) (peace, conflict). Base transitions:

• \(p_{pp}=0.98,\; p_{pc}=0.02\).
• \(p_{cp}=0.20,\; p_{cc}=0.80\).

Treatment \(D_t\in\{0,1\}\) only available when \(S_t=1\). Effect \(\gamma\):

\[ p_{cp,t}\;=\;\min(p_{cp}+\gamma,\,1), \]

decaying linearly to baseline \(p_{cp}\) over \(\ell\) periods after \(D_t=1\).

Target: \(Y_t=\bigvee_{h=1}^{w}\{S_{t+h}=1\}\), \(w\in\{3,12\}\)

Base run calibrated to \(P(D{=}1)\approx 1/85\).

Sweep results

What happens around treatment

• Before \(t\) (in conflict): our forecast underpredicts \(Y\) (average error \(>0\)).
• At \(t-w\): treatment effect enters the target window and \(Y\) starts falling (on average) toward the forecast.

What happens around treatment II

• What happens in the previous plot should be taken with a grain of salt, as it portrays the average error across all treatment events. The 95% “confidence intervals” of those errors are very wide.

Conclusions

• Ceasefire agreements do not seem to carry enough signal to improve the forecasts of violence, given the current observables.
  • In order to use them, we would need to find a way to detect which ones are more likely to have stronger treatment effects than the current ATT estimates.
  • Getting a better prediction for \(D_{t+h}\) alone could even hurt the performance of our model if it is not accompanied by better detection of more impactful ceasefires.

Next steps

• Develop a systematic framework to help decision-makers rank policies based on the OOS predictive power of their usage of statistically significant variables.

• Develop a new approach through Neural Networks:

  • Embeddings-based news encoder. Develop a (potentially multilingual) contextual encoder which could even be applied to local news at sub-national resolution. This approach could provide a richer text signal, potentially improving \(\widetilde{d}(X_{it})\) and forecasts at small administrative levels.

  • Causally-regularized loss. Add an identification penalty alongside the predictive loss: \[ \mathcal{L}(\theta)\;=\;\underbrace{\mathbb{E}\bigl[\ell\bigl(Y_{i,t+h},\,f_{\theta}(X_{it},D_{it})\bigr)\bigr]}_{\text{predictive}}\;+\;\lambda\,\underbrace{\mathcal{R}_{\text{causal}}\bigl(\widehat\tau_h(\theta);\;\tau_h^{\text{target}}\bigr)}_{\text{identification penalty}} \] In a similar fashion as PINNs (Raissi et al. (2019)), but instead of a physics residual, a causal residual with valid identification properties.

  • Reinforcement-learning policy layer. Decision-maker \(\pi\) chooses actions \(a_t\in\mathcal{A}\) (intervention timing, allocation of limited resources, etc.) given the information set, optimizing a Bellman value: \[ V^{\pi}(X_{it})\;=\;\mathbb{E}_{\pi}\!\bigl[\,r(X_{it},a_t)\;+\;\delta\,V^{\pi}(X_{i,t+1})\;\big|\;X_{it}\,\bigr] \] Importantly, conditional on first establishing that the available data supports a valid offline policy-learning environment. Iskhakov et al. (2020) provide very interesting insights on this topic.