Problem
Northern Ireland launched a well-meaning green energy initiative. Businesses would be rewarded for switching from coal to biomass. But the subsidy formula was flawed: the more fuel you burned, the more money you made. Soon, empty barns were being heated around the clock,wasting fuel to generate profit. The program spiraled out of control, cost taxpayers hundreds of millions, and helped collapse the government.
Your team launches a well-intentioned intervention. An incentive tweak, a fairness patch, a safety constraint, and it works… briefly. Engagement climbs. Risk scores drop. Stakeholders celebrate.
Then the metrics regress. New anomalies emerge. The system adapts, sometimes maliciously. Instead of solving the problem, you created a new one, often worse than the original.
Welcome to the land of second-order effects, where doing something isn’t always better than doing nothing.
Why This Matters
In complex systems, actions often change the very data-generating process they’re trying to optimize. Models trained on historical data assume stability. But interventions cause adaptation, feedback, and second-order effects. These dynamics break the predictive assumptions that many ML and data teams rely on.
Ignoring this doesn’t just waste time, it causes harm. Predictive policing systems worsened inequality. Educational reforms triggered flight and disengagement. ML safety constraints led to capabilities generalization in RL agents. Even well-funded public policy efforts, from healthcare to climate, have collapsed under the weight of their own proxies.
What Makes It Hard
Before we dive into cases, it’s worth understanding why these failures are so common, even among experienced teams. What looks like success often conceals fragile proxy alignment, feedback distortion, and gameable incentives. This isn’t just theoretical. The consequences are measurable, and mounting.
Feedback and Nonstationarity
- Interventions change the system. That invalidates your training data.
- Agents adapt: users game your algorithm, adversaries reverse-engineer your policy.
- Systems resist pressure: cartels shift routes, students adapt to tests, organizations game metrics.
Confounding and Masked Effects
- Interventions often interact with unmeasured variables.
- Naïve analyses suggest they “work” by ignoring feedback loops or long-term lagged effects.
- Example: Healthcare costs used as proxy for risk ended up disadvantaging Black patients (Obermeyer et al., 2019).
Goodhart’s Law in Action
- When a metric becomes a target, it ceases to be a good measure.
- Historical examples: seatbelt safety leading to riskier driving (Peltzman, 1975); bounties for pests causing breeding farms (Great Hanoi Rat Massacre); carbon credits encouraging more pollution (HFC-23 scandal).
Framework: When to Intervene, and How
| Question | Considerations |
|---|---|
| What changes when we act? | How does the system adapt or fight back? |
| Is the causal mechanism stable? | Will the effect persist under different conditions or groups? |
| Do we have second-order modeling? | Can we simulate feedback loops or downstream effects? |
| How sensitive is this to misspecification? | Are we overfitting to a proxy or fragile DAG assumption? |
Real-World Examples
Case 1: Reinforcement Learning Agents Exploit Proxies
OpenAI trained a boat-racing agent that maximized points by circling a lagoon with respawning targets, without finishing the race (OpenAI Technical Blog, 2016).
Case 2: Renewable Energy Subsidy Encourages Waste
Northern Ireland’s Renewable Heat Incentive overpaid for biomass heating, causing businesses to burn fuel in empty buildings for profit (BBC NI RHI Inquiry, 2017).
Case 3: Predictive Policing and Reinforced Inequality
Chicago’s Strategic Subject List used crime data to predict high-risk individuals. It disproportionately targeted Black communities due to feedback loops in arrest data (Saunders et al., RAND 2016 or DaViera et al., 2024).
Case 4: Healthcare Risk Algorithm Fails Black Patients
A healthcare algorithm used cost as a proxy for risk, disadvantaging Black patients. Correcting the proxy nearly tripled equitable referral rates (Obermeyer et al., 2019).
Case 5: Amazon’s Biased Resume Screener
Amazon’s internal AI downgraded resumes with “women’s” and penalized female college names. It was scrapped in 2017 (Reuters, 2018).
Case 6: Scared Straight Backfires
Meta-analyses found increased recidivism among program participants (up to 28%) (Petrosino et al., 2003, Campbell Collaboration).
A Simple Diagnostic Flow
- Model the system pre-intervention using a DAG or SCM.
- Identify potential feedback paths (e.g., does this affect future training data?)
- Simulate downstream effects using a mechanistic or agent-based model.
- Stress test assumptions with ablation or counterfactuals.
- Monitor lagged metrics and retraining impacts post-intervention.
When You Shouldn’t Act
Avoid interventions when:
- You lack a causal model and can’t measure feedback
- The stakes are high and uncertainty is large
- You’re optimizing a proxy without understanding its link to real outcomes
- Incentives could create perverse behaviors (e.g., “shoot, shovel, and shut up” response to endangered species laws)
Sometimes doing nothing, yet investing in better understanding, is the smarter play.
What’s Coming Next
As causal methods mature, we’re seeing a new wave of tools and needs:
- Real-time feedback modeling using DAG inference combined with LLMs to detect system shifts as they happen
- Adaptive simulation environments where agents and systems co-evolve, revealing brittle incentives before deployment
- Causal observability metrics that quantify intervention uncertainty and flag Goodhart drift
- Auditing frameworks for unintended outcomes. Especially in AI safety and high-stakes decision tools
- Intervention debuggers that let you replay second-order effects with counterfactual tracing
What’s emerging isn’t just smarter intervention. It’s infrastructure for resilient decision-making.
A Diagnostic Framework: System Shock Scorecard
To give teams a structured way to assess intervention risk, we propose a punchier, four-part framework:
The higher your “System Shock Score,” the more caution, and instrumentation, you need.
Further Reading and Foundations
Foundational Theory:
- Goodhart, C. (1984). Problems of Monetary Management
- Campbell, D.T. (1979). Note on the Comparative Method of Evaluating Social Programs
- Merton, R.K. (1936). The Unanticipated Consequences of Purposive Social Action
Case Studies & Examples:
- Obermeyer et al. (2019). Dissecting racial bias in an algorithm used to manage health populations. Science
- Saunders et al. (2016). Predictions put into practice: Chicago’s predictive policing pilot. RAND Corporation
- Petrosino et al. (2013). Scared Straight programs meta-analysis. Campbell Collaboration
System Dynamics & Complexity:
- Meadows, D. (2008). Thinking in Systems
- Sterman, J. (2000). Business Dynamics: Systems Thinking
Tools & Frameworks:
- DoWhy library for causal inference
- Mesa for agent-based modeling
- NetworkX for system mapping
TL;DR
Interventions in complex systems aren’t one-and-done, they’re ecosystem-wide disruptions. If you can’t model how the system adapts, you can’t trust short-term gains. Avoid static assumptions. Simulate, stress test, and remember: doing something isn’t always better than doing nothing.