Economics An analysis · Goodhart's Law

Getting What You Pay For

Reward a number and you get the number, not the thing the number was standing in for. A field guide to incentives that produced the opposite of their intent.

By Emmanuel Awosika
"The best way to increase wolves in America, rabbits in Australia, and snakes in India, is to pay a bounty on their scalps. Then every patriot goes to raising them." Mark Twain, 1907

There is a story, told in every economics seminar, about the British in colonial Delhi. Worried about cobras, they put a bounty on dead ones. People bred cobras for the reward, the officials cancelled the scheme, the breeders released their now worthless snakes, and the city ended up with more cobras than it started with. It is a perfect little fable, and it is almost certainly not true. No record of the event survives. The economist who named the effect built it on an anecdote.

That gap between the famous story and the missing evidence is the whole subject. The mechanism the cobra fable describes is real and everywhere. It has a name, Goodhart's Law: when a measure becomes a target, it stops being a good measure. Reward a proxy and people optimise the proxy, not the goal it was meant to track. The trick, if you want to say something true rather than just repeat the fable, is to find the cases where the numbers actually exist. They are rarer than the stories, and they are more convincing.

The fable's documented twin real data

The cobra story has a real counterpart, and this one comes with records. In 1902, under French rule, Hanoi faced a plague-carrying rat infestation in its new sewers and paid one cent per rat tail. The historian Michael Vann found the daily tally sheets in a colonial archive: about a hundred forms logging rats killed per day from April to July. The count climbed from a few hundred a day into the thousands, reached 15,041 on May 30, and peaked at 20,114 in a single day on June 12. Then it fell away to a few thousand, a few hundred, and on the last page, none.

05,00010,00015,00020,000AprMayJunJulJune 12: 20,114 in one dayMay 30: 15,041nonerats killed per day, 1902 →
Rats killed per day in Hanoi, from the colonial archive tallies, April to July 1902. Labelled points are dated figures from the records. Source: Vann, "Of Rats, Rice, and Race," 2003.

The collapse to nothing looks like success and was the opposite. The rats had not gone. Within three months officials found the bounty was being gamed: catchers clipping tails and freeing the rats to breed, and farms springing up to raise rats for their tails. The tally had stopped counting rats removed and started counting rewards claimed.

Tax the proxy, and watch people avoid it real data

In 1696, England taxed houses by their number of windows, on the reasonable theory that bigger houses had more windows and richer owners. The 1747 schedule set the trap precisely. Below ten windows you owed nothing. At ten, the tax applied to every window at once, so adding a tenth window did not cost you tax on one window, it cost you tax on all ten.

0d90d180d270d360d51015202530clusterclustercluster10th window: 0 → 60d
The 1747 window tax owed, in pence, by number of windows. The vertical jumps at 10, 15 and 20 are the notches. Source: Oates and Schwab, Journal of Economic Perspectives, 2015.

People did the obvious thing: they bricked up windows to sit just under a notch. Because the Ludlow tax rolls survive, this is not a story but a distribution. In a sample of 493 houses, 18.9 percent had exactly 9 windows and only 4.3 percent had 10; 17.8 percent had 14 and just 1.6 percent had 15. The gaming left a fingerprint you can plot.

0%5%10%15%20%8918.9%104.3%131417.8%151.6%18196.5%201%windows in the house (each group spans one notch) →
Share of houses by window count, Ludlow, 1747 to 1757 (493 dwellings), grouped by notch. At each threshold the bar just below towers over the bar just above. Source: Oates and Schwab, JEP 2015.

Reward the metric, and the metric evaporates real data

Three centuries later, Wells Fargo paid its staff on the number of accounts they opened per customer. Staff opened around 3.5 million accounts nobody asked for. When the fraud was settled in September 2016 and the quota pressure lifted, the tell arrived: legitimate new account openings fell off a cliff, down more than 40 percent year on year within a month. The size of the collapse is a rough measure of how much of the prior growth had been manufactured to hit the target.

5%-10%-25%-40%-55%Sep '16Oct '16Nov/DecJan '17Feb '17Sept 8 settlementChecking accountsCredit-card applications
New account openings at Wells Fargo, change versus a year earlier, after the September 8, 2016 settlement. Nov and Dec were not disclosed in these releases. Source: Wells Fargo monthly disclosures, 2016 to 2017.

The same shape shows up in public services. England's four-hour A&E target held near its 95 percent standard for years, then was missed every month from July 2015 as the system strained. The purest gaming signal there is subtler and needs patient-level data: a spike in discharges in the last minutes before four hours, and a cliff just after, because once the clock is breached there is no further penalty. The target changed what got optimised, which is the moment care stopped being the measure.

Reward the score, and the answers get erased real data

When federal law tied school funding and educators' jobs to test scores, Atlanta produced one of the largest cheating scandals in American education. Teachers and principals erased students' wrong answers and pencilled in the right ones, and it was caught by the fingerprint it left. Every class in Georgia was scored by how far its wrong-to-right erasure rate sat above the state average, in standard deviations. A class more than three deviations out was flagged, a one-in-370 event. Atlanta's flagged classes were not at four or five. They ran into the 20s, 30s and 50s. Investigators found cheating in 44 of the 56 schools they examined, implicated 178 educators, and 11 were convicted at trial in 2015.

031020304050every legitimate classlives here (within 3 SD)flag line, 1 in 370Atlanta's flagged classes: 20 to 50 SD outParks Middle, the extremestandard deviations above the state average →Share of classes flagged for high erasuresExpected by chance0.27%Worst Atlanta school75.4%
Georgia classes by how far their wrong-to-right erasure rate sat above the state average, in standard deviations. Legitimate classes cluster within 3; the flag line marks a 1-in-370 event. Below: by chance, 0.27 percent of classes clear the flag line; at the worst Atlanta school, 75.4 percent did. Source: Governor's Office of Student Achievement; Georgia Bureau of Investigation, 2011.

The cases with no numbers mechanism

Some stories still leave no series to plot, and there the honest move is to show the mechanism rather than invent the data. A US Army post that paid a bounty per feral pig tail in 2007 found the pig population rose, because the scheme raised pig fertility and offspring survival. And in 2025 the shape arrived inside the modern office, when Amazon reportedly began ranking developers by their internal AI-tool usage, aiming for 80 percent of them to use it weekly. Usage is easy to run up. Whether it makes the software any better is the assumption nobody has yet measured.

Amazon's leaderboard, drawn as the risk

what is counted
AI tokens used, weekly usage
rewards →

assumed link ⟿
the actual goal
better, faster software

Emerging, 2025. Reported target, no outcome data yet.

This is the discipline the cobra fable lacks. Where the tax rolls or the bank's own disclosures survive, plot them and let the fingerprint make the argument. Where only a cartoon survives, and the Soviet factory that met its tonnage quota with a single giant nail is exactly that, a 1954 satire rather than a record, say so and keep it as a parable. The mechanism is real in every case. The proof is not, and pretending otherwise turns a true idea into folklore.

Provenance. Real data: the window tax schedule and the Ludlow window-count distribution (Oates and Schwab, JEP 2015); Wells Fargo openings (bank disclosures, 2016 to 2017); the Atlanta erasure analysis (Georgia Bureau of Investigation, 2011); the Hanoi daily rat-kill tallies (Vann, 2003); NHS four-hour performance (NHS England). Mechanism only, no series exists: Fort Benning pigs (2007 to 2008); Amazon leaderboard (reporting, 2025). Kept as parable, not evidence: the cobra bounty and the Soviet giant-nail cartoon.
← Back to Damned Statistics Part of 100X Research · 100xresearch.org