Tag Archives: endogenous-base-rates

Dispatches from the Underground: 3/20/26

Posted on by
Reply

A new recurring feature, starting this week. A few items from the news — sometimes updates to arguments this blog has already made, sometimes new events the existing framework illuminates without quite rising to a full post — followed by a couple of things we’re watching that haven’t ripened yet. Everything here is shorter than a regular post, but hopefully not shorter than it needs to be.

In the News

The DNI and the Negotiator Walk Into a Senate Hearing

This week, Steve Witkoff — President Trump’s envoy to the ongoing conflict with Iran — told reporters he had asked Vladimir Putin whether Russia was providing military support to Iran. Putin said no. Witkoff said he took that at face value.

In the same week, and in the same approximate direction, DNI Tulsi Gabbard testified before the Senate and said she does not take Putin at his word — and offered to elaborate in a classified setting.

This blog has argued that a principal who publicly announces strategic ignorance doesn’t thereby escape accountability: a meta-principal is watching outcomes regardless of what the principal claims to know. Witkoff’s move is that argument’s mirror image. He isn’t strategically ignorant of something he could learn — he’s strategically credulous about something his own government has formally assessed as false. The meta-principal (Congress, the intelligence community, the public) is observing. Gabbard’s very public non-endorsement of Witkoff’s position is itself bilateral accountability in action: the DNI just went on the record, which means Witkoff now has a named co-principal who disagrees with him in an unclassified forum.

The structure is the same. The game is slightly more embarrassing.

The Surveillance System Watching Itself

Speaker Johnson is pushing a clean 18-month extension of Section 702 surveillance authorities before the April 20 deadline, using White House backing to override members of his own caucus — notably the libertarian wing — who are demanding judicial warrants before the government reviews communications involving U.S. citizens.

Two distinct structural stories are running here in parallel. The first is the setter model: the decision about which version of the bill comes to the floor — the clean extension or the warrant-amended one — is itself a policy choice, made before any vote is cast. Much of the most consequential agenda-setting, as we’ve discussed, happens in the determination of what never gets considered. The second is something we’ve been calling the self-referential problem: Section 702 permits the surveillance of foreign persons, but the system that does so collects communications of U.S. persons incidentally. Whether Congress can meaningfully oversee this program partly depends on whether it can inspect a system whose operations it cannot fully observe — and which the program itself affects (the endogenous base rates problem, in a different register, from this post). The rules are watching the rules watching themselves. Johnson’s maneuver doesn’t resolve that problem; it just decides, for another eighteen months, that we won’t look directly at it.

Moving the Debt Doesn’t Move the Problem

The Trump administration announced a three-phase plan to move the federal student loan portfolio — along with FAFSA management — from the Department of Education to Treasury.

The student loan portfolio sits inside Education not because Treasury couldn’t technically service it, but because the loans are coupled to a regulatory ecosystem that Education also runs: borrower defense adjudications, income-driven repayment oversight, accreditation enforcement, institutional eligibility determinations. Moving the asset while leaving the regulatory apparatus behind doesn’t simplify the system. It separates functions that were coupled for reasons that are now invisible to the people doing the separating. The debt moves to Treasury; the disputes, the court orders, the appeals, the eligibility questions, and the unhappy phone calls stay rooted in rules that Education still administers. When the jurisdictional collisions start, and they will, the question of who owns the problem will have a distinctly less legible answer than it does today. See also: conservation of impossibility.

In the Queue

The Bondi Subpoena

House Oversight Chair James Comer issued a subpoena to Attorney General Pam Bondi this week — and then immediately told reporters he “personally doesn’t see any reason for a deposition,” suggesting that members who voted for it were “embarrassed.” The subpoena is, in other words, a signal that is publicly announcing its own insincerity.

This inverts the core logic of “Signaling through Obstruction”: a vote or a procedural move has informational content because it is costly, and that cost is what credibly separates genuine preference intensity from performance. A subpoena accompanied by its author’s own interpretation that it shouldn’t be taken seriously is cheap talk in a subpoena-shaped container. Bondi can — and almost certainly will — treat Comer’s public statements as the authoritative gloss on how to respond.1 We’re watching whether this evaporates, escalates despite Comer’s best efforts, or produces the kind of procedural theater that generates a lot of floor speeches and no depositions.

Is a Prediction Market That Causes Its Predictions Still Predicting?

Arizona this week became the first state to allege that Kalshi — a federally regulated prediction exchange — has committed criminal violations by operating an unlicensed gambling platform. The legal question is interesting, but the structural question underneath it is more so.

When a prediction market shows a candidate at 8% odds, it isn’t passively estimating probability. It’s shaping the behavior of donors, reporters, staffers, and the candidate herself — and it may be triggering the very withdrawal it was predicting.2 Honest and Effective covered the strategic-withdrawal-as-kingmaking problem; Can a Game Know Its Own Rules? covered the general structure of penalties that open games rather than close them. The combination here is: a market that causes the outcomes it forecasts has become something other than a measurement instrument. It’s an agenda-setter with a ticker. We’ll have more to say about this one.

1 For an earlier, blog-level treatment of the same dynamic — forced votes as constituent signals rather than genuine policy moves — see “Make Me an Offer I Can’t Refuse (to Reject).”

2 On classifiers that reshape the populations they classify, see The IRS Is Here to Help. So Is ICE., and the endogenous base rates discussion therein.

Also published today: a follow-up to All Statistics Are Local on the CPI, tariffs, and whose inflation the headline number is actually measuring — Your Basket May Vary.

Your Basket May Vary (or, “The Average Is Not Your Neighbor”)

Posted on by
1

Yesterday’s post argued that a national aggregate can accurately represent no one’s actual experience when the underlying data are structured the right way — or, more precisely, the wrong way. The incarceration example was instructive precisely because it was symmetrically uncomfortable: the same Simpson’s paradox that embarrasses the reform-is-working narrative also embarrasses the cities-are-dangerous one. Nobody was being served by the aggregate. It was just leaking information in both directions simultaneously.

Today’s example has the same formal structure and considerably sharper politics.

The February Consumer Price Index came in at 2.4 percent year-over-year — right in line with expectations, “tame” in the words of one analyst, and a figure that has been used, with some justification, to argue that inflation is under control. Shelter — the single largest component of the index, accounting for roughly a third of its weight — rose 3 percent annually and was the largest contributor to the monthly increase. Gasoline fell about 5.6 percent year-over-year, a meaningful downward pull. Food was up 3.1 percent. Apparel jumped 1.3 percent in February alone, the largest monthly gain since September 2018.

So far so good. The CPI is doing exactly what it is designed to do: aggregating price movements across a basket of goods into a single number. The question worth asking — which the CPI is not designed to answer — is whose basket.

The CPI-U covers “all urban consumers.” Its weights reflect how much the average urban consumer spends in each category, based on expenditure surveys updated annually. This means the index gives heavy weight to gasoline because the average urban consumer spends a meaningful share of income on it. It gives even heavier weight to shelter because housing is a large share of almost everyone’s consumption. When gasoline falls sharply, it pulls the index down in proportion to how much the average consumer spends on gas. When shelter rises moderately, it pushes the index up in proportion to its share of average expenditures.

Here is the problem. “Average” is doing a lot of work in that sentence, and it is not doing it evenly. Gasoline price movements disproportionately affect households that buy a lot of gasoline — which correlates strongly with income, geography, and commute patterns. A drop in gasoline prices is a real benefit to a two-earner suburban household with long commutes and a larger vehicle. It is a smaller benefit, or none at all, to a lower-income urban household that relies on transit. The CPI credits gasoline’s decline to everyone’s inflation rate in proportion to average expenditure. For households whose actual gasoline consumption is well below average, that credit is fictional.

The tariff layer makes this considerably less abstract. The Yale Budget Lab’s most recent analysis finds that the current tariff regime raises prices about three times more for households in the bottom income decile than for households in the top decile, expressed as a share of post-tax income. The average household loses somewhere between $600 and $800 in purchasing power annually under the current regime. For households at the bottom of the distribution, the loss is proportionally larger; for households at the top, smaller.

The CPI will absorb all of this as a modest, broadly distributed price increase — because the CPI is an expenditure-weighted average, and expenditure-weighting assigns more influence to the consumption patterns of households that spend more. In a sufficiently skewed income distribution, the expenditure-weighted index can be simultaneously below the personal inflation rate experienced by the median household and well below the rate experienced by the bottom two quintiles. This is not a measurement error. It is a design choice. The index was designed for monetary policy and Social Security cost-of-living adjustments, not as a comprehensive account of household welfare, and the BLS says so, quietly, in their own documentation.1

The formal point is the same one from yesterday: the aggregate is not wrong. It is accurately computing what it was designed to compute. What it cannot do — given its construction — is tell you whose experience is being tracked. A headline inflation rate of 2.4 percent coexists, right now, with a tariff regime hitting the lowest-income households at roughly three times the rate of the highest-income ones, a food index up over 3 percent, and an energy price shock from the Iran conflict that had not yet registered in February’s numbers and will in March’s.2 The aggregate smooths all of this into a single, manageable-sounding figure.

All statistics are local. The national average is not your neighbor’s experience, and it is increasingly not yours either.

With that, I leave you with this.3

1 The BLS notes explicitly that the CPI-U “is not applicable to all consumers and should not be used to determine relative living costs.” The Chained CPI (C-CPI-U) attempts to account for substitution behavior — when gasoline gets expensive, people drive less; when beef gets expensive, people buy chicken — and is a closer approximation to a cost-of-living index. But it is issued with a time lag, is less widely reported, and rarely makes it into the headline discussion. The index that shapes public perception of inflation is the one that was designed for a different purpose.

2 EY-Parthenon estimates headline CPI will rise to roughly 3.3 percent in March, driven primarily by the energy shock. The February report, as J.P. Morgan’s senior markets economist put it, is “a bit stale at this point.” The gap between the headline number and the lived experience of households at the bottom of the income distribution will widen before it narrows.

3 Dolly Parton, “9 to 5.” The paycheck never seems to stretch as far as the index says it should.

All Statistics Are Local

Posted on by
2

A friend and colleague gave a talk today about incarceration trends in the United States (she knows who she is, and I owe her a coffee). The importance of the problem — and the genuine messiness of the data — got me thinking about how difficult it is to convey the right lessons from debates about prison reform. It was only somewhere on the walk home that I remembered Maggie and I are working on a chapter right now that is almost precisely on point.

The national story about incarceration in the United States sounds, on its face, like genuine progress. The prison and jail population peaked in 2008 at roughly 2.3 million people and has declined since — down to about 1.7 million by 2020, the lowest rate since 1995. Reform advocates can point to these numbers with some justification. The trend is real. The question is whether it is the right trend to be reading.

Here is what the data actually show once you look underneath the aggregate. Urban counties — large jurisdictions housing some of the country’s biggest jails — have led the decarceration trend substantially. These counties have also, because they are very large, dominated any national population-weighted calculation. When you sum everyone in American jails and prisons and divide by total population, the enormous weight of urban America pulls the aggregate down. Rural counties tell a different story. Jail incarceration rates in rural America have risen for decades, and by 2018 the median rural jail admission rate was roughly 80 percent higher than the median urban rate — a near-complete reversal from the pattern that prevailed in the 1990s, when urban jails were the dominant feature of American incarceration. The Vera Institute, which has done the most systematic county-level analysis of these trends, documents what it calls a near-universal urban-to-rural shift in prison admissions across states — and notes that this shift persists regardless of whether a given state’s overall admissions are declining. That last clause deserves a moment’s pause.

What we have are two trends moving in opposite directions, and a national aggregate that accurately reflects neither. This is Simpson’s paradox — the phenomenon in which a trend visible within every subgroup of a dataset disappears or reverses when the groups are combined. The mechanism here is population size. Urban counties are large enough that their declining rates pull the national population-weighted average down, even as the majority of American counties — numerically, geographically — are moving in the opposite direction. The typical American lives in a jurisdiction where incarceration is falling. The typical American county is one where it is rising. Both of these sentences are true. They are true about the same data.1

The paradox is not merely a statistical curiosity. It points toward something deeper about how reform gets measured and, by extension, how it gets held accountable. Tip O’Neill’s observation that all politics is local is usually taken as a practical claim about electoral incentives — voters care about their own circumstances, so politicians had better attend to them. That reading is correct, but it undersells the formal implication. Scholars of legislative behavior (Fenno and Mayhew, especially) showed us that legislators don’t simply respond to local interests among many competing concerns — they organize their entire representational world by geographic proximity, allocating attention and credit-claiming in a hierarchy ordered by how close to home an interest sits. The practical consequence is that the politically relevant signal about prison reform, for most legislators, is not the national aggregate. It is what is happening in their district, their county, their constituent calls. A senator representing a rural state whose counties are experiencing rising incarceration is not going to be moved by a national decarceration trend — not because they are inattentive, but because the signal that reaches them is local and the accountability they face is local. The national aggregate is simply not in their information environment in any politically actionable sense.

This is where O’Neill’s maxim acquires a formal bite that goes beyond the conventional reading. If the relevant unit of democratic accountability for criminal justice is the county — and it largely is, because sheriffs, prosecutors, and jail administrators are all locally elected and locally accountable — then a national aggregate is not merely incomplete. It is measuring outcomes at a level that corresponds to almost no legislator’s actual decision environment. Reform that appears only in national aggregates has not yet reached the accountability structure that would sustain it.

The connection to what political scientists call the Ostrogorski paradox should be clear, and it is not coincidental. Ostrogorski’s observation was that a majority of voters can support a party while opposing a majority of that party’s positions, because majority preference is not transitive across issues. The spatial version of the same problem is this: you can have a national majority for reform — in the sense that most people live in jurisdictions where incarceration is falling — while simultaneously having a majority of the relevant decision-making jurisdictions moving in the opposite direction. The compound majority problem is not just about issue bundling. It is about the geometry of representation itself.

None of this is comfortable for either side of the standard reform debate. The national decarceration trend is real, and the people and organizations who produced it deserve credit. But a reform movement whose gains are concentrated in large urban counties, while rural and small-city jails keep growing largely unnoticed, has not solved the problem it set out to solve — it has solved the part of the problem that was easiest to measure in the way the measurement was already set up to reward. That is a different thing. Distinguishing between the two requires disaggregating the data, sitting with the messiness my colleague’s talk made vivid, and asking what level of aggregation the question actually demands. All politics is local. So is all accountability. The aggregate, as a rule, is neither.

With that, I leave you with this.2

1 This is not the first time this blog has encountered Simpson’s paradox lurking inside an apparently straightforward policy debate. Readers with long memories may recall a 2012 post on how the paradox was embedded in the measurement of Adequate Yearly Progress under No Child Left Behind. The structure is identical; the stakes are higher.

2 Gil Scott-Heron, “The Revolution Will Not Be Televised.” The national aggregate, like television, will not be where the real story is happening.