How We Estimate Outcome Probabilities
Our forecasts are probability estimates, not certainties. For each argued case we assign a probability to each Justice's likely vote, using three independent analytical methods, and derive a forecast winner, vote split and confidence from them. When the methods agree, confidence rises; when they disagree, we show the uncertainty. Every forecast is scored against the Court's decision on our accuracy scorecard, which uses only the version saved before the decision.
What's new for OT 2026
What's new for OT2026: base-rate anchoring, open-period question counts, a unanimity default, two sentiment readings (Claude Sonnet 5.5 and Jev) (2026-10-06)
Changes from the OT2025 post-mortem. OT2025 forecasts did not beat simply picking the petitioner (22 vs 23 of 32), were overconfident, and forecast a unanimous decision only once (wrongly) when nine cases were unanimous. The anchoring, question-count and unanimity changes were back-tested before adoption; anchoring now also accounts for recusals; the research on each Justice is wider; our sentiment classifier moves to Claude Sonnet 5.5; and Jev adds a second sentiment reading that is recorded and scored but not used in the forecast.
Two probabilities for every forecast: raw and calibrated.
The raw probability is what our method produces and what the scorecard scores. The calibrated probability pulls it toward the historical petitioner win rate (68%, Supreme Court Database OT2010–24) on the log-odds scale: p = expit(logit 0.68 + 0.4 × (logit p_raw − logit 0.68)). A raw 90% becomes 79%; a call for the respondent stands only if its raw probability is at least 75%.
Why: OT2025 forecasts sat about three times too far from the base rate: calls made at 60–80% confidence came true 59% of the time.
Evidence: Re-fitting the rule without each OT2025 case in turn, it improved our Brier score from 0.217 to 0.213 — but did not beat simply forecasting 68% for every case (0.204). It limits overconfidence; it is not evidence of skill.
Data: forecasts/calibration.json · accuracy/#raw-vs-calibrated
With a recusal, the calibration anchors the full Court and then removes the absent Justice.
When a Justice does not take part, the forecast also estimates the outcome with that Justice voting as their record suggests. That full-bench figure is pulled toward the 68% base rate as usual, and the effect of the absence is then applied on the log-odds scale: logit p = logit anchor(p_full) + (logit p_sitting - logit p_full). The raw probability is still the forecast for the Justices who actually sit.
Why: The base rate averages over absences whose direction is unknown, but a recused Justice's likely vote usually is known. Anchoring only the eight-Justice figure pulls a forecast back toward the petitioner even when the absent Justice was a near-certain petitioner vote. In Suncor v. Boulder County (Alito recused), plain anchoring turned a raw 58% Boulder forecast into a calibrated 58% for Suncor; the recusal-aware figure is 56% Boulder.
Evidence: A reasoned adjustment, not a back-tested one: recusals are too rare for a fit (petitioners won 65.3% of 259 eight-member cases, OT2000-24). It changes only the calibrated figure; the raw probability, which the scorecard scores, is unaffected.
Question counts use the open period of argument only, per lawyer.
Since OT2021 each lawyer's time ends with a round in which the Chief Justice calls on every Justice in turn. We now leave that round out and divide each side's Justice turns by the number of lawyers who argued for it. Per Justice the count becomes a calibrated probability, used only as a modest shift.
Why: Questions in the round carry no signal about votes, and because every lawyer gets a round, the side with more lawyers (usually the side the Solicitor General supports) drew more questions — and usually won. With the round included, 'the side asked more questions loses' was right in 49.7% of OT2021–25 cases.
Evidence: Open-period counts per lawyer, OT2021–25: winner 62.7% of cases (still below 'the petitioner always wins', 68.5%, so never used to pick a winner); individual votes 64.4%; fitted on OT2010–19, the per-Justice probability adds about 4 points over the base rate out of sample.
Data: arguments/elp-phases.json · arguments
Technical cases default to a unanimous forecast.
Every case is framed before it is forecast. A procedural or technical case with no federal party is forecast unanimous unless each predicted dissenter has written or joined an opinion on the same question; federal-party technical cases and hybrid cases carry a minimum probability of unanimity. When two splits are equally likely, the more lopsided one is the headline. The publishing step enforces all of this.
Why: Nine OT2025 cases were unanimous and we called none correctly. Most were technical cases forecast at 5-4 to 7-2. The written rule favoring unanimity in technical cases came after those forecasts, and none of them had been framed.
Evidence: On OT2025 the rule would have lifted exact-split accuracy from 11 to 18 of 32 (an optimistic figure: the rule was designed after seeing those cases). In the Supreme Court Database, OT2015–24, non-constitutional cases were unanimous 48–52% of the time, often enough for the default to pay off in 9 of 10 terms.
Each Justice's pre-Court opinions, confirmation testimony and public statements are part of the per-Justice research.
Alongside their Supreme Court votes and separate opinions, the research on each Justice now searches the 2,572 lower-court opinions the current Justices wrote before joining the Court, their confirmation-hearing answers and year-end reports (only the nominee's own words count), and their tagged speeches and writings. Evidence is ranked: Supreme Court opinion written > joined > pre-Court opinion written > stated views on method > oral argument.
Why: These sources were already on the site (Justice profiles) but the forecasting research did not use them, so a Justice's own prior writing on a doctrine could be missed when the Supreme Court record was thin.
Evidence: A sourcing change rather than a back-tested rule; its effect will show in the OT2026 per-Justice scores.
Data: justices
Our sentiment classifier now runs on Claude Sonnet 5.5.
The classifier is unchanged in every other respect: the same seven categories, prompt and inputs (each Justice turn plus which side's lawyer is being questioned) and the same per-Justice fusion formula. Only the model changes, from Claude Sonnet 4.6 to Claude Sonnet 5.5 (claude-sonnet-5-5); every output file records the model. It feeds forecasts made from OT2026 on. Suncor v. Boulder County and Johnson v. Congress, first forecast with Sonnet 4.6, were switched to Sonnet 5.5 on 2026-10-06, before their decisions (the Sonnet 4.6 runs are kept). Neither forecast changed: Suncor stays Boulder, raw 0.58, 4-4; Johnson stays Johnson, raw 0.53, 5-4. In Johnson, Alito's lean toward the government weakened and Kagan's and Barrett's became neutral, but after review sentiment carried no weight in those Justices' probabilities.
Why: On the same OT2025 cases and inputs, Sonnet 5.5 read individual votes better than Sonnet 4.6.
Evidence: 14 decided OT2025 cases, 122 votes, same side labels for both models: Sonnet 5.5 88 correct (72%), Sonnet 4.6 78 (64%), 'every Justice votes petitioner' 72 (59%); case-level 95% interval for the gain +4 to +12.5 points. Retroactive (run after the decisions) and mostly divided cases. Jev's rich-context reading got 99 of the same 122, but with more context, so the two are not a like-for-like model comparison.
An experiment: TypeSafe's Jev reads every OT2026 argument in parallel; it is scored, not used.
For every OT2026 argued case, each Justice turn of five words or more is sent to Jev (jev-1.13) with the case name, question presented, party names, which party's lawyer is being questioned and that lawyer's previous answer. Jev returns the probability that the Justice doubts that lawyer's position and the probability that the Justice supports it. A Justice's score is the average of doubt minus support, signed so that positive leans petitioner; the lean is its sign. It is published on each case page before the decision, kept out of the forecast, and scored afterwards. The method was fixed on 2026-10-06, before any OT2026 decision. It runs alongside our own classifier (now on Claude Sonnet 5.5); the case page shows both readings side by side. It runs on every argued OT2026 case, whether or not we forecast it, and each case page summarizes Jev's labels: how often each side's lawyer drew doubt or support, the most common label, the most one-sided Justice, the questions it read as most doubtful, and how confident it was. Results are tracked case by case and scored as decisions arrive.
Why: Run after the decisions on 14 OT2025 cases, asking directly whether a Justice doubts or supports the lawyer read individual votes better than our seven-category classifier. Whether that holds on cases not yet decided is the test.
Evidence: OT2025, 14 decided cases, 122 votes: Jev 99 correct (81%) vs our classifier 78 (64%) and 'every Justice votes petitioner' 72 (59%); case-level 95% interval for the gain over our classifier, +10 to +25 points. Optimistic: retroactive, the best of seven variants tried, and 12 of the 14 cases were divided. On the 11 with a pre-decision forecast, our published per-Justice forecasts got 79 of 97 (81%), Jev 81 of 96. Cost about 2 cents per case.
Data: about/#jev · experiments/jev-ot2026.json · forecasts/jev-backtest.json
A trained Jev system joins the experiment: a calibrated probability per Justice, recorded before each decision.
For each argued OT2026 case, Claude Opus 5.5 writes the case's questions (through the API, exactly as in training) from the anonymized opening statements (each side's position, the 2-4 issues with the answer that helps each side, concessions to watch, off-ramps, and the direction the petitioner seeks). Jev then answers about 17 questions on every anonymized exchange (the lawyer's previous answer, the Justice's remark, the reply and the Justice's follow-up), such as doubt, support, friendly question, devil's-advocate testing, own view, pressure for a concession, severity, impatience, limits-testing hypotheticals and the issue lean. Each Justice's answer rates toward each side go into a logistic regression trained on OT2010-2025 votes. It is a third column next to Sonnet 5.5 and the simple Jev reading; it is not used in the forecast. Fixed on 2026-10-07.
Why: The simple Jev reading asked two questions and combined them by hand. Asking more targeted questions about each exchange and learning their weights from past votes gives better-calibrated probabilities.
Evidence: Trained on OT2010-2020, it called 74.1% of OT2021-2024 votes (1,686 votes; ideology baseline 64.1%, question counts 66.4%); refit on OT2010-2024 it called 72.4% of OT2025 votes (427 votes, scored once), log loss 0.541 vs 0.650 for the ideology baseline; its confident calls (probability above 0.75 or below 0.25) were right 88% of the time. Anonymized throughout; content can still identify famous cases, so OT2026 is the clean test.
OT 2025 forecasts are unchanged: they stay as published and are scored on the raw probability saved before each decision.
Type: Historical voting patterns + brief analysis
AI: Claude Opus for argument extraction and matching
Type: AI-classified sentiment per utterance
AI: Claude Sonnet 5.5 for semantic classification (from OT 2026; Sonnet 4.6 before), with Jev as a parallel reading
Type: Computational (no AI)
Citation: J. Legal Studies, vol. 39 (June 2010)
Two Sentiment Readings (Claude Sonnet 5.5 and Jev): How They Are Run
From OT 2026 every argued case gets two independent readings of how each Justice treated each side's lawyer. Both are published side by side on the case page before the decision and scored afterwards.
1. Claude Sonnet 5.5: our sentiment classifier (used in the forecast)
- Input. Every Justice turn of five words or more, with one line saying which side's lawyer is being questioned.
- Question. Claude Sonnet 5.5 assigns probabilities to seven categories: skepticism, hostility, concern, approval, clarifying, engagement, neutral.
- Score. Per Justice, negative tone (skepticism, hostility, concern) toward the respondent's lawyer minus toward the petitioner's, plus positive tone (approval, engagement, clarifying) toward the petitioner's minus the respondent's (Turcian & Stoicu-Tivadar fusion weights). Positive leans petitioner.
- Why Sonnet 5.5. Same prompt, inputs and formula as before; only the model changed (it was Sonnet 4.6). On 14 decided OT 2025 cases (122 votes), Sonnet 5.5 matched 88 votes (72%) against 78 (64%) for Sonnet 4.6.
2. TypeSafe's Jev: a parallel reading (not used in the forecast)
- Input. Every Justice turn of five words or more, with the case name, the question presented, the party names, which party's lawyer the Justice is questioning, and that lawyer's previous answer.
- Question. Jev (version jev-1.13) answers two yes-or-no questions per turn and gives a probability for each: does the Justice express doubt about, or disagreement with, that lawyer's position; and does the Justice express agreement with, or support for, it.
- Score. For each Justice, the average over turns addressed to one side of P(doubt) minus P(support), signed so that a positive score leans petitioner. A score within 0.05 of zero is marked weak.
- Why it is not used yet. Run after the fact on the same 14 OT 2025 cases, it matched 99 of 122 votes (81%), against 72 (59%) for "every Justice votes for the petitioner". That figure is optimistic: the cases had been decided, it was the best of seven variants we tried, and 12 of the 14 were divided. It earns a place in the forecast only if it holds up on cases that had not been decided when it ran. The method was fixed on October 6, 2026.
3. The trained Jev system (not used in the forecast)
A two-step version of Jev. First Claude Opus 5.5 writes questions specific to the case from the anonymized opening statements: each side's position, the issues the case turns on with the answer that helps each side, concessions to watch for, and ways the Court could avoid the main question. Jev then answers about 17 questions on every exchange between a Justice and a lawyer, with names removed: does the Justice doubt or support the lawyer, hand them a friendly question, test them as devil's advocate, state their own view, press for a concession, probe the limits of their rule, and which way the remark leans on each issue. A model trained on how each Justice's answers related to their votes in OT2010–2025 turns them into a probability that the Justice votes for the petitioner. Out of sample it called 72–74% of votes (an ideology baseline: 64–66%), and its confident calls were right 88% of the time.
Scoring. After each decision, every Justice's lean in each reading is compared with the vote, separately from our forecast's own score.
The Jev experiment. Jev runs on every argued OT 2026 case, forecast or not. Each case page summarizes its labels (how often each side's lawyer drew doubt or support, the most common label, the most one-sided Justice, the questions it read as most doubtful, and its own confidence), and the running results are tracked in jev-ot2026.json. If it keeps beating our classifier on cases decided after it ran, we will consider adding it to the method, with its own changelog entry.
Jev costs about 2 cents per case; the Sonnet 5.5 reading a few dollars.
Code: run_fusion_analysis.py, src/research/jev_sentiment.py (--mode rich --publish),
src/research/sentiment_readings.py. Per-case data: /data/cases/<case>/sentiment-readings.json;
OT 2025 test: jev-backtest.json.
Data Sources
| Source | Description | Size |
|---|---|---|
| Spaeth Database (SCDB) | Supreme Court case metadata, justice votes, issue codes (1946–2024, plus OT2025 coded by us) | 9,399 decisions |
| Spaeth Opinions Dataset | Full opinion text linked to SCDB case IDs | 9,334 decisions with opinion text |
| Case Briefs | Petitioner and respondent briefs, replies, supplemental filings | Per case (6 PDFs for Trump v. Cook) |
| Oral Argument Transcripts | Official SCOTUS transcripts with speaker identification | Per case (447 utterances for Trump v. Cook) |
| Oral Argument Audio | Official SCOTUS audio recordings | Per case (~2 hours for Trump v. Cook) |
How Estimates Are Combined
Each methodology produces an independent probability estimate for how each justice will vote. The final integrated estimate weighs all three methods:
- When all three agree: High confidence in the estimate
- When two of three agree: Moderate confidence, with the dissenting method flagged
- When methods disagree: Lower confidence, with detailed explanation of why tone and quantity may diverge
- Technical cases (from OT 2026): forecast unanimous unless each predicted dissenter has written or joined an opinion on the same question
- Two numbers (from OT 2026): the raw probability, and a calibrated one pulled toward the historical rate at which petitioners win (68%), with the anchoring applied to the full Court when a Justice is recused
- Sentiment (from OT 2026) comes from Claude Sonnet 5.5; Jev's parallel reading is shown next to it on each case page but not combined into the forecast (why)
The overall case outcome probability accounts for correlation between swing justices (Roberts, Kavanaugh, and Barrett tend to move together) and the procedural posture of the case.
Continuous Learning
The methodology is not fixed — it is a living system that is recalibrated every time the Court hands down a decision. When a case we tracked is decided, we score our published estimate against the actual result — the winning side, the vote split, and each individual justice — and we post that record openly, hits and misses alike (see the track record).
Each decision is then analyzed for which signals helped and which misled, and recurring patterns become explicit rules that change how the three methods are weighted. For example, from scored cases the model learned that a justice grilling one side hardest is often stress-testing the side it will ultimately favor — so oral-argument question counts, on their own, do not settle the winner. The controlling precedent governs — unless the legal question is genuinely open (no case squarely on point), in which case the oral-argument signal is promoted and given decisive weight. Lessons like this are recorded, versioned, and applied to every subsequent estimate.
Through the 2025 term the system has called the winning side correctly in a majority of the decided cases it tracked (the live tally is on the accuracy page). Every new decision updates both the score and the method — so the estimates you see reflect everything the model has learned to date, not a fixed formula.
Methodology changelog
Every change to how we make or score forecasts, newest first. Forecasts are made, frozen and scored under the method in force when they were published; a change never rewrites an earlier forecast. Also available as JSON.
| Date | Change |
|---|---|
| 2026-10-06 | What's new for OT2026: base-rate anchoring, open-period question counts, a unanimity default, two sentiment readings (Claude Sonnet 5.5 and Jev) Changes from the OT2025 post-mortem. OT2025 forecasts did not beat simply picking the petitioner (22 vs 23 of 32), were overconfident, and forecast a unanimous decision only once (wrongly) when nine cases were unanimous. The anchoring, question-count and unanimity changes were back-tested before adoption; anchoring now also accounts for recusals; the research on each Justice is wider; our sentiment classifier moves to Claude Sonnet 5.5; and Jev adds a second sentiment reading that is recorded and scored but not used in the forecast.
|
| 2026-10-05 | Transcript parser recognizes every Justice since OT2005 The parser knew only the current nine Justices, so in older transcripts Justices Scalia, Kennedy, Ginsburg and Breyer were dropped and their words folded into the lawyer's turn. Fixed; per-side counts now agree with an independent count of the raw transcripts in 94% of OT2010–19 arguments (was 74%). Current-term forecasts were unaffected. |
| 2026-10-03 | Forecast record and a rigorous scorecard Only the version of each forecast saved before the decision is scored, and every version is archived. The scorecard adds baselines (always the petitioner; the historical win rate), Brier score, log loss, calibration, bootstrap intervals and a comparison of the three methods; forecasts saved after a decision are listed and excluded. A diagnostic model of unanimity was added. The finding: on 32 genuine forecasts, no measurable edge over the base rate. |
| 2026-06-19 | Winner gate recalibrated on scored cases After scoring the first blind tests: the sentiment of the Justices' questions is weighted more heavily; the question-pattern composite becomes an input to the winner only at high confidence; and (June 26) the winner is settled from the directly on-point precedent before oral-argument signals are considered, because the side questioned hardest is often the side the Court favors. |
| 2026-06-12 | Multi-agent forecasting pipeline Each forecast is built by a sequence of research steps: frame the actual question and assign a track (procedural-technical, ideological or hybrid); map the controlling precedent; research each Justice's record on it; red-team the draft; synthesize the forecast. A guard refuses to forecast a case already decided. |
| 2026-06-03 | Lessons feedback loop Every decided case is scored and its lessons recorded per case and in a cumulative lessons file that later forecasts must consult. |
| 2026-03-15 | Forecast integrity rules Knowledge from after a decision is removed from per-Justice forecasts, and each case page compares the three methods side by side. |
| 2026-02-08 | Three methods Forecasts combine the sentiment of each Justice's questions at argument (classified by Claude), Epstein–Landes–Posner question counts, and each Justice's voting record in related precedent from the Supreme Court Database; forecasts are presented as probability estimates. |
Academic References
- Epstein, L., Landes, W. M., & Posner, R. A. (2010). Inferring the Winning Party in the Supreme Court from the Pattern of Questioning at Oral Argument. Journal of Legal Studies, 39(2).
- Turcian, A. & Stoicu-Tivadar, L. (2024). Multimodal sentiment analysis using audio and text fusion approach. Applied Sciences.
- Black, R. C., Treul, S. A., Johnson, T. R., & Goldman, J. (2011). Emotions, Oral Arguments, and Supreme Court Decision Making. Journal of Politics, 73(2).
- Spaeth, H. J. et al. (2025). The Supreme Court Database. Washington University Law.