Measure Twice

Ten rebuilds got this Wins Above Replacement model for Test cricket from a first pass at "runs above a replacement player" to where it stands now. Most of the moves were straightforward improvements. Three were real bugs, caught by checking a result that looked wrong until it wasn't wrong any more, and one of those three was big enough to flip who tops the entire leaderboard. Twelve more were plausible-looking ideas that got built, tested against held-out data, and correctly rejected. One more is still open. Three separate attempts at the same real question, each looking promising before failing to hold up under closer checking, with nothing yet to show for it. All of it is below, in order.

The build, in order

AUC wasn't tracked as a formal held-out metric until v9; earlier versions were checked against known cricket history instead, not a validation score. Higher AUC is better; the current shipped baseline is 0.985.

v1 — the foundation, and where "replacement level" really comes from

The core unit is Runs Above Replacement (RAR), actual output minus what a replacement-level player would have produced in the same conditions. "Replacement level" means the batting and bowling of players with 10 Tests or fewer in their whole career, the players cycling through the fringe of a squad, not the league average. Every performance is also adjusted for the specific bowlers or batters faced (weighted by a rolling, era-adjusted rating of their own quality) and converted to wins using a runs-per-win rate fitted separately for each five-year era, since the draw-rate shift means a given run advantage buys a different amount of winning probability in 2003 than it does in 2023.

v2 — the swing and the miss

The theoretically "correct" version of this model tracks win probability ball by ball (runs, wickets down, balls remaining) rather than using averages at all, the way basketball's most advanced models track win probability possession by possession. I built it, trained it on 1.4 million deliveries, and it beat a naive baseline comfortably.

Then I built the leaderboard, and every great bowler of the era had fallen to the bottom of the table, while fringe part-time bowlers rose to the top.

v3 — using it for stakes, not value

Rather than discard the win-probability model, I narrowed its job: don't use it to price events, use it to measure leverage, how much a wicket would swing the win probability right now, regardless of whether one falls. That's a property of the match state, and match states are far smoother than individual ball outcomes. Multiply v1's already-solid RAR by that leverage, and a century scored under real pressure starts counting for more than the same century in a match already won.

The first real test case was Ben Stokes' unbeaten 135 at Headingley in 2019, with England chasing 359 and the last pair at the crease. Leverage-weighting pushed his match value up, correctly recognising that almost all of it came in the tensest stretch of the match, without the raw win-probability noise problem creeping back in.

v4 — the single-match ceiling problem

A single match can be worth at most one win to a team. So when v3's leverage multiplier started producing individual matches worth more than a full win, that was a real problem, not a rounding curiosity. It implied a player's own contribution exceeded the entire value of the result, with nothing left over for ten teammates. Basketball WAR never runs into this, and the reason was structural, not a missing safety cap. It never converts to wins at the game level in the first place. A season is the unit that gets converted, built from a rate (value per possession) times a volume (minutes played).

Restructured the same way, v4 accumulates leverage-weighted value in runs, which have no natural per-match ceiling, across a full season, and converts to wins exactly once, on that total. No match-level number is ever treated as a bounded share of one win, so there's nothing left to cap.

v5 — the discipline gap, and the patch that outlived its usefulness

Even after all of that, the leaderboard read strangely. Bowling performances dominated the season table almost completely. That felt off, and it was measurable. Per ball of work, bowling was producing 51% more value than batting. Not explained by leverage (nearly identical between disciplines). Not explained by how wickets get priced either, or so it seemed at the time, that specific assumption is exactly what proved wrong, five versions later.

The fix, at the time, was a calibration constant applied to bowling RAR, rescaling it so both disciplines landed on an identical per-ball rate. Rather than being picked once and forgotten, it recomputed itself from the data on every run, which meant it kept quietly re-balancing the two disciplines back to parity through v6, v7, v8, and v9, no matter what else changed underneath it, and that persistence was the real problem. A real, underlying mispricing (see v10) was there the whole time, and this patch was erasing the evidence of it on every single run instead of ever letting it surface. Removed in v10, once the actual cause was found.

One refinement from this stage does still hold. Bowlers are compared to a type-matched replacement, not a pooled pace-and-spin average. A spinner is judged against a replacement-level spinner, a seamer against a replacement-level seamer. That's part of what caught Ashwin's Nagpur spell (see Root Cause's findings section), and it's the direct answer to a fair worry. A spinner running through a raging turner isn't doing what a fast bowler would've done on the same surface, and shouldn't be judged as if they were.

v6 — the wrong tool, then the right one

The idea was simple. Does a player benefit from playing at home? The data says yes, clearly. Batters average more at home than away; bowlers concede fewer runs at home than away. I built a home/away factor off exactly that, and the leaderboard moved in sensible-looking directions.

Then I checked it against a specific player, because a home/away split has an obvious blind spot. "Away" means every country that isn't your own, averaged together, and those countries are not remotely similar. Bumrah is the case that breaks it. He's an Indian seamer, and the assumption going in was that he'd be getting an artificial discount for his home Indian performances, since fast bowling in India has a reputation for being the hard way round.

The fix was to replace the home/away flag with a factor keyed to the specific hosting country and the bowler's own type. Rebuilt this way, Bumrah's home Indian figures come out as his most valuable, not his least, and the country×type factors line up with the general shape of India's reputation for tough pace-bowling conditions. The heatmap in Root Cause is the current version of this reading.

v7 — position, a bug of my own making, and a further refinement

Reconstructed real batting position from the ball-by-ball sequence (who's at the crease first, tracked innings by innings) rather than assuming it, then compared each player to a replacement level specific to their slot. First version used three buckets (opener, middle, everyone from #7 down), and the leaderboard mostly moved the way you'd expect.

Mostly. Mark Boucher's career total jumped dramatically, a large enough move on a single player to stop and check rather than wave through.

Fixed by splitting the tail: opener, middle, a dedicated bucket for #7 specifically, and #8–11 as the true tail. Boucher's career total came back down to a much less dramatic level, a real, modest adjustment instead of an artifact.

The obvious next question was this. If four buckets was an improvement on three, is eleven an improvement on four? Built it properly, exact position 1 through 11 for every innings, each with its own era-specific replacement bar. The position factors came out exactly as cricket would predict, close to neutral at the top of the order, peaking at #4, and falling steadily to #11. #4 isn't the hardest place to bat, it's the easiest, since it's where the best batter in most sides gets to build an innings against an older ball, with a platform already in place. That's part of why it carries the highest bar, though not the whole reason. The factor is built from what everyone who has ever batted at #4 really scored there, and teams don't send a random player to #4, they send their best one. So some of that reading is truly easier conditions, and some of it is just that the sample at #4 is disproportionately good players. Either way, the effect on the bar is the same. A good batter is expected to cash in at #4, so the model correctly demands more production before crediting anyone there with value above replacement. This version, eleven individual positions, is what the batting side of the model uses.

v8 — the bowling pitch factor was answering two questions with one number

The clearest single example was Danish Kaneria's 5 for 205 against Sri Lanka at Karachi in 2009. On inspection, the value the model was assigning it looked wrong, and it was, which is what sent me looking at truncated and rain-affected matches as a class, rather than treating it as a one-off. Pulling every drawn Test where fewer than 20 wickets fell across the whole match, then comparing the model's implied pitch difficulty against what every bowler who played that match had conceded, showed the same pattern over and over. The model's implied replacement economy was routinely two to three times the real scoring rate of the match. Not a one-off rounding gap, but a systematic mismatch across dozens of matches. That's a sign the number means something other than what it's supposed to.

Fixed by decomposing the single ratio into two independently-shrunk per-ball rates, an economy factor (runs conceded per ball vs era norm) and a wicket-rate factor (wickets per ball vs era norm), each shrunk on balls bowled rather than wickets taken, since balls stay stable through a truncated match even when wickets don't. Validated against a held-out prediction check before trusting it. On a weighted log-error basis, the decomposed version predicts match outcomes noticeably better than the old combined ratio did. Rechecked individually, the truncated matches that had shown the worst mismatch came back into line with their actual scoring rates, and the bowlers who played them kept the credit they'd earned.

v9 — a weighting mismatch that had been inflating bowling since v1

Checking v8's fix meant building a proper predictive validation for the first time in this whole project. Instead of eyeballing whether a number matches known cricket history, hold out a slice of matches, fit everything else on the rest, and see whether a given adjustment predicts the held-out data better than not having it. Tested against opposition quality, the adjustment that's been in the model since v1, the result was uncomfortable. Opposition quality alone made predictions worse than ignoring the opposition entirely, 7.4% worse on a weighted error basis. A shuffled, meaningless version of the same factor did even worse, so it wasn't pure noise. But it wasn't earning its place either.

Fixed by rebuilding the league baseline to use the exact same balls-weighted methodology as the opposition-quality numerator, so the two sides of the ratio are finally measuring the same thing. Once fixed, that same held-out test flipped cleanly. Opposition quality alone went from making predictions 7.4% worse to 3.7% better, and opposition plus country conditions together went from a 4.7% improvement to a 12.6% improvement, nearly triple.

Two more checks came out of the same push, both worth naming even though neither changed the model. Batting's equivalent pitch factor turned out to already be using the truncation-adaptive shrinkage built for v8, since it shares the same underlying calculation and doesn't have bowling's specific ratio-conflation problem, so nothing to fix there. And a two-way check on batting position (does netting player quality out of the #4 premium change anything) found a striking-looking result in-sample. Almost the entire #4 premium collapsed once you controlled for who bats there. It then failed a proper held-out test. The old, selection-contaminated position factor predicts held-out performance slightly better than the "corrected" one, because this model is built around replacement level, and teams essentially never put a true replacement player at #4. The pooled reading, contamination and all, turns out to be the more honest proxy for that slot. Tested properly, and correctly left alone.

v10 — a wicket wasn't worth what the model thought it was

This was supposed to be a smaller check. There'd been a query about whether the model was simply bowler-weighted by nature, since more players bat in an innings than bowl at all, so a fixed pool of value naturally concentrates among fewer bowlers. Checking that specific mechanism directly, it held up completely. Batters and bowlers split an equal-sized pool of value under the old calibration (by construction), and the higher average bowling total per player tracked almost exactly with the smaller number of players who bowl (headcount ratio 1.375, mean-value ratio 1.402). Correctly implemented, not a bug. That should have been the end of it.

But a second, independent check was still available. The win-probability model from v2, abandoned for individual rankings but never tested as an aggregate pricing tool. Its own replacement-level-adjusted totals (batting vs bowling, each netted against its own discipline's fringe-player baseline, not a shared one) put batting at roughly 3.6× bowling's aggregate value, a completely different picture from the current model's forced 50/50. Tempting, but a dead end. Rechecking which individual bowlers that measure rated highly showed the same failure v2 was abandoned for in the first place. Lyon, Anderson and Starc all came out with bowling value at or below zero, an absurd result driven by elite bowlers being deployed at exactly the moments a wicket is already expected, so a "surprise"-sensitive model systematically undercredits them. The aggregate total inherits that same contamination. Not usable.

Re-pricing wickets at their validated ~33-run value and rerunning the pipeline changed almost nothing at first glance. The v5 calibration constant just quietly absorbed the difference, rescaling upward to force the two disciplines back to equal totals regardless of the more accurate wicket price underneath. That was the actual discovery. v5's calibration wasn't correcting an imbalance; it was hiding one, on every single run, for five versions. Batting and bowling RAR were never guaranteed to land on equal totals in the first place. They're each measured against their own discipline's replacement baseline, and there's no law of cricket that says those two gaps have to match. Forcing them to match was a choice made under the assumption that the pre-calibration imbalance was necessarily a bug. Once the actual bug (the wicket price) was found and fixed, that assumption no longer had anything to hide behind.

Dropped the v5 calibration step entirely and let the corrected wicket price set the ratio on its own. Batting came out ahead of bowling by roughly 1.83× in aggregate WAR, the number behind the 64.7/35.3 split in Root Cause, and the reason Joe Root leads that leaderboard outright rather than Nathan Lyon. The rest of the pipeline's validated checks (opposition and country predictive power) were rerun afterward and came back unchanged, confirming nothing else in the model broke on the way through.

None of that, on its own, proves the new split is closer to the truth. A wicket price that's individually well-measured could still combine badly with everything else in the model. So the last check was the most direct one available. For every decisive Test in the dataset, sum each team's batting and bowling RAR and see whether the gap between the two teams predicts who won. Fit on a random 70% of matches, evaluated on the held-out 30%, the old calibrated model already did well at this (AUC 0.982); team RAR is, unsurprisingly, a strong signal of match outcomes either way. The new, repriced and uncalibrated model did better. AUC 0.985, log-loss 0.154 against the old model's 0.175, accuracy 94.3% against 91.5%. Repeated across 200 random resamples of the split, the new model beat the old one on held-out accuracy in all 200. That's the strongest evidence in this whole project that a specific fix made the model better, not just different.

Tested and rejected: home-only country conditions

A follow-up question from the Anderson-vs-Steyn debate in Root Cause. South Africa's pace-friendly country factor is what holds Steyn's number down. Is that factor really measuring conditions, or is it partly measuring how good South Africa's own bowlers happen to be? Home bowlers outperform touring ones in every country by some margin, but checking directly, South Africa's gap is the largest of any checked, by a clear margin.

And touring pace bowlers in South Africa take wickets at essentially the global average rate for the era, not elevated at all. That's a clean, checkable pattern. The "South Africa is pace-friendly" reading looks like it's being generated almost entirely by South Africa's own bowlers (Pollock, Ntini, Steyn, Morkel, Philander, Rabada, Nortje, a remarkable production line) outperforming everyone else there, not by the pitches rewarding pace bowling generally. Steyn himself only accounts for 8% of the South African pace sample, which rules out simple self-skewing, but his cohort collectively is large enough to move the country-wide reading.

The fix seemed obvious. Re-estimate the country factor from away (touring) bowlers only, isolating the conditions effect from the home-talent effect, and apply that neutral reading to everyone, home or away. Rebuilding the pipeline this way changed the leaderboard substantially. Steyn's bowling WAR jumped from 5.12 to 8.22, and the same mechanism lifted every bowler who does most of their work at home, pulling the aggregate batting/bowling split from 64.7/35.3 back to roughly 55/45.

Validated on matches held out entirely from fitting, it made predictions worse, not better. Weighted RMSE of 0.52 against the original pooled version's 0.48, worse than using no country factor at all (0.50). Scored only on the home-bowler cases the fix specifically targeted, the gap was even wider. Bootstrapped across 200 resamples of the train/test split, the away-only version lost to the original every single time. A home bowler's historical edge in their own conditions turns out to be useful information for predicting how they'll do there, not bias to strip out, the same lesson as the position two-way check above, arrived at independently. Reverted; Steyn's number in Root Cause is unchanged.

Tested and rejected: pricing a wicket by the batter's quality

A wicket is currently priced the same regardless of who got out. The same ~33-run base value whether the dismissed batter averages 50 or 3, discounted only by leverage, never by the batter's own quality. A natural next question, raised directly rather than found by chasing a suspicious result, should dismissing a recognised top-order batter count for more than cleaning up a number 11?

First attempt priced each wicket by the dismissed batter's own trailing quality rating, the same shrunk, era-adjusted per-player figure already used for opposition quality, relative to the league average that year. Built and checked against the leaderboard, and something looked wrong immediately. Bowling WAR rose for nearly every bowler checked, not a mix of winners and losers the way a real redistribution of credit should look.

Fixed by pricing the replacement bowler's hypothetical wickets at the same average opposition quality the real bowler faced that match, computed from every ball bowled rather than just the wicket-taking ones, instead of a flat neutral rate. This version behaves the way a real redistribution should. 460 players moved up, 487 moved down, 117 barely changed. Anderson, Lyon, Steyn, Ashwin and Broad all rated lower; Root, Kohli, Smith and Tendulkar all rated higher. Held out and tested, it came much closer to holding up. AUC 0.985 to 0.982, winning 25.5% of the 200 bootstrap resamples, up from 0%, but still a net loss.

Tried to find a version that won before giving up on the idea. Tightening how far a single wicket's price could swing made it worse, not better (14.5%). Widening that range further made no real difference (26.0%). Blending the flat and victim-adjusted prices halfway made it worse again (12.5%). Nothing tested came close to beating the current model, and there was no sweet spot between no signal and full signal to find.

A third attempt tried a completely different, far more robustly-sampled construction. Instead of a continuous per-player rating, bucket the dismissed batter into top-6 or tail (#7–11) and price the wicket using the win-probability model's own empirical event value for that bucket, computed directly from ball-by-ball win-probability swings rather than batting averages. The split is real. Top-6 wickets cost the batting side a win-probability swing equivalent to ~45.5 runs (18,004 events), tail wickets ~23.8 runs (10,653 events), both far bigger samples than any per-player rating could offer. Built with the same symmetric construction that fixed the continuous version (replacement wickets priced at the bowler's own average top6/tail mix faced that match, not a flat rate), and it did worse, not better. AUC 0.985 to 0.978, winning just 1.0% of the 200 bootstrap resamples. The likely reason is that top-order wickets fall earlier in an innings, when win-probability swings are naturally larger because the game state is still uncertain, and tail wickets disproportionately fall once a match is already decided, the same match-state confound that sank the innings-level pitch-wear idea above. Leverage is already applied as its own separate step in this pipeline, so a wicket price built straight from raw win-probability swings most likely double-counts that effect rather than adding anything new.

Three attempts now, all at the same real question, rejected for related but distinct reasons. An asymmetric baseline that inflated every bowler uniformly; then, once that bug was fixed, a real but weak signal that adds noise rather than value; and a coarser, better-sampled version of the same idea that ended up measuring leverage a second time rather than victim quality at all. Left as tested and rejected, alongside the country-conditions idea above.

Stress-tested and held up: is the country factor unfair to Lyon and Steyn?

The away-only test above was the first attempt at a specific worry, that the country x type factor makes Australia look artificially brutal for spin and South Africa artificially kind to pace, inflating Nathan Lyon's rating and depressing Dale Steyn's for reasons that have nothing to do with real conditions. Away-only was one way to test that. It wasn't the only one, and it's worth showing the rest of the attempts rather than stopping at the first result that happened to go against the worry.

Venue-level conditions. Sydney Cricket Ground's spin wicket rate (0.0130/ball, 17,235 balls, 25 Tests) sits meaningfully above Australia's overall spin rate (0.0124), and the Gabba sits meaningfully below (0.0113), a real difference the country-level factor can't see, since it treats every Australian venue identically. Built a venue-level refinement, shrunk toward each venue's own country x type baseline, and layered it on top of the existing country factor. AUC dropped from 0.985 to 0.981 on the held-out test, taking just 8% of the 200 bootstrap resamples. Tried much heavier shrinkage (roughly 4x stronger) in case thin-sample venues were adding noise; barely moved (7%). Breaking SCG's rate down by innings explained why: it isn't elevated from ball one, it's elevated almost entirely in the 4th innings (0.0091 in the 1st, climbing to 0.0208 in the 4th, while the rest of Australia barely moves across the same span). A flat, whole-match venue factor smears a real but narrow, late-match-specific effect across innings where it doesn't apply, adding noise rather than signal.

Innings-level pitch wear. The SCG finding pointed at something bigger. Across the whole dataset, not just one venue, wicket rate climbs sharply through a Test's four innings, and far more for spin than pace (pace +11% from 1st to 4th innings then flat; spin +53%, climbing the whole way, ending above pace's own rate). Batting average falls 25% over the same span. Every bucket here has 80,000+ balls behind it. Built an innings-number refinement, ball-weighted per bowler per match since the underlying tables are already aggregated per match, not per innings. The held-out result was worse than the venue-specific version's, despite the far larger, cleaner-looking sample behind it. AUC fell from 0.985 to 0.980, and it won just 1 of the 200 bootstrap resamples. The likely reason is that a team's 4th innings is disproportionately a run-chase or a collapse, so wickets falling faster there is tangled up with which side is already losing, not purely pitch physics. A bowler closing out a tight chase in the 4th innings is doing exactly the thing that predicts winning; discounting those wickets as "expected, conditions were tougher" removes real signal rather than adding noise.

Four narrower checks, each aimed directly at whether the Lyon/Steyn gap specifically is an artifact:

That last door didn't stay closed for long. Hand-labeled sub-type for the same ~186 frontline bowlers already in the pace/spin heuristic (off-spin, leg-spin, left-arm orthodox, left-arm wrist for spinners; left-arm vs right-arm for pace), covering 90% of all balls bowled by type. Built a sub-type refinement layered on the existing country x type factor the same way venue and innings were, and it failed too. AUC slipped from 0.985 to 0.981, taking just 2.5% of the 200 bootstrap resamples. The labels are real cricket knowledge, not guesses, and the coverage is good, but the result is the same shape as everything else in this section, a plausible-looking refinement that makes the model worse, not better, once tested.

Seven separate mechanisms tried now, three of them full model rebuilds and four of them narrower, targeted checks, and none of them turns "Australia is hard for spin, South Africa is easy for pace" into an artifact. Several actively strengthen the case that it's real. The gap survives restricting to standardised early-match conditions, it survives checking for a compression effect, and correcting for Lyon's own weight in his own baseline would make his rating go up, not down. This is the same kind of result as the replacement bar note elsewhere in this piece, a reading that looks at first like it could be a bug, tested from every angle available, and holding up as a real feature of the data rather than one.

Two more tested and rejected: team strength and ball age

Team Elo. A different angle on the same underlying question opposition quality already addresses. Individual victim-quality pricing measures the specific players faced, but team-level effects (form, momentum, depth beyond the XI) might carry information a player-level average misses. Built a standard Elo rating from match results alone (K=24, every team starting at 1500, strictly pre-match with no lookahead), normalised to each five-year era's mean rating to avoid a drift bias. Checked first whether it was even worth building. Correlation with the existing balls-weighted opposition-quality factor came back at 0.55, meaningfully different information, not redundant, unlike most of what else got tested this session. Applied as a second multiplicative factor alongside the existing opposition-quality adjustment, on both batting and bowling. This was the single worst result of anything tried this session, despite passing its own pre-check. AUC fell from 0.985 to 0.965, and it lost every one of the 200 bootstrap resamples. The likely explanation is that win/loss-based Elo is noisy at the individual-match level (rain-affected results, dead-rubber team rotations, small early-career samples under an aggressive K-factor), and that noise outweighed whatever real signal the 0.55 correlation implied.

Ball age. A more mechanically grounded idea. The laws of cricket give the fielding side a second new ball from over 80 of an innings, so wicket-taking conditions shouldn't be flat across an innings even setting pitch wear and match state aside. Checked the raw pattern first. Wicket rate is 0.0162 in overs 1–20 (new-ball assistance), drops to 0.0152 in overs 21–50 (worn ball, no swing), then climbs back through 0.0158 in overs 51–80 to 0.0188 in overs 80 and beyond, a U-shape, not the steady climb the already-rejected innings-number version showed, and a shape that matches the actual new-ball rule rather than generic pitch deterioration. Built a ball-age refinement from this, layered on the existing country x type factor the same way venue and innings were. This was the closest any refinement came to beating the current model all session, and it was still a net loss. AUC only slipped from 0.985 to 0.984, taking 27.0% of the 200 bootstrap resamples. Lyon and Steyn both move only slightly (Lyon up, Steyn essentially flat), a far gentler redistribution than most of what else was tried, consistent with a mechanism that's real but modest rather than a bug in the reading. Tried smoothing the four hard bucket edges into a continuous spline fitted to the actual per-over wicket-rate and economy curves, in case the discreteness itself was the problem. It wasn't. 26.5% of the 200 bootstraps, essentially identical to the bucketed version. The ceiling here isn't bucket-boundary noise.

Nine attempts now at improving on the current model's baseline factors, all rejected, one of them close. At some point a consistent pattern like this becomes its own finding. The current combination of adjustments looks like it's sitting near a real ceiling for this validation metric, not waiting on the next refinement to unlock it.

A ceiling check: does a smarter model find more signal?

Every validation in this piece uses the same tool, logistic regression on one number, a team's total RAR differential. Worth asking directly whether that's leaving real predictive signal on the table, or whether the current combination of adjustments is truly close to the limit of what these inputs can predict, rather than just under-modelled.

Gave a gradient-boosting classifier (scikit-learn's HistGradientBoostingClassifier, avoiding an external XGBoost dependency) the same batting and bowling RAR differentials, first as two separate features instead of one pre-summed number, then with team Elo and era added for a richer shot at finding leftover signal. The simplest change alone, just not pre-summing batting and bowling into one number, before any tree model got involved, won 99.5% of 200 bootstraps. AUC 0.988 to 0.996, log-loss more than halved (0.154 to 0.074), accuracy 94.3% to 98.6%. The tree-based versions matched this but didn't beat it. A much bigger jump than anything else tested this session.

That result needed checking before trusting it. The standardised coefficient on batting differential came out roughly 4.4x the coefficient on bowling differential (the two are correlated 0.865 in this dataset, a team winning the batting battle usually wins the bowling one too, so they're far from independent signals). That's the same shape as v2's original failure, where the raw win-probability model also rated batting far above bowling, traced at the time to elite bowlers being deployed at exactly the moments a wicket is already expected, undercrediting them in a model sensitive to surprise. Worth checking directly whether this is that same bug back in a different form. Is the batting/bowling asymmetry here being driven by the same bowlers v2 flagged, Lyon, Anderson, Starc, and the rest?

The more likely explanation is that batting runs sit almost directly on the final scoreboard margin, a continuous signal. Bowling's contribution runs mostly through wickets, a discrete, lumpy event (at most 20 in a match) priced at one validated average (~33 runs, stable at 33.0 ± 0.5 across 200 bootstraps, per v10) that's accurate on average across the whole dataset but inherently higher-variance at the level of any single match. Batting RAR is a smoother signal of match-level contribution; bowling RAR is lumpier even when correctly calibrated on average.

That distinction matters for what to conclude. WAR's job is measuring average contribution to winning across a career, using the best available average pricing, which v10 already validated independently and robustly. This check shows single-match outcome prediction is inherently easier from batting data than bowling data, a narrower, different claim than "bowling is overvalued" or "WAR should be reweighted." Left as a documented finding, not a model change.

Pursued anyway: a joint hierarchical model

The country, sub-type, and ball-age refinements above were each fit the same way. One factor, estimated on its own, shrunk toward its own separately-chosen prior, then multiplied into the pipeline sequentially. The natural next question is whether fitting all three jointly (one shared model that lets the data decide how much each combination of country, sub-type, and ball-age should borrow strength from the others, instead of three independent passes) does better. My honest read going in was that this was unlikely to help, since it changes how already-tested signals are estimated rather than adding a new one, and every attempt to add or restructure signal this session had landed at or below the current baseline. Pursued anyway.

Built as a ridge regression on one-hot country x type, sub-type, and ball-age-bucket features, fit simultaneously for both economy and wicket-rate, with the regularization strength chosen by cross-validation rather than picked by hand, the same shrinkage principle as everywhere else in this project, just fit jointly instead of sequentially. The first result was the most striking of the entire session. The only model change out of eleven attempts to beat the current one. AUC 0.985 to 0.990, log-loss down 13%, winning 82.5% of the 200 bootstraps.

That result was strong enough to need real stress-testing before trusting it, not just a shrug and a ship. First check. The original cross-validation split balls randomly, which doesn't respect match boundaries, so balls from the same match could leak across the inner train/test folds used to pick the regularization strength. Refit with match-grouped cross-validation instead. The result survived, a bit weaker (76.0% of bootstraps) but clearly still a real win. Second check. Walk-forward validation at five different cutoffs (2015 through 2021, training only on the past each time). Won at every single one. By the two standard checks this session has used throughout, this looked solid.

Rejected, and not adopted as the model. The result is real in a narrow sense (it did score better, consistently, across both random and temporal splits), but it isn't a real, generalizable improvement in how conditions get priced. Both of the standard checks this session relies on test temporal generalization (does the pattern hold in the future); neither tests player generalization (does the pattern hold for players other than the ones defining it). This one needed the second kind of check to catch, and it's a real methodological lesson for reading any future granular refinement, not just this one. Passing bootstrap and walk-forward validation isn't sufficient once a factor gets fine-grained enough that individual players can dominate their own comparison cell.

Also tested and rejected: a time-varying replacement bar

The fringe/replacement classification used everywhere in this project decides a player's status from their final, whole-career Test tally, a real, if narrow, form of foresight. A player who eventually plays 150 Tests is never counted as replacement level, not even in his first Test, when he was, at that exact moment, entirely indistinguishable from a real journeyman. Fixed by rebuilding the whole replacement-level pipeline with a time-varying version instead. A player's fringe status entering any given match is decided only by Tests played strictly before it, never their eventual total.

The fringe pool nearly tripled (1,391 to 3,989 bowling-match rows), since it now includes every future star's rookie-year performances alongside true journeymen. Rebuilt the full pipeline (opposition quality, pitch factors, wicket price, country and position, all of it) on top of this new baseline, and the result was almost entirely one-directional. 980 of roughly 1,064 players rated lower, only 2 rated higher, though the career correlation with the current model stayed fairly high (0.943). AUC fell from 0.985 to 0.982 on the held-out check, taking 11.0% of the 200 bootstrap resamples.

The fix closes a real gap. The replacement bar no longer has any foresight baked into it at all. But the practical effect isn't a fairer re-ordering of who's better than whom; the high correlation with the current model says the ordering barely moved. It's mostly a uniform, one-directional recalibration of the whole scale upward, since an eventual star's rookie performances tend to be better than an ordinary journeyman's, dragging the replacement average up wherever they're now included. That extra signal turned out to add noise to the baseline rather than sharpen it. Rejected; the whole-career definition, foresight and all, is still what's shipped.

Checked and found sound: is the model overconfident at the extremes?

AUC only tests whether the model ranks matches correctly relative to each other, not whether its real probability numbers mean what they say. A badly overconfident model could still rank perfectly. Worth checking directly. Does a predicted 90% win probability correspond to a real 90% win rate, particularly in the high-stakes, late-match scenarios where that distinction matters most?

Built a reliability curve on the held-out test matches. Predicted probability bucketed against actual outcome. Most buckets land within a few percentage points of the diagonal. The extreme end (predicted above 85% win probability, 107 matches) shows mild overconfidence, not a dramatic one. The model's average prediction there was 98.8%, the real win rate 97.2%. Tried fixing it with isotonic regression, fit on the training split and applied to the held-out one, and it made the Brier score slightly worse (0.0451 to 0.0457), not better. With only 215 held-out matches to fit a recalibration curve on, isotonic regression, a flexible, non-parametric method, has too little data to learn a reliable correction and mostly just fits noise specific to that sample. The model's calibration is already reasonably sound as it stands; a formal recalibration step isn't earning its complexity.

Checked before building: does a keeper deserve a lower batting bar?

Root Cause already flags this as a known gap. A keeper-batter's real value is mostly in the gloves, and that's invisible to a model built purely from ball-by-ball data, leaving only a modest batting record on show. The natural fix would be a lower replacement bar for keepers specifically, reflecting the selection tradeoff. Before building a whole new layer for it, checked whether that premise holds.

Keepers aren't hand-labelled anywhere in this pipeline, but Cricsheet's own dismissal records make them identifiable without any external data. Whoever is credited on a stumping is unambiguously the keeper for that team that match, since only a keeper can effect one. Extended to catches by cross-referencing against a roster of players with three or more career stumpings. Covers 1,506 of 1,780 team-matches, 84.6%.

With keepers identified, compared their batting average directly against non-keepers at the same batting position. "Keeper vs replacement level" isn't the right test, since an established keeper, like an established specialist, comfortably clears a fringe-level bar regardless. The real question is whether keepers underperform true specialists in the same slot. They don't. At every position checked, keepers outperformed non-keepers. Position 5 (+5.5%), position 6 (+7.2%), position 7 (+6.3%, the most common keeper slot, on a solid 1,297-dismissal sample), position 8 (+60%, on a thinner sample). The "gloves over runs" tradeoff the whole idea rests on doesn't hold in this dataset.

Almost certainly an era effect. This dataset covers 2001–2026, spanning cricket's shift toward proper wicketkeeper-batters (Gilchrist onward, then Sangakkara-era Sri Lanka, Pant, de Kock, Bairstow) rather than a specialist gloveman tolerated for his hands alone. Building a lower replacement bar for keepers would have moved in exactly the wrong direction, crediting them for clearing a bar they're already beating without any help. Checked directly before committing to a build, and the premise didn't survive contact with the data. No model change.

Also tested and rejected: pricing a wicket by how set the batter was

A cricketing intuition that hadn't been tested yet. A fresh batter's wicket should be cheaper than a well-set one's, since survival itself is a signal of difficulty overcome. Checked directly in the win-probability data before building anything. Bucket all 28,657 Test wickets in this dataset by the dismissed batter's cumulative balls faced, and read off the model's own empirical cost per bucket.

The gradient was real, monotonic, and well-sampled. Fresh wickets (0–15 balls faced, 9,062 of them) cost the batting team about 31 runs of win probability; well-set ones (80+ balls, 6,158 of them) cost about 41, a roughly 31% spread, rising smoothly through two intermediate buckets in between. That's exactly the kind of signal that earned the top-6/tail wicket-price split its place in the model already.

Built the same way as that split. A bucket-specific multiplier on the shipped, era-adjusted flat wicket price, with the replacement bowler's hypothetical wickets priced at the same settledness mix the bowler bowled to, so neither side is compared against a flat neutral rate. But the risk flagged before writing a line of code was the one that sank it. A batter who survives 80+ balls is usually also batting for a team in a stronger position generally, so the higher "cost" of dismissing them partly reflects match state the model already prices through leverage, not the batter's skill in getting through the new-ball period. Held-out validation confirmed it. The bootstrap comparison favoured this version in just 6.0% of 200 resamples, and a 2020 walk-forward split showed the same pattern out of time. A real, clean signal in the raw win-probability data, and still a leverage confound rather than new predictive information once run through the same test every other candidate has to pass.

Also tested and rejected: an explicit win/draw/loss model

The win-probability model at the heart of this whole project (v2 onward) regresses a single target, win_points, coded 1/0.5/0 for a win/draw/loss. That collapses two very different match states into the same number. A team cruising to a near-certain draw (say 95% draw, 2.5% win, 2.5% loss) and a team in a truly balanced 50/50 win-or-loss battle both average out to 0.5, even though they're nothing alike. An explicit 3-class model, predicting P(win), P(draw), and P(loss) separately rather than one blended number, can in principle tell them apart.

Built one: a gradient-boosting classifier on the same four match-state features (innings, run lead, wickets down, balls left), trained on the same held-out split, collapsed back into an expected win value (P(win)×1 + P(draw)×0.5) for everything downstream that consumes a single win-probability number, leverage and the validated wicket price alike. One real limitation going in. Scikit-learn's gradient booster doesn't support the monotonic constraints (more wickets down can only ever hurt the batting team) for a 3-class target the way it does for the original binary regressor, so this version has no built-in guarantee against a wicket occasionally looking good for the team that just lost one.

That gap showed up directly in the numbers. Re-deriving the wicket price from this model's own event values, the same bootstrap methodology that produced the original's tight 33.0 ± 0.5 runs figure came back at 35.6 ± 0.9, a wider, noisier estimate, and one era in particular (2005–2009) swung from 33.5 to 42.5 runs, a jump too large to trust. Re-leveraging career WAR with this model's own leverage index told a similar story. Still highly correlated with the current rankings (0.996 on total WAR, 0.983 on rank), but leverage itself agreed with the original much more loosely (correlation 0.84) than a real refinement should.

Held out and tested the way every wicket-price change this session has been. Rebuilt bowling RAR with the new price and ran it through the standard match-outcome check. It lost decisively. AUC fell from 0.985 to 0.978, and it lost all 200 of the 200 bootstrap resamples, a result echoed in a 2020 walk-forward split (0.989 to 0.985). The 3-class distinction is conceptually the more honest model of what a draw really is. In practice, without the monotonic constraint that keeps the original regressor's predictions smooth, it produces a noisier wicket price that generalizes worse, not better. Rejected; the single-target regressor, cruder distinction and all, is still what's shipped.

An open problem: allrounders and roster flexibility

A question worth taking seriously, chased through three different attempts, none of which held up cleanly enough to ship. Batting and bowling WAR, summed for one player, should in principle already measure "how much better is this player in this slot than a replacement would be", but only if the replacement baseline reflects what a real substitute would cost the team. A team fielding a true allrounder effectively gets two contributions from one roster slot, which can mean fitting a stronger batting order and a deep enough bowling attack into the same eleven players. If a true dual-threat replacement is harder to find than the population-level replacement averages assume, an allrounder's real value could be understated.

First attempt: does the batting order get stronger further down, when the allrounder plays? Restricted to each player's own era, so "without them" means real absences rather than a different generation of teammates.

Jadeja shows a clean, monotonically increasing effect: every tail position rates stronger with him in the side, climbing from +1.3 at #6 to +6.1 at #9. Nobody else replicates that shape. Shakib shows a partial version, with solid bumps at #6 and #8, but negative at #9 and barely there at #7. Stokes shows almost nothing, and what's there isn't consistent in direction. Kallis can't be tested at all: he played 88 of South Africa's 94 Tests in his own era, leaving only 6 matches without him, not a real counterfactual, just noise dressed up as one. One clean signal, one partial one, one that doesn't replicate, and one that can't be tested. Not a general effect, or at least not one this particular measure can see clearly.

Second attempt: does a true allrounder let the team use fewer other bowlers? A more direct version of the same idea, not who bats where, but how many other players need to contribute overs at all. This one looked much better at first. All five players above showed fewer other bowling contributors (30+ balls) per match when they played, from −0.47 (Stokes) to −0.99 (Ashwin), consistent in direction across every case.

It didn't survive the next check. Splitting by bowling type revealed the real driver. Pure specialist spinners, with no batting ability at all, reduce a team's need for other bowlers by almost exactly as much as true batting-bowling allrounders who happen to bowl spin (4.22 vs 4.23 other contributors at high workloads, from an identical starting point). Specialist pace bowlers show a similar pattern to batting-allrounder pace bowlers too. The effect is a mechanical property of bowling type. A spinner can sustain a much bigger share of a day's overs than a seamer can, so any team leaning on a lot of spin needs fewer bodies to cover the same total overs, regardless of whether that spinner can bat. Jadeja and Ashwin showing the strongest numbers in the first pass wasn't evidence of two-way value; it's because they're both spinners, and any specialist spinner would show the same thing. Building a credit off this would have rewarded spin-bowling allrounders for being spinners, not for being allrounders, while pace-bowling allrounders like Stokes and Kallis would have been shortchanged for the same reason. Dropped.

Third attempt: does an allrounder's presence predict winning better than their own summed WAR already does? A cleaner test in principle, since it doesn't depend on guessing the mechanism. If there's hidden value the additive model misses, a team with a recognised two-way contributor (batting and bowling WAR both meaningfully positive per Test, 14 players in this dataset qualify) should win more often than its raw batting-plus-bowling differential alone predicts. Adding an allrounder-presence indicator to the held-out match-outcome model did improve the fit slightly (AUC 0.985 to 0.986, log-loss 0.154 to 0.150), but the indicator's own coefficient came back negative, the opposite of what a hidden bonus would look like. The more likely explanation isn't that allrounders hurt their teams; it's a selection effect running the other way. Teams often lean on a true allrounder specifically because they're short elsewhere in batting or bowling depth, not because allrounders make an already-strong side stronger. Whatever is happening here, it isn't evidence for crediting allrounders more.

Three different approaches, three non-results: one real-looking pattern that doesn't generalise past a single player, one that was a bowling-type artifact rather than a batting-skill effect, and one direct test that came back with the wrong sign. I don't think there's a credit mechanism to build from any of this. The most honest conclusion available right now is that the existing additive model, batting WAR plus bowling WAR, each measured on its own terms, may already be close to right, or that if a real roster-flexibility bonus exists, it isn't detectable with what's in this dataset. Left open rather than closed, since "we tried three ways to find it and couldn't" is a different, weaker claim than "it isn't real." For now, nothing changed in the model.

That's the full build. For what it found, the leaderboard, the head-to-head debates, and where the model still falls short, back to Root Cause.