CI-LLM Research: Problem, Method, Results

A short summary of what we studied, how we measured it, what we found, and what to do with it.

Problem

Code-review LLMs flood PRs with noisy findings. Naive “show everything” policies hurt signal-to-noise and reviewer trust. We need a repeatable way to:

Generate findings without biasing the generator toward conservative, low-value output
Score findings without the model grading itself
Ground-truth “was this fixed?” without counting whitespace-only edits as fixes
Tune how aggressively we filter (severity, confidence, density, style, snippets) against catch rate, annoyance, and volume

Method

Layer	What we did
Data	1,022 PRs, 445 gold PRs across 7 repos (C, Rust, TS, JS, Python). AST-hash labels; excluded repos with unusable PR workflows.
Generation	Unconstrained LLM generation (4 models × repos); meta-eval with Claude 3 Haiku — decoupled from the generator.
Labeling	AST comparison for `historically_fixed`; comment/whitespace-only → not fixed.
Objectives	Smooth metrics: Laplacian CatchRate+, log-scaled Annoy-, Volume-; DensityMax (comments per 100 changed lines), not a flat comment cap.
Policies	10k simulated policies per model×repo over 5 parameters (`Smin`, `Cmin`, `DensityMax`, `Allow_Style`, `Require_Snippet`).
Optimization	D2H (distance to ideal: high catch, low annoyance, low volume). Compared naive (no filter), random search (B samples from 10k), and EZR active learning (Naive Bayes acquisition, same B). Budget sweep B ∈ {50, 500, 1000, 3000}.

Results

Ground truth & generation

Gold PRs and per-repo stats as in the full study; ~15–17% label precision (historically fixed) across generators at volume — the problem is inherently noisy.

Filtering beats the naive baseline

D2H drops ~34–40% vs accepting all findings (naive D2H roughly 0.58–0.67 → optimized roughly ~0.39–0.41 region depending on budget).

Random search vs active learning

Random search wins or ties in most runs at every budget; EZR Bayesian AL does not reliably beat random sampling on this landscape.
The 5D policy space is small; ~400 distinct D2H values in 10k samples — a staircase surface, not a smooth gradient.
AL has higher variance than random; little room left for a surrogate to improve once B ≥ 50.

Interpretable rules (from trees / exploration)

Recurring good policies: tight DensityMax, high Cmin, moderate Smin, often Require_Snippet = 0, sometimes Allow_Style = 0.

Thesis & recommendation

The win is architecture, not the optimizer.
Unconstrained generation + post-hoc filtering (with AST-grounded labels and decoupled scoring) is what improves the quality–noise tradeoff. The policy search problem in this setting is easy: random search with a modest budget (e.g. B ≈ 50) is fast, stable, and sufficient; fancy Bayesian active learning does not pay off here.

Practical takeaway: Invest in generation quality and filtering semantics; use simple random search over the simulated policy table unless the policy dimensionality or evaluation cost grows substantially.

Appendix — Result tables

Tables below mirror RESEARCH_STATUS.md (v2 pipeline).

Phase 1 — Ground truth scale

1,022 PRs crawled, 445 gold PRs across 7 active repos.

Repo	PRs	Gold PRs	Gold Lines	Language
redis/redis	200	135	40,238	C
microsoft/vscode	200	127	105,917	TypeScript
pallets/flask	125	57	23,683	Python
django/django	97	53	1,878	Python
rust-lang/rust	122	38	16,503	Rust
expressjs/express	47	22	7,500	JavaScript
curl/curl	31	13	379	C

Excluded: torvalds/linux (email patches, 0 PRs), golang/go (Gerrit, 0 PRs), facebook/react (4% gold rate).

Phase 2 — LLM generation + meta-eval

4 generator models × 7 repos = 28 runs. Meta-evaluator: claude-3-haiku (decoupled from generators).

Generator	Total Findings	Avg/PR	Notes
`gpt-4o-mini`	5,496	6.7	Baseline model
`deepseek-v3.2`	12,902	15.7	Highest volume
`gpt-5-mini`	15,551	18.9	Highest volume overall
`gemini-3-flash-preview`	5,789	7.0	Fastest generation (~10 min)

Phase 3 — Labeling (`historically_fixed`)

Generator	Findings	Fixed	Precision
`gpt-4o-mini`	5,496	831	15.1%
`deepseek-v3.2`	12,902	1,917	14.9%
`gpt-5-mini`	15,551	2,462	15.8%
`gemini-3-flash-preview`	5,789	977	16.9%

Phase 4 — Policy optimization (D2H: random vs EZR active learning)

D2H = Euclidean distance from normalized objectives to ideal (CatchRate=1, Annoyance=0, Volume=0). Lower is better.

Naive: accept all LLM findings (no filtering); D2H ≈ 0.58–0.67
Random: best D2H from B random picks out of 10k simulated policies
AL: EZR acquire.guess() with likelier (Naive Bayes acquisition), same B
Budget sweep: B ∈ {50, 500, 1000, 3000}, independently seeded per repeat

Budget sweep summary (4 models × 5 repos = 20 experiments per budget level)

Budget	Repeats	AL Wins	Ties	Rand Wins	Avg Rand D2H	Avg AL D2H	Avg abs Δ
50	20	0	5	15	0.413	0.436	0.024
500	10	1	7	12	0.400	0.410	0.011
1000	10	2	9	9	0.397	0.408	0.013
3000	10	0	9	11	0.394	0.407	0.013

Budget = 50 (repeats = 20, n = 28) — AL wins 0, ties 6, rand wins 22

Model	Repo	Naive	Rand	σ_R	AL	σ_AL	Δ(R−AL)	Winner
deepseek-v3.2	curl/curl	0.669	0.3497	0.0200	0.3616	0.0904	-0.0119	Rand ✓
deepseek-v3.2	django/django	0.592	0.3946	0.0082	0.3917	0.0791	+0.0029	≈
deepseek-v3.2	expressjs/express	0.645	0.4104	0.0183	0.4210	0.0584	-0.0106	Rand
deepseek-v3.2	microsoft/vscode	0.591	0.4319	0.0062	0.4397	0.0818	-0.0077	Rand ✓
deepseek-v3.2	pallets/flask	0.590	0.4034	0.0171	0.4400	0.0915	-0.0366	Rand ✓
deepseek-v3.2	redis/redis	0.663	0.4596	0.0018	0.4614	0.0292	-0.0018	≈ ✓
deepseek-v3.2	rust-lang/rust	0.653	0.4011	0.0110	0.4018	0.0669	-0.0007	≈
gemini-3-flash	curl/curl	0.586	0.3578	0.0218	0.3943	0.1080	-0.0365	Rand ✓
gemini-3-flash	django/django	0.590	0.4073	0.0086	0.4424	0.0846	-0.0351	Rand ✓
gemini-3-flash	expressjs/express	0.611	0.3302	0.0271	0.3908	0.0983	-0.0607	Rand ✓
gemini-3-flash	microsoft/vscode	0.579	0.4358	0.0069	0.4407	0.0799	-0.0049	≈ ✓
gemini-3-flash	pallets/flask	0.597	0.4300	0.0093	0.4481	0.0679	-0.0181	Rand ✓
gemini-3-flash	redis/redis	0.635	0.4262	0.0091	0.4607	0.0516	-0.0345	Rand ✓
gemini-3-flash	rust-lang/rust	0.601	0.4186	0.0218	0.4324	0.0915	-0.0138	Rand ✓
gpt-4o-mini	curl/curl	0.658	0.3979	0.0182	0.4243	0.0533	-0.0264	Rand ✓
gpt-4o-mini	django/django	0.639	0.4440	0.0106	0.4440	0.0426	+0.0000	≈
gpt-4o-mini	expressjs/express	0.649	0.4159	0.0135	0.4807	0.0637	-0.0649	Rand ✓
gpt-4o-mini	microsoft/vscode	0.579	0.4291	0.0137	0.4628	0.0820	-0.0337	Rand ✓
gpt-4o-mini	pallets/flask	0.607	0.4052	0.0142	0.4481	0.0858	-0.0430	Rand ✓
gpt-4o-mini	redis/redis	0.613	0.3961	0.0084	0.4360	0.0710	-0.0400	Rand ✓
gpt-4o-mini	rust-lang/rust	0.616	0.3366	0.0172	0.3591	0.1231	-0.0225	Rand ✓
gpt-5-mini	curl/curl	0.587	0.3836	0.0140	0.4243	0.0876	-0.0407	Rand ✓
gpt-5-mini	django/django	0.640	0.4273	0.0043	0.4555	0.0555	-0.0281	Rand ✓
gpt-5-mini	expressjs/express	0.582	0.3751	0.0132	0.3847	0.0957	-0.0096	Rand ✓
gpt-5-mini	microsoft/vscode	0.647	0.4531	0.0038	0.4531	0.0432	+0.0000	≈ ✓
gpt-5-mini	pallets/flask	0.593	0.4173	0.0042	0.4382	0.0900	-0.0208	Rand ✓
gpt-5-mini	redis/redis	0.622	0.4016	0.0095	0.4542	0.0597	-0.0525	Rand ✓
gpt-5-mini	rust-lang/rust	0.614	0.3749	0.0185	0.3920	0.0866	-0.0171	Rand ✓

Budget = 500 (repeats = 10, n = 20) — AL wins 1, ties 7, rand wins 12

Model	Repo	Naive	Rand	σ_R	AL	σ_AL	Δ(R−AL)	Winner
deepseek-v3.2	curl/curl	0.669	0.3081	0.0105	0.3238	0.0161	-0.0156	Rand ✓
deepseek-v3.2	django/django	0.592	0.3834	0.0030	0.3988	0.0132	-0.0154	Rand ✓
deepseek-v3.2	microsoft/vscode	0.591	0.4203	0.0082	0.4396	0.0261	-0.0193	Rand ✓
deepseek-v3.2	pallets/flask	0.590	0.3933	0.0024	0.3920	0.0031	+0.0013	≈ ✓
deepseek-v3.2	redis/redis	0.663	0.4574	0.0002	0.4574	0.0000	+0.0000	≈ ✓
gemini-3-flash	curl/curl	0.586	0.3373	0.0063	0.3452	0.0315	-0.0079	Rand ✓
gemini-3-flash	django/django	0.590	0.3978	0.0029	0.4091	0.0129	-0.0113	Rand ✓
gemini-3-flash	microsoft/vscode	0.579	0.4271	0.0023	0.4602	0.0134	-0.0331	Rand ✓
gemini-3-flash	pallets/flask	0.597	0.4142	0.0083	0.4045	0.0187	+0.0097	AL
gemini-3-flash	redis/redis	0.635	0.4216	0.0033	0.4238	0.0183	-0.0022	≈ ✓
gpt-4o-mini	curl/curl	0.658	0.3816	0.0058	0.3979	0.0134	-0.0163	Rand ✓
gpt-4o-mini	django/django	0.639	0.4440	0.0027	0.4440	0.0087	+0.0000	≈ ✓
gpt-4o-mini	microsoft/vscode	0.579	0.3978	0.0121	0.4232	0.0061	-0.0254	Rand ✓
gpt-4o-mini	pallets/flask	0.607	0.3802	0.0053	0.3913	0.0105	-0.0111	Rand ✓
gpt-4o-mini	redis/redis	0.613	0.3878	0.0024	0.3932	0.0115	-0.0054	Rand ✓
gpt-5-mini	curl/curl	0.587	0.3728	0.0050	0.3711	0.0163	+0.0017	≈
gpt-5-mini	django/django	0.640	0.4219	0.0028	0.4434	0.0184	-0.0215	Rand ✓
gpt-5-mini	microsoft/vscode	0.647	0.4466	0.0024	0.4484	0.0023	-0.0018	≈
gpt-5-mini	pallets/flask	0.593	0.4121	0.0029	0.4114	0.0070	+0.0006	≈
gpt-5-mini	redis/redis	0.622	0.3934	0.0018	0.4134	0.0032	-0.0200	Rand ✓

Budget = 1000 (repeats = 10, n = 20) — AL wins 2, ties 9, rand wins 9

Model	Repo	Naive	Rand	σ_R	AL	σ_AL	Δ(R−AL)	Winner
deepseek-v3.2	curl/curl	0.669	0.3026	0.0091	0.2955	0.0191	+0.0071	AL
deepseek-v3.2	django/django	0.592	0.3832	0.0026	0.3888	0.0180	-0.0056	Rand ✓
deepseek-v3.2	microsoft/vscode	0.591	0.4133	0.0053	0.4474	0.0217	-0.0341	Rand ✓
deepseek-v3.2	pallets/flask	0.590	0.3920	0.0012	0.3915	0.0006	+0.0005	≈ ✓
deepseek-v3.2	redis/redis	0.663	0.4574	0.0001	0.4574	0.0000	+0.0000	≈
gemini-3-flash	curl/curl	0.586	0.3310	0.0033	0.3982	0.0301	-0.0671	Rand ✓
gemini-3-flash	django/django	0.590	0.3978	0.0027	0.3951	0.0116	+0.0028	≈
gemini-3-flash	microsoft/vscode	0.579	0.4242	0.0021	0.4500	0.0103	-0.0258	Rand ✓
gemini-3-flash	pallets/flask	0.597	0.4026	0.0082	0.4023	0.0078	+0.0003	≈
gemini-3-flash	redis/redis	0.635	0.4191	0.0033	0.4222	0.0121	-0.0031	≈ ✓
gpt-4o-mini	curl/curl	0.658	0.3816	0.0036	0.3816	0.0139	+0.0000	≈
gpt-4o-mini	django/django	0.639	0.4440	0.0050	0.4440	0.0000	+0.0000	≈ ✓
gpt-4o-mini	microsoft/vscode	0.579	0.3941	0.0080	0.4172	0.0070	-0.0231	Rand ✓
gpt-4o-mini	pallets/flask	0.607	0.3793	0.0056	0.3922	0.0084	-0.0129	Rand ✓
gpt-4o-mini	redis/redis	0.613	0.3875	0.0025	0.4050	0.0065	-0.0175	Rand ✓
gpt-5-mini	curl/curl	0.587	0.3662	0.0059	0.3597	0.0139	+0.0065	AL ✓
gpt-5-mini	django/django	0.640	0.4206	0.0020	0.4434	0.0096	-0.0228	Rand ✓
gpt-5-mini	microsoft/vscode	0.647	0.4466	0.0009	0.4455	0.0014	+0.0011	≈ ✓
gpt-5-mini	pallets/flask	0.593	0.4088	0.0024	0.4091	0.0043	-0.0003	≈ ✓
gpt-5-mini	redis/redis	0.622	0.3927	0.0007	0.4123	0.0020	-0.0196	Rand ✓

Budget = 3000 (repeats = 10, n = 20) — AL wins 0, ties 9, rand wins 11

Model	Repo	Naive	Rand	σ_R	AL	σ_AL	Δ(R−AL)	Winner
deepseek-v3.2	curl/curl	0.669	0.2955	0.0030	0.2955	0.0000	+0.0000	≈ ✓
deepseek-v3.2	django/django	0.592	0.3796	0.0020	0.3767	0.0150	+0.0030	≈ ✓
deepseek-v3.2	microsoft/vscode	0.591	0.4097	0.0027	0.4468	0.0194	-0.0371	Rand ✓
deepseek-v3.2	pallets/flask	0.590	0.3909	0.0005	0.3909	0.0000	+0.0000	≈ ✓
deepseek-v3.2	redis/redis	0.663	0.4574	0.0000	0.4574	0.0000	+0.0000	≈
gemini-3-flash	curl/curl	0.586	0.3310	0.0024	0.3982	0.0046	-0.0671	Rand ✓
gemini-3-flash	django/django	0.590	0.3933	0.0014	0.3951	0.0020	-0.0017	≈ ✓
gemini-3-flash	microsoft/vscode	0.579	0.4226	0.0014	0.4438	0.0009	-0.0212	Rand ✓
gemini-3-flash	pallets/flask	0.597	0.3997	0.0023	0.4167	0.0072	-0.0171	Rand ✓
gemini-3-flash	redis/redis	0.635	0.4139	0.0017	0.4217	0.0031	-0.0077	Rand ✓
gpt-4o-mini	curl/curl	0.658	0.3816	0.0008	0.3816	0.0000	+0.0000	≈
gpt-4o-mini	django/django	0.639	0.4357	0.0067	0.4440	0.0000	-0.0084	Rand ✓
gpt-4o-mini	microsoft/vscode	0.579	0.3898	0.0035	0.4164	0.0021	-0.0266	Rand ✓
gpt-4o-mini	pallets/flask	0.607	0.3738	0.0053	0.3905	0.0012	-0.0167	Rand ✓
gpt-4o-mini	redis/redis	0.613	0.3868	0.0023	0.3932	0.0016	-0.0064	Rand ✓
gpt-5-mini	curl/curl	0.587	0.3592	0.0052	0.3586	0.0000	+0.0005	≈ ✓
gpt-5-mini	django/django	0.640	0.4179	0.0021	0.4487	0.0063	-0.0309	Rand ✓
gpt-5-mini	microsoft/vscode	0.647	0.4456	0.0004	0.4455	0.0000	+0.0001	≈ ✓
gpt-5-mini	pallets/flask	0.593	0.4064	0.0014	0.4091	0.0000	-0.0027	≈ ✓
gpt-5-mini	redis/redis	0.622	0.3918	0.0006	0.4113	0.0011	-0.0195	Rand ✓