AI code search · Learning to rank
Allfather
Bug report in. Broken function out. Ranks up to 45,000 functions in 0.44 seconds. At function level it’s level with the best Claude 3.5 agent, for about 1/2,300 of the cost.
- 79.4%
- of 291 real bugs had a changed function in the top 10, out of 10k–45k candidates
- 93.0%
- file-level hit@10 on 300 SWE-bench Lite issues
- 79.7%
- function hit@10 on a sealed holdout from 184 unseen projects
- $0.0002
- estimated GPU cost per issue, vs $0.46–0.79 for agents
The problem
Before anyone can fix a bug, they have to find the code behind it. Agents can do this, but a Claude 3.5 agent costs tens of cents per issue, as their papers report. The question here was how far a cheap, measured retrieval cascade can go.
The answer key is objective: the functions that the maintainers’ actual fix changed. Every issue is searched at the commit where the bug was reported, because searching at HEAD leaks the fix.
Plain BM25 started at 0.308 function recall@10. The final system reaches 0.772, and each step that moved it has a paired bootstrap behind it.
The ranking cascade
- 01
Parse
Tree-sitter splits the repository into functions. 86% of files are skipped in a run because they’re byte-identical to one already indexed.
- 02
Retrieve
Code-aware BM25 and dense embeddings cover all ~40k units. The ~450-unit pool holds a gold unit for 96.7% of dev issues.
- 03
Rank
LambdaMART weighs 52 features: retrieval scores, traceback frames, mentioned names, git history and past fixes.
- 04
Rerank
A LoRA-tuned cross-encoder reads the issue and code together for the top 20, using a shared-prefix KV cache.
- 05
Ship
A browser pipeline, a GitHub Action and a nightly job that scores bugs fixed that week.
Results
- Plain BM250.308
- Code-aware tokenizer0.428
- Exclude test files0.519
- Hybrid with embeddings0.642
- LambdaMART, 52 features0.729
- Fine-tuned cross-encoder0.772
SWE-bench Lite
| Level | Issues | hit@1 | hit@5 | hit@10 | MRR |
|---|---|---|---|---|---|
| Function | 291 | 45.0% | 73.9% | 79.4% | 0.560 |
| File | 300 | 69.0% | 89.3% | 93.0% | 0.783 |
Nothing was trained or tuned on these issues. Function rows exclude 9 issues whose fix is module-level only.
What each step bought
| Change | Δ recall@10 | 95% CI | p |
|---|---|---|---|
| Naive → code-aware tokenizer | +0.121 | [+0.076, +0.166] | <0.001 |
| Exclude test files | +0.091 | [+0.058, +0.126] | <0.001 |
| Call-graph expansion | +0.003 | [+0.000, +0.010] | 0.72 |
| BM25 → hybrid, overall | +0.123 | [+0.070, +0.174] | <0.001 |
| Hybrid → LambdaMART | +0.087 | [+0.049, +0.128] | <0.001 |
| LambdaMART → + cross-encoder | +0.043 | [+0.013, +0.074] | 0.005 |
Paired bootstrap at function level, 291 issues.
Next to published systems
| System | Models | File Acc@5 | Function Acc@10 |
|---|---|---|---|
| OpenHands | Claude 3.5 | 90.15 | 70.07 |
| LocAgent | Claude 3.5 | 94.16 | 77.37 |
| SweRank | 7B + 32B | 95.99 | 88.69 |
| Allfather | 0.5B + 0.6B | 90.51 | 76.28 |
Scored the LocAgent and SweRank way: 274 Lite issues, counted only if every changed location is in the top k. The other rows are copied from their papers.
In the browser, on 30 projects kept out of training
| Stage | hit@1 | hit@10 | MRR |
|---|---|---|---|
| Keyword search only | 32.9% | 66.0% | 0.435 |
| Instant learned ranking | 44.2% | 74.7% | 0.542 |
| + static-embedding index | 43.9% | 76.6% | 0.544 |
| + MiniLM rerank of top 20 | 44.0% | 76.4% | 0.549 |
| File ranker | 68.0% | 94.2% | 0.774 |
747 bugs. This is what a visitor’s new repository can expect from the on-device pipeline.
Key decisions
A sealed holdout, declared in advance
The fast configuration was frozen, with every model hashed and every metric declared, then scored once on bugs filed after its models were published. The log records all 92 times Lite has been scored.
Exact search over HNSW
Each snapshot answers one query, so HNSW would need about 10,000 queries on one commit to repay its build. Exact search is faster here and lossless, so the optimisation went into indexing and encoding instead.
Call-graph expansion stays opt-in
It looked promising but gained only +0.003 and wasn’t significant. Only 15% of misses are one call from a top hit, so the ceiling was low.
Content addressing
Work is memoised by blob id, cutting parsing 7×. Embedding is keyed by content hash: 6.67M unit slots became 185k distinct texts, about 15 minutes of GPU instead of about 9 hours.
Skip the 50 MB download
With the GPU reranker on, the browser’s own language model added nothing measurable (81.9% vs 81.7%, p = 0.30), so the site doesn’t fetch it when the GPU answers.
What broke
A corrupted embedding store
Earlier LTR and cross-encoder numbers were measured on a corrupted store. They were superseded and the incident was recorded.
An unverifiable baseline, removed
An early performance table compared against code that predated the git history. It couldn’t be re-run, so the comparison was deleted rather than kept.
A zero-shot reranker that added nothing
The cross-encoder only improved results after one LoRA epoch on in-domain hard negatives.
What these numbers don’t prove
- Allfather is level with the best LocAgent agent at function level, behind SweRank, and behind both at file level. Its models are about an order of magnitude smaller.
- The holdout and Lite differ in projects, fix size and date, so comparing them is descriptive rather than controlled.
- On SWE-bench Lite, the browser pipeline reaches 69.1% function hit@10 against the GPU pipeline’s 79.4%.
- The first nightly Python-only run scored 66.3% function hit@10 on 80 fresh bugs, lower than on benchmarks.