Towards unboundedness

One of Elicit's core bets is unboundedness: we try to build Elicit so that it can always improve on its answer (or on its confidence in its answer). Unboundedness, or even just less-boundedness, lets Elicit give more useful decision advice than generic AI systems that hit a ceiling early.

To check whether Elicit has this property, we took questions from BioDecision, our benchmark for reasoning through pharma decisions, and asked Elicit and three frontier models to keep iterating on their answers. Elicit kept getting better, while Claude Opus 5.5, Fable 5.2, and GPT-6.1 Sol mostly didn't.

What we did

Every system wrote an answer and then got six more tries at it, for a total of seven versions per question.

We ran Elicit on its Smartest setting. Between versions, it called its independent reviewer, a separate agent that audits the answer. In this experiment, the independent reviewer had a different focus each time. This focus could be the research process in general, or a more specialized one such as whether a surrogate endpoint really predicts the outcome you care about, or whether an observational study holds up as a stand-in for a trial.

We also ran BioDecision on Opus 5.5 and Fable 5.2 (in Claude Code) and GPT-6.1 Sol (through OpenAI's API), all at max effort and with web search. Between iterations, they got the same generic nudge every time: "Review your previous answer, find any problems, and then provide a complete improved answer based on any problems you found."

We graded all of it against the BioDecision answer rubrics, on a 0–100 scale, using Claude Opus 5 (which was the grader we used earlier, so the runs can be compared). To check that the result doesn't hinge on that one grader, we also had GPT-5.6 Terra and Gemini 3.1 Pro grade the first and last versions of every answer, and averaged the three, as we did for our BioDecision launch.

The results

This quick experiment consists of runs on 18 questions for each of the four systems. Elicit's average went from 77 to 87, up 9.3 points. Opus 5.5 went from 75 to 77 (up 2.2), Fable 5.2 from 67 to 69 (up 2.8), and GPT-6.1 Sol from 63 to 64 (up 0.8). Elicit's improvement beat Opus 5.5's by 7.1 points, with a 95% confidence interval of 0.7 to 13.5; it beat Fable 5.2’s by 6.5 (−0.5 to 13.6) an Sol's by 8.5 (2.8 to 14.2).

Averaged across all three graders, Elicit's improvement beat Opus 5.5's by about 7 points (7.2), though with 18 questions the interval just reaches zero (0.0 to 14.5); it beat Fable 5.2’s by 7.1 (1.9 to 12.4) and Sol's by 9.5 (5.1 to 13.9). Every grader saw Elicit gain the most: between 6.7 and 9.4 points, against at most 2.8 for any of the models.

We previously saw the same shape when developing BioDecision, comparing with the previous generation of models: over 37 questions, Elicit gained 9.4 points while Claude Opus 5 gained 3.5 and GPT-6 Astra 2.1 points.

An illustrative example is a tumor-board chair asking whether an observational finding justifies delaying treatment. Elicit's first draft scored 9 points, and six reviews later it scored 45. Opus 5.5 went from 5 to 9, Fable 5.2 from 0 to 5, and GPT-6.1 Sol stayed at 0 the whole way.

What explains it

From reading the transcripts, the frontier models spend later rounds polishing — tightening an argument, fix a number, or reword a recommendation. In one case, Opus 5.5 said at round six that "the analysis has now converged." Fable 5.2 kept finding real problems for longer, but mostly in its own figures and wording; by the last round it went back to the web on only a third of questions, and in one case concluded that "further review passes are unlikely to move the answer." By themselves, frontier models are not always able to smartly figure out what to research in order to improve an answer.

By contrast, Elicit's reviewers keep asking new questions, and Elicit can go back to the literature to answer them. For example, if one reviewer asks whether a biomarker result transfers to the relevant patient population, Elicit can go find out, which is how the final answer scored higher than a first draft that had never considered the issue.

Caveats

This is a quick experiment on only 18 questions. The direction has been consistent across every version of this experiment that we've run, but the intervals are wide. One of the reasons we have fewer results than our earlier 37 BioDecision questions is that Opus 5.5 refuses some biological sciences questions. Also, a few sessions didn’t complete for various reasons, so we left them out.

Fable 5.2, which came out while we were writing this post, ran on a newer version of Claude Code than Opus 5.5, and unlike Opus 5.5 it answered every question.

This experiment is about a slightly altered version of Elicit. A regular Elicit answer is capped at two independent reviews per answer, but here we allowed it to do six in order to study unboundedness.

We also noticed that Elicit's reports are longer than the frontier models' answers. Within Elicit, though, the longer reports actually scored a bit lower, so length doesn't seem to be doing the work.

Rather, it is Elicit’s thoughtful harness — informed by what is needed to improve answers to scientific questions — that allows it to turn additional compute into further improvement, compared to generic prompting of frontier models.

Thanks to Ben Rachbach for the first version of this post.

注意转载授权 · 阅读原文