Introducing ParseBench V2

Over the past year, ParseBench has done something remarkable. It’s become an industry standard for evaluating the accuracy of document parsing, a critical step in automating knowledge work. We see that adoption in both the scale of community use, with more than 50,000 dataset downloads a month, and the teams citing ParseBench, from NVIDIA, Cohere, and Perplexity to the open-source teams behind LightOnOCR-3, PyMuPDF, and Infinity-Parser2.

At the time of launch, ParseBench-1 was a huge step up in dataset quality and metric coverage compared to previous benchmarks. That said, the deterministic metrics that we defined for ParseBench-1 suffered from a similar issue that has plagued prior benchmarks, they aren’t always aligned with human judgment on how bad a parse error was. For example, a parser that flips “75%” to “57%” changes a critical fact, while returning a chart’s labeled values as a list instead of a table changes only the presentation. Yet in our tests, a harmless formatting change could cost more points than a wrong number.

Today, we’re proud to raise the standard for document parsing benchmarks with ParseBench-2, our most comprehensive document parsing benchmark yet. It is the only benchmark that provides the extreme granularity and repeatability afforded by deterministic metrics, while also providing high alignment with human judgment. This release brings five major improvements to what we evaluate and how we measure parsing accuracy:

• Severity-tuned scoring, an industry-first for document parsing benchmarks. ParseBench-2 is the only benchmark that tunes a deterministic parsing metric by its severity - a human-defined judgment of how consequential errors are for downstream applications.
• Massively Expanded Dataset Scale: We more than doubled the initial document corpus within ParseBench-1, expanding from 2.1k to 4.3k cases. This includes even more real-world enterprise documents with hard tables, charts, forms, and more.
• More reliable reference data. We corrected errors and clarified ambiguous annotations across all five original dimensions so parsers are scored against a more accurate account of the source page.
• Complete chart evaluation. ParseBench-2 checks every value across 1,025 charts. A parser now has to recover the full data series to earn full credit.
• Forms as a sixth dimension. ParseBench-2 checks whether values stay attached to the right fields, checkboxes retain their state, and blank fields stay blank.

Building document parsing benchmarks is hard. Traditional document parsing benchmarks (olmOCR-bench, OmniDocBench) used brittle metrics that overpenalized stylistic errors and underpenalized semantic degradations. Recent LLM-as-a-judge approaches can recognize equivalent representations beyond a fixed rule set, but (a) lack granularity in measuring across specific dimensions, (b) are subject to bias from the judge model and prompt, and (c) lack determinism and repeatability.

We spent countless hours making ParseBench-2 the most comprehensive but fair benchmark out there. We painstakingly crafted 330 (!) error types to measure every possible failure mode of document parsing, assigned severity levels to each error type, and calibrated each parse metric against the severities of every single error. Today, we are excited to open it to the public so that everyone can use, customize, and contribute to an open standard for document parsing in production.

The benchmark dataset, evaluation code, and the full scientific paper behind the study ate all publicly available.

• Dataset: Hugging face
• Code & Evaluation: GitHub
• Research Paper

Building better, severity-weighted accuracy metrics

We wanted the ParseBench-2 metrics for each dimension to reflect how an expert would judge a parser’s output. A text metric should penalize a wrong number more heavily than a minor typo. A table metric should penalize a value assigned to the wrong year and give the same score to equivalent tables in supported formats.

The challenge is that any deterministic metric has inherent biases for ‘good’ or ‘bad’ inputs. Certain document properties will cause a metric to move more or less. For instance, on a standard edit distance metric measured over a text-dense page, swapping the digits in a number will barely cause the number to move (even if it is catastrophic for downstream use cases).

Our proposed solution is to work backwards by first defining the goal, and then hillclimbing our parse metric towards that goal. We first define a large family of 330+ “error types” - these are test scenarios where we start with a correct parsing output, and inject perturbations. Some of these changes are intentionally harmless (extra HTML markup), while others are quite important (header misalignment in a table, or a hallucinated value in a table). We then define a severity policy, or a human judgment of how good or bad each test scenario is. We classified each change as critical, major, minor, or harmless and set a target for how much the score should drop.

We used those tests to calibrate ParseBench-2’s metrics across each of the 6 main dimensions: text, tables, formatting, charts, layout, and forms. We comprehensively tested dozens of metrics per dimension - this includes metrics from ParseBench-1 alongside metrics from other published benchmarks. We then aggressively iterated on each parsing metric to match the severity policy as closely as possible. This is aided through explicit severity caps, which ensure that a serious error carries a substantial penalty even when the rest of the page is correct.

Our resulting metrics in ParseBench-2 are much better aligned than ParseBench-1. For instance, take a 459-cell table within our ParseBench dataset with two scenarios:

• A swapped number (very bad)
• Returning the table in Markdown instead of HTML (harmless).

Under the ParseBench-1 scoring methodology, the number change only penalized the score by 0.13 points out of 100, while returning a correct table in Markdown instead of HTML brought its score to zero. On ParseBench-2, the wrong number now costs 10 points, while the markdown output deducts 0.

A Comprehensive, Customizable Scenario Catalog

We define a comprehensive catalog of 330+ error types, spread across our 6 dimensions of text, formatting, tables, charts, forms, and layout. We then assigned severity levels to every one of these error types.

These test scenarios distinguish missing content, incorrect values, broken associations, invented information, and lost formatting. A misspelled label and a value assigned to the wrong header can receive different severity classifications, even if they change similarly small portions of the output. Equivalent presentations are explicitly classified as harmless.

The severity levels are assigned based on expert judgments of which error types caused failures in production. That said, the severity policy is designed to be very customizable by any user. You can set your own severity levels for any scenario, and choose the parse metrics that align the most with those severities! ParseBench-2 is not only a fixed benchmark, but a framework to designing parse metrics from severity policies so that you can optimize for the dimensions that you care about the most.

Expanding what ParseBench evaluates

ParseBench-2 combines corrected annotations with more complete references and additional challenging datasets across six dimensions.

Text: Columns and sidebars sometimes come back with the right words in the wrong place. ParseBench-2 evaluates transcription and reading order separately, so correctly transcribed text earns credit even when its position in the output changes

Formatting: A strikethrough can mark text as deleted, so losing or inventing it can change how an agent interprets the page. ParseBench-2 checks both preserved and added formatting, while accepting equivalent Markdown and HTML syntax.

Tables: A number under the wrong header is a wrong fact, even if the number itself was read correctly. ParseBench-2 checks every value against its headers and row labels, including multi-level headers and grouped rows.

Charts: ParseBench-1 spot-checked a few plotted values per chart, so an error anywhere else went unseen. ParseBench-2 compares against complete reference tables instead, covering 36,958 values across 1,025 charts.

Visual grounding: Source locations help agents verify extracted content and generate precise citations. ParseBench-2 checks whether a parser locates each block where its content appears on the page. A parser can return a paragraph as one block or line by line and earn full credit either way, provided its boxes point to the right text. The evaluation covers 538 pages.

Forms: Forms now count toward the overall score. ParseBench-2 checks that each value, checkbox state, and signature stays with the right field, and that blank fields stay blank, since an empty box is information too. The core dataset contains 500 pages from 244 government form templates across 30 countries, 450 with synthetic typed values and 50 blank. A separate set adds 100 mostly blank public forms.

OCRia-1.5: A New Pareto Frontier of Price/Performance

In conjunction with ParseBench-2, we’re introducing OCRia-1.5, our new family of document parsing models powering LlamaParse’s Cost Effective, Agentic and Agentic Plus modes. These models represent a significant step up in capability, even compared to OCRia-1 which we released a few weeks ago. They have gotten better across all dimensions, and especially at tables, forms, and visual layout.

We will cover this family of models in a follow-up post. Our research loop is only accelerating, and we continuing to aggressively push on simultaneously improving cost, accuracy, and latency everywhere.

What ParseBench-2 reveals about parser performance

An overall score summarizes performance across the benchmark. The dimension scores show where systems differ, helping teams judge whether a parser’s strengths match the documents they need to process. We ranked 41 default configurations across all six dimensions: 18 document-parser configurations, 14 direct vision-model configurations, and nine open-weight configurations.

LlamaParse Agentic Plus leads overall. Agentic Plus scored 89.8, ahead of the highest-scoring direct vision model at 85.7 and the strongest document parser from another provider at 83.9. LlamaParse Agentic scored 86.0. Agentic Plus led all ranked configurations on formatting, tables, and charts.

Charts show the widest capability gap. Charts are the ultimate capability divider. Scores on charts spanned from 1.2 to 92.6, the widest gap across any evaluated dimension. Most standard parsers and vision models essentially give up on charts, emitting a brief caption or missing numerical data entirely, with five ranked systems scoring below 4. Extracting complete, value-by-value data series from visual graphics remains the single clearest separator between basic OCR and true agentic document engines.

Visual grounding remains a challenge for many frontier models. While frontier multimodal models read prose well, they struggle to pinpoint where elements actually sit on the page. Direct vision models achieved a median Visual Grounding score of 64.5, compared to 83.8 for specialized parsers, with Agentic Plus reaching 87.5. For workflows that require verifiable citations, bounding boxes, or audit trails, specialized parsers remain essential.

Accuracy and processing cost

At production volumes, cost per page affects which level of accuracy a team can afford to deploy. ParseBench reports processing cost alongside accuracy so teams can compare those trade-offs.

Higher prices do not consistently buy higher accuracy. Among specialized parsers scoring roughly 81 to 83, prices ranged from $0.0063 to $0.0300 per page. LlamaParse Agentic scored 83.4 at $0.0125 per page. The strongest competing document parser scored 83.2 at $0.0300 per page, more than twice the price for comparable performance.

LlamaParse advances the accuracy–cost frontier. Agentic Plus delivers higher accuracy at less than half the cost of the strongest direct vision model. It scored 88.4 at $0.0281 per page, compared with 85.0 at $0.0649 per page for that model. That’s a 3.4-point accuracy advantage at approximately 57% lower processing cost.

Alongside ParseBench-2, we’re introducing OCRia-1.5, our new family of document parsing models powering LlamaParse’s Cost Effective, Agentic and Agentic Plus modes. These models continue our work on delivering accurate document parsing across different cost and processing requirements.

Try ParseBench-2

ParseBench-2 gives you a more meaningful measure of parsing accuracy, accounting for both how much a parser gets right and how much its mistakes matter.

The datasets, evaluation code and scoring criteria are open. Download the benchmark, put your own parser to the test and compare its performance across all six dimensions. We’ll keep adding challenging documents and failure cases. If there’s an error we should test, or a severity choice you’d make differently, tell us. Your feedback will help shape what we evaluate next.

Explore the leaderboard or read the research paper

Download ParseBench-2

Get all the documents and ground truth.

git clone https://github.com/run-llama/ParseBench
cd ParseBench
uv sync --extra runners
uv run parse-bench run <pipeline_name> --dataset parsebench-2

Browse the dataset (opens in a new tab)Benchmark code (opens in a new tab)

注意转载授权 · 阅读原文