AutoMark：让自主科研代理发现更优的 LLM 水印方案

AutoMark: Enabling Autoresearch to
Discover Better LLM Watermarks

Abstract

With LLM watermarking being deployed commercially and now required by regulations, improving its reliability and effectiveness has become crucial. Yet, recent progress in the field of LLM watermarking has increasingly been driven by improving details of existing methods, an effort fundamentally limited by the pace of human researchers. In this work, we enable for the first time the autonomous discovery of new distortion-free state-of-the-art watermarking schemes. To enable this, we (i) establish strict criteria to ensure that watermarks are reliable (e.g., they do not have an unexpectedly high false positive rate), (ii) propose rigorous statistical tests to automatically evaluate whether a watermarking scheme satisfies our criteria, and (iii) design an evaluation suite to rank watermarks along three key dimensions: detectability, quality, and robustness. By running our framework with 3 frontier models (GPT-6 Astra, Opus 5, Gemini-3.8 Flash), we discover over 50 different watermarking schemes, including several that outperform prior works along all key dimensions. We complement this by a manual study of the discovered schemes, distilling the key ideas into smaller components, and individually studying the impact of each component across dimensions (detectability, quality, robustness) to better understand how the proposed schemes operate. Importantly, we find that the agents, on top of improving existing ideas, also discover fundamentally new ideas (e.g., aligning watermark scores with random per-request direction). Overall, our work establishes the first steps of fully autonomous watermarking research, enabling the discovery of more reliable and effective watermarks. Our code is available here, and a blogpost to visualize our results here.

1 Introduction

Large Language Model (LLM) watermarking, which embeds a signal invisible to humans in model outputs, has emerged as the standard for tracing AI-generated text: regulators mandate it (European Parliament and Council of the European Union, 2024), and major providers deploy it (Dathathri et al., 2024; Anthropic, 2026). This makes research on new watermarking schemes crucial as they have to be reliable enough to prevent false accusations and robust enough to prevent evasion across real-world deployments. Recent progress has been steady but limited with improvements mostly driven by refining existing designs, e.g., with entropy weighting (Lee et al., 2024), or key switching (Sander et al., 2026), and the pace of iteration capped by that of human researchers. Autonomous research agents are thus a natural and promising direction for further accelerating LLM watermarking research.

Autoresearch

Autoresearch systems let agents propose, implement, and evaluate ideas in a loop (Romera-Paredes et al., 2024; Novikov et al., 2025), and have already led to scientific breakthroughs (OpenAI, 2026). Fields with readily verifiable outputs, such as mathematics with Lean-generated proofs, are especially well suited to this approach. Watermarking can also benefit from autoresearch, but its evaluation requires more care: useful properties such as detectability, robustness, and quality trade off, so improvement on a single metric need not represent a better watermark. Also, agents may exploit gaps in the evaluation harness (i.e., reward hacking), e.g., artificially inflating detectability/robustness by increasing the rate of falsely flagged human texts. A separate concern is that agents may produce results faster than humans can understand them, especially when their explanations are poor, making the resulting solutions difficult to trust. In this work, we show that autoresearch, when carefully executed with an adequate harness and methodology, discovers watermarks that are trustworthy, explainable, and Pareto-superior to existing watermarking schemes.

This work: autoresearch for LLM watermarks

To enable autoresearch for watermarking, we mathematically formalize what makes a watermark valid (Section 3.2) using two criteria: soundness in expectation over the private key, and, given a fixed text distribution, the probability of sampling a key that leads to false flags should be small. Given these criteria, we design our autoresearch harness illustrated in Figure 1. An agent is fully isolated in an environment and discovers new watermarking schemes (left). It can submit its scheme to a private verification pipeline (middle) that (i) tests whether the proposed scheme violates our criteria (using tailored tests we designed in Section 3.3), and, upon passing, (ii) evaluates the schemes along three key axes (detectability, robustness, and quality).

Running our harness on three state-of-the-art agents, GPT-6 Astra, Opus 5, and Gemini-3.8 Flash, we generated over 50 schemes, with some outperforming prior work on all our evaluated axes. To understand the key insights of these schemes, we manually classify them all into five atomic components (Figure 1, right) and ablate their individual effects (Sections 4.2 and A). We also select two schemes, ConcordMark and RotationFirst, that we evaluate on a held-out evaluation set (Section 4.3). We find that ConcordMark achieve state-of-the-art robustness and is Pareto-superior to all evaluated prior work, whereas RotationFirst has performance similar to prior work at much higher text quality.

Main contributions

Our main contributions are:

• •

  We formalize the criteria for reliable watermarking, and design statistical tests that automatically detect violations (Sections 3.2 and 3.3).

• •

  We build an autoresearch harness for LLM watermarking around this verification (Section 3.4).

• •

  We show that agents running our harness push the Pareto frontier of LLM watermarks, with two new schemes that outperform prior work on a held-out evaluation (Sections 4.1 and 4.3).

• •

  We distill the agents’ schemes into 39 new components, ablate each of them, and identify fundamentally new watermarking ideas (Section 4.2).

2 Related Work

LLM watermarking

LLM watermarks modify the sampling procedure of an LLM to embed a (key-dependent) signal, that a later statistical test can detect. Red-Green watermarks (Kirchenbauer et al., 2023) achieve this by distorting the next-token distribution through a pseudo-random green token bias. In contrast, recent schemes, including all in this work, are distortion-free, i.e., they preserve the next-token distribution in expectation over the key. AAR (Aaronson, 2023) uses the Gumbel-max trick, KTH (Kuditipudi et al., 2024) inverse-transform sampling, DiPMark and MCMark (Wu et al., 2024; Chen et al., 2025) reweight the distribution based on score-rankings, and SynthID (Dathathri et al., 2024), deployed in production, uses tournament sampling. Most recently, TextSeal (Sander et al., 2026) extends Gumbel-max sampling with dual-key generation and entropy-weighted detection.

Watermark properties

Beyond detectability, benchmarks and toolkits (Piet et al., 2025; Tu et al., 2024; Pan et al., 2024) evaluate watermarks along several, often competing, properties. Robustness measures whether the watermark remains detectable after the text is edited, e.g., by word substitutions, translation, or paraphrasing (Kirchenbauer et al., 2024; Kuditipudi et al., 2024). Quality measures the impact of the watermark on the generated text, e.g., on perplexity or downstream tasks (Tu et al., 2024). Distortion-free schemes preserve the distribution of each generation, but, as a repeated context always yields the same scores, they reduce output diversity across generations (Gloaguen et al., 2026) which noticeably affects quality. Security measures whether an adversary can learn the watermark rules to spoof or scrub it (Jovanović et al., 2024), and is outside the scope of this work. Lastly, reliability requires the false positive rate of the detector to be controlled, e.g., Fernandez et al. (2023) show that the z-tests used in early works underestimate the false positive rate, and propose exact tests with de-duplication of repeated (context, token) pairs. Yet, in prior works, these guarantees hold only in expectation over the private key: a specific deployed key can still exhibit a high false positive rate.

Autoresearch

Program search systems (Romera-Paredes et al., 2024; Novikov et al., 2025; Lange et al., 2026) let LLMs iteratively propose programs scored by an automatic evaluator, leading to new results in verifiable domains such as algorithm design and mathematics (OpenAI, 2026). Other works automate the entire research loop, from generating ideas to writing papers (Lu et al., 2026), discovering new training objectives (Lu et al., 2024), or improving LLM training (Karpathy, 2026), and benchmarks measure the research capabilities of agents (Chan et al., 2025; Wijk et al., 2025). Yet, all require a strict evaluator as otherwise agents may exploit loopholes, e.g., by modifying tests instead of solving the task (Zhong et al., 2026). To our knowledge, we are the first to propose an automated pipeline to verify and evaluate watermarks, enabling autoresearch for LLM watermarking.

3 Method

In this section, we introduce the necessary background on LLM watermarking (Section 3.1), explain our design for automatically verifying watermarking schemes (Section 3.3), and finally detail our autoresearch framework (Section 3.4).

3.1 Preliminaries on LLM Watermarks

In this part, we describe the four key components of most watermarking algorithms.

Mathematical description

Let Σ be the finite vocabulary, and let ω∈Σ∗ be a finite sequence of tokens. At each step t of autoregressive generation, the LLM returns pt∈Δ⁡(Σ), the conditional probability distribution of the next token given ω<t. Without a watermark, the next token is sampled according to pt. With a watermark, we assume the existence of a sequence (Uat) of pseudo-random vectors in [0,1]|Σ| whose entries Uat,v are i.i.d. uniform variables seeded by the context hash at, and a watermark function f:[0,1]|Σ|×Δ⁡(Σ)→Δ⁡(Σ) that transforms the next-token probability distribution pt into a watermarked probability distribution f⁡(U~at,pt). We refer to f as the logits transformation and (Uat) as the watermark score, where (U~at) are i.i.d. uniform variables derived from (Uat) and in most cases U~at=Uat. We give concrete examples in Section D.4. The next token ωt, sampled according to f⁡(U~at,pt), will be correlated with the sequence (Uat).

For detection, given any text ω∈Σ∗, we recover the corresponding scores Uat,ωt at retained positions t∈D, with N:⁣=|D|, after deduplicating repeated (seed, token) pairs. By assumption, under the null hypothesis (i.e., the text was generated without knowledge of the watermark), the retained scores Uat,ωt are i.i.d. uniform random variables, whereas when the watermark is used, they are not. We can therefore build a statistic and a corresponding statistical test to detect the watermark, i.e., a detector.

Additional requirements

Furthermore, we require the watermark to be distortion-free,

This property ensures that a watermark has minimal impact on quality: with perfect randomness, the quality of the watermarked model is indistinguishable from its base model. Yet, in practice, as we explain below we can still observe mild quality degradation within distortion-free watermarks.

Practical considerations

In practice, the sequence (Uat) needs to be computed from the sequence of tokens ω so that it can be recovered at detection time. A typical way to derive (Uat) is to use a hash of the preceding tokens and a private watermark key ξ to obtain a seed at for a pseudorandom number generator (App. G). The resulting scores Uat,v remain approximately i.i.d. across distinct (seed, token) pairs and can be derived directly from the text given the private key. We refer to this as context seeding. Seeding creates some practical problems: in case of a repeated (seed, token) pair, the watermark scores are no longer i.i.d., making detection unsound under the null. For a single request, we use a cache mechanism: if a (seed, token) pair is repeated we no longer apply the watermark. For repeated requests however, we cannot apply this cache as in the limit, the watermark is no longer applied. This lack of i.i.d. has also direct implications for the distortion-free property (Equation 1) which in practice is therefore also violated across a larger corpus with repeated tokens. Hence, even for theoretically distortion-free schemes, we have to measure their impact on quality.

3.2 What Defines a Valid Watermarking Scheme

To automatically verify watermarking scheme validity at scale, we first need to formally define what makes a watermarking scheme valid. In particular, we identify two key criteria: soundness over the key and soundness given a key. Though our definitions build upon prior work, we find in App. C that several existing schemes fail our more stringent validity checks. In App. B, we further motivate our criteria by observing the failure modes of schemes designed by autoresearch without these safeguards.

Soundness over the key

Soundness over the key is the property that, in expectation over the watermarking key ξ, the detector is a valid statistical test. Formally we require,

where Pξ​(ω) is the p-value returned by the detector. Most prior watermarking schemes satisfy this property, which distinguishes statistical watermarks from zero-shot LLM-text detectors. However, this property alone does not guarantee that the watermarking scheme is reliable.

For instance, consider a deliberately naive watermarking scheme: using the private key ξ, we randomly partition the vocabulary into a hundred classes and declare a text watermarked if its last token lies in the first class. Averaged over the key, this occurs with probability at most 1/100, so the scheme satisfies Equation 2. Yet its FPR can be high for a specific key: if, for some ξ, "." lies in the first class, most human texts will be classified as watermarked. Thus, while Equation 2 bounds the FPR in expectation over the private key, it does not ensure a well-behaved FPR for each fixed key.

Soundness given a key

We therefore introduce a novel second requirement for watermarking schemes: soundness given a key. Ideally, we want to require that for a human text distribution and a p-value threshold α, the false positive rate is bounded by α. This is arguably impractical as it would require precisely specifying the human text distribution, which, e.g., noticeably differs between languages. Instead, we move the randomness to the key: we require that, for every distribution ℋ0 over texts, the probability of sampling a key whose FPR on ℋ0 exceeds α is at most β.

For β=5%, if a watermarking scheme satisfies Equation 3, then for any fixed human text distribution, the probability of sampling a key for which the detector is not well calibrated (on that distribution) is at most 5%. Interestingly, as we show in App. I, a scheme satisfying Equation 3 can be derived from a scheme satisfying Equation 2 at the cost of its power: The same scheme with its p-value multiplied by 1/β also satisfies Equation 3 (while still satisfying Equation 2).

3.3 Automatic Verification

Now we describe how we automatically test whether a proposed watermarking scheme violates our criteria (Section 3.2). To this end, we design three statistical tests to test whether a scheme is distortion-free (Equation 1), sound over the key (Equation 2), and sound given a key (Equation 3).

1. 1.

   Distortion-freeness (Algorithm 1): for given unwatermarked next-token distributions, we average the watermarked distributions over independently sampled private keys and test whether it significantly differs from the original. We check both individual probability shifts and smaller shifts spread across tokens, using independent samples to identify and test a potential bias direction. If either test detects a significant deviation after multiple testing corrections, the scheme is rejected.

2. 2.

   Soundness over the key (Algorithm 2): for a given set of non-watermarked texts, the test verifies for independently sampled private keys whether the empirical false positive rate is abnormally high. If it is, it means that the scheme likely does not satisfy Equation 2 and thus is rejected.

3. 3.

   Soundness given a key (Algorithm 3): for independently sampled private keys, we sample a separate set of non-watermarked texts (human or LLM-generated) per key and screen for an abnormally high empirical false positive rate. We then test whether too many keys screen positive, accounting for both the allowed fraction β of bad keys and the probability that a sound key screens positive. If they do, the scheme likely violates Equation 3 and is rejected.

3.4 A Framework for Watermark Autoresearch

Next, we describe our harness designed around three components: a dedicated vLLM watermarking library, a coding agent environment, and the automatic verification & evaluation pipeline (Section 3.3).

Research harness

We illustrate the autoresearch loop in Figure 1. Our watermarking library is designed around four primary components: the context seeding, the watermark score, the logits transformation, and the detector, following the formalization described in Section 3.1. A watermark is an implementation of all 4 components. We defer additional design choices to App. E.

The research environment is initialized with the watermarking library and the baseline evaluations. The agent is tasked with proposing a watermarking algorithm that outperforms prior schemes in several key aspects: raw detectability and robustness to several attacks (e.g., word substitution, word deletion, paraphrasing, and backtranslation) while remaining distortion-free and statistically sound. We detail the initial prompt in Section J.1. We also give the agent a public configuration specifying how to evaluate and verify the proposed scheme. Once the agent completes its task, its proposed scheme is automatically evaluated (with a private configuration), and the evaluation results are added to the environment alongside the baselines. Then, a new agent takes over with access to all prior results and code modifications. The loop continues indefinitely. Inside the environment, the agent has full access to the Internet (e.g., it can download third-party libraries or read papers), a dedicated GPU, and API keys for third-party AI providers (primarily for experimenting with paraphrasing attacks).

To stimulate the agent even more, we run a two-phase setup. The first agent is told it has 4 available submissions to the private evaluation before being shut down: this allows it to set up its environment, and adjust in case of unexpected failures. Then, after the initial 4 submissions, subsequent agents are told they only have 1 submission left. We found in Section E.2 that this combined setup led to more creative solutions from the agents and less hyperparameter tuning.

4 Evaluation

In this section, we analyze the results of running our autoresearch framework across state-of-the-art agents. In Section 4.1 we analyze the trajectories of the agents during the autoresearch loop, while in Section 4.2 we break down the schemes proposed by the agents into individual categories and assess which metrics they impact. Lastly, in Section 4.3, we select two watermarking algorithms from the autoresearch runs that outperform prior work on our evaluated metrics.

Experimental setup

We compare against the most popular prior and deployed schemes: AAR, SynthID, and TextSeal. Our private grader measures detectability and robustness as the TPR@1%FPR on 1000 Llama-3.1-8B 200 to 300 tokens long replies to ELI5 (Fan et al., 2019) prompts, on clean text and under four attacks: word deletion, synonym substitution, back-translation, and paraphrasing. For quality, we measure the distance to the unwatermarked model in perplexity, and in diversity (Self-BLEU) across 100 replies to the same 10 prompts, and, as these metrics are highly correlated, sometimes average them into a single quality metric (see App. K for un-aggregated results). Each metric is the worst over 5 watermark keys to mitigate across-key variance. We defer details to App. D. Visualizations of our evaluation results are available in our blogpost.

4.1 Autoresearch Trajectories

We first focus on how the metrics evolved over the number of submissions to the evaluation harness.

Research agents and settings

We use four different research agents: three frontier models GPT-6 Astra (xhigh), GPT-6 Astra (low), Opus 5, and a state-of-the-art efficient model Gemini-3.8 Flash (high). We let each agent run until its submissionbudget was exhausted. Using standard API prices, the research-agent costs are $357.08 for GPT-6 Astra (xhigh), $123.29 for GPT-6 Astra (low), $435.96 for Opus 5, and $13.81 for Gemini-3.8 Flash (high).

Main results

Figure 2 shows the evolution of robustness to paraphrasing and of the watermark quality for different agents, with the baselines for comparison, for a total of 52 watermark schemes being evaluated. We find that all tested agents manage to improve their submissions over time, outperforming baselines in most cases. For two baselines (AAR and SynthID), the evaluated configurations do not pass our soundness given a key check: as we show in App. C this means that their robustness is slightly inflated. Moreover, only the submissions from GPT-6 Astra (both low and xhigh) are provably sound given a key. In contrast, submissions from Opus 5, Gemini-3.8 Flash and the TextSeal baseline configuration satisfy the property only empirically (i.e., they do not fail our check). This means that the p-values of GPT-6 Astra are overly conservative, and, as we find in App. C, calibrating them to simply pass our empirical check increases the paraphrasing TPR of their best submission by 13–15 percentage points, reaching robustness similar to that of Opus 5.

Research dynamics

The research processes of GPT-6 Astra and Opus 5 were similar, proposing a range of novel and creative watermarking ideas (App. A). Their first submissions mostly consisted of tweaks to the TextSeal baseline and attempts to correlate both the private verification and evaluation results with their local evaluation set-up. Both GPT-6 Astra agents then decided to submit only schemes that are provably sound given a key by multiplying their p-values by 1/β (Section 3.2) whereas Opus 5 opted for empirical calibration on its public verification evaluation, which in practice turned out to correlate well with the private verification. Lastly, Gemini-3.8 Flash was significantly less creative. All its submissions consisted of hyperparameter tuning of the TextSeal watermark to improve the metrics and did not result in any new ideas.

4.2 Breakdown of the Agent Research Directions

Next, we manually go through the schemes proposed by all agents, decompose them into five components (the four components from Section 3.1 and the watermark unit, a generalization introduced by the agents), and evaluate the impact of each component on watermark properties. We summarize the decomposition in Figure 3, and provide a detailed explanation of each component in App. A.

Watermark unit

Most watermarks operate at the token level: each token is pseudo-randomly scored and the watermark modifies the logits according to both the score and the model probability distribution (Section 3.1). We find that the agents proposed to generalize this idea: let 𝒢:Σ→ℕ be a mapping from the token space to a set of equivalence classes. In any watermark algorithm, one can replace a token v with its class 𝒢⁡(v) and, given the next-token probability distribution pt∈Δ⁡(Σ), define the probability of each class g∈𝒢⁡(Σ) as Pr[g]=∑v∈Σpt(v)𝟙{𝒢(v)=g}. If the watermark decides to sample a class g, the actual token is then sampled according to the residual probability

For instance, using a Lexical group as 𝒢 naturally increases the scheme robustness but slighly lowers quality: if tokens from the same lexical group are swapped, the context remains unchanged.

Context seeding

For context seeding, agents explore the trade-off alluded to in Kirchenbauer et al. (2024): increasing the context size lowers the robustness but improves the quality. Also, for small contexts, because of deduplication during detection (to guarantee soundness over the key (Section 3.1)), lowering the context size may also hurt detectability. Historically, prior works have converged on hashing the tuple of the candidate token and the k preceding tokens (Fixed-length in Figure 3). Agents iterated upon this idea along three main axes. The first idea is that tokens should not be treated equally: less frequent tokens should be dynamically assigned a lower k. This increases robustness without impacting quality significantly (because the distortion-free guarantee ensures that quality degrades only when contexts are repeated across texts). The second idea is to dynamically increase the context size: the first time we use a tuple as a context, it is of size 1. The next time we are about to use the same tuple, we increase the context size until the context has not been seen yet. This keeps the robustness benefits of short contexts without lowering the detectability. The third axis is more creative: assign tokens to categories, and when sampling a token from a given category, use a fixed-window context over the previous tokens from that category only. This increases robustness: edits in one category do not change the context for other categories.

Score

The score refers to the pseudo-random variable used by the logits transformation. For instance, with Red-Green watermarks, the scores are the colors of the tokens. Without loss of generality, as in Section 3.1, we refer to the seeded watermark score as Ut∼𝒰​([0,1])|Σ| and the final score used in the logits transformation as U~t. With different score designs, the agents tried to increase the quality while minimizing the impact on detectability. To do that, they consistently add a small amount of true randomness to the watermark. This is conceptually similar to the approaches by Sander et al. (2026); Dathathri et al. (2024) which also add true randomness to the watermark sampling to improve quality. Yet, we find that agents explore concepts that substantially differ from prior works. With the Gaussian copula, they propose sampling random Gaussian noise ηt∼𝒩⁡(0,I|Σ|) and applying the following transformation

where ρ∈[0,1] is a scheme parameter that trades off detectability for quality (in the limit, ρ=0 has no watermark). In Figure 4, we compare this approach to TextSeal’s random key switching (keeping all other watermarking components fixed) and find that it is strictly better: it outperforms TextSeal along all evaluated dimensions. With Channels, we use a finite pool of private keys and for each request randomly decide which key to use. Detection is run with every key and uses the Bonferroni correction. Private geometries are more complex, and we explain them further in App. A.

Logits transformation

For logits transformations, most proposed schemes settle on Gumbel-max sampling (Gumbel race) from Aaronson (2023): as proven in Gloaguen et al. (2026), it maximizes watermark detectability. Yet, we find that the agents commonly propose a Gumbel-max sampling extension via residual clocks (ResidualRace). Given a context seed at, let Uat be the uniform watermark score. The first time at is visited, we initialize residual clocks Rat=−log⁡Uat; otherwise, we use previously computed scores. Then we sample the next token according to the exponential race:

and replace the winning token’s score with a fresh exponential random variable: Rat​(u∗)∼Exp⁡(1). In Figure 5, we compare ResidualRace against the Gumbel race, keeping all other components fixed. ResidualRace improves robustness against every attack at a slight cost to quality. Yet, adding Gaussian copula noise to the initial clocks recovers the quality of the Gumbel race while retaining most of the robustness gains. We find that most other logits transforms proposed by agents are designed to only increase the quality of the schemes at the expense of detectability. As we show in App. A this generally offer worse trade-offs than e.g., using the Gaussian copula.

Detection

Most agents initially converged independently on the Gamma detector. Let ω∈Σ∗ be a sequence of tokens and Ut​(ωt) the corresponding uniform watermark score at position t. Then we have

which follows a Gamma​(|ω|,1) distribution under the null. As we show in Figure 6 (left), they further improve it with Clustering and Excess thresholding. Lastly, using a model forward pass during detection to estimate the model probabilities can further improve detectability by a significant margin over entropy weighting, a prior method that also relies on a forward pass during detection (Figure 6, right). For Private geometries, which use a different class of detectors, we defer their full evaluation to App. A.

4.3 Independent Evaluation of the Best Directions

Next, we highlight two state-of-the-art schemes proposed by the agents that outperform prior works: ConcordMark as a robust watermarking scheme, and RotationFirst as a scheme with a strong quality focus. We give an in-depth description of both schemes and provide additional evaluation in App. H.

Experimental setup

To mitigate overfitting due to the autoresearch process, we use a different evaluation protocol detailed in Section D.5. Notably, we generate 1000 replies to English prompts, and 1000 replies to Chinese prompts from WildChat (Zhao et al., 2024a), use DeepSeek-v4 Flash for backtranslation and paraphrasing, and evaluate replies generated by both Llama-3.1-8B and Qwen3.8-27B. Lastly, we pool the results over 5 watermarking seeds and report the worst result. For consistency with the baselines, we do not apply the soundness given a key correction (Section 3.2).

Main results

Table 1 compares the two selected agent schemes with the three baselines. We bold the results from the agent schemes that are better than the best baseline, and highlight the best results of each column in green. Each cell reports the metrics on English text (left) and Chinese text (right). For each scheme, high means the detector requires a model forward pass. We see that ConcordMark (high) is Pareto better than all the baselines: on almost all metrics, it outperforms the best baseline. RotationFirst significantly outperforms all schemes on all quality metrics: its diversity is very close to the unwatermarked model. At the same time, it keeps relatively strong detection and robustness metrics making this scheme very suitable for model providers that desire strong watermarking with the smallest impact on quality. Importantly, for both agent schemes, the results from the autoresearch evaluation (Figure 2) transfer to new settings with different prompts and attackers. This suggests that autoresearch, with adequate verification, is a reliable method for improving watermarking schemes.

5 Conclusion and Limitations

In this work, we enable for the first time autoresearch for LLM watermarking with rigorous validity criteria (Section 3.2), statistical tests to verify them (Section 3.3), and a tailored harness (Section 3.4). We show that frontier agents go beyond hyperparameter tuning to introduce novel ideas (Section 4.2), with two of their schemes, ConcordMark and RotationFirst, outperforming prior work on held-out data (Section 4.3).

Limitations

In autoresearch, the key challenge is to build powerful verification that agents can’t exploit, and that faithfully transcribes the desired property (i.e., in our case a watermark that is robust and with controlled false positives given a key). For watermarking, there are many desirable properties we did not include in this work e.g., security, radioactivity, applicability to open-weights setting, or extension to the multibit setting. We believe that, future research, should not focus on establishing new schemes but rather on how to build verifier and evaluation for those properties.

LLM Usage

The goal of this work is to enable coding agents to discover novel and better watermarking schemes: hence, all the new watermark proposals in this work have been designed by fully autonomous coding agents. Yet, organizing the schemes into individual components and understanding their impact on watermark properties have required significant manual efforts. Notably, all ablation evaluations of the components have been designed by humans and executed with AI-assistance. The autoresearch harness used in this work has been coded with AI-assistance. The main paper has been written by hand, with AI-assistance for polishing the writing and the visuals. The proofs have been LLM-generated, then proofread and edited by humans.

Ethics Statement

This paper enables autonomous agents to discover new LLM watermarking schemes, which we believe has a positive societal impact: better watermarks support the provenance, auditing, and mitigation of large-scale misuse of AI-generated text. While our verification tests screen for schemes with unsound false positive rates (Section 3.3), LLM watermarks should not be used as sole evidence in high-stakes decisions. We also acknowledge that publicly disclosing new schemes, along with a detailled component-level analysis, may facilitate targeted attacks on them. Yet we argue that the benefits of advancing research in this area clearly outweigh the risks.

Reproducibility Statement

Our autoresearch harness is described in Section 3.4, and all details of the verification, the evaluation, the research agents (including their versions and reasoning efforts), and the baselines are given in App. D, with the full agent prompt in App. J. We fully specify the agent-generated components in App. A and the two selected schemes, ConcordMark and RotationFirst, in App. H. We also release our code, including the harness, the watermarking library, each agent final environment, a standalone implementation of all described components, and a standalone optimized implementation for ConcordMark and RotationFirst.

References

• Aaronson (2023) Scott Aaronson. Watermarking of large language models. Talk at the Simons Institute for the Theory of Computing, August 2023. URL https://simons.berkeley.edu/talks/scott-aaronson-ut-austin-openai-2023-08-17.
• Anthropic (2026) Anthropic. How Claude’s text watermark works. Anthropic News, August 2026. URL https://www.anthropic.com/news/claude-text-watermark.
• Bahri & Wieting (2026) Dara Bahri and John Wieting. A watermark for black-box language models. Transactions on Machine Learning Research, 2026. URL https://arxiv.org/abs/2410.02099.
• Chan et al. (2025) Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. MLE-bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=6s5uXNWGIh.
• Chen et al. (2025) Ruibo Chen, Yihan Wu, Junfeng Guo, and Heng Huang. Improved unbiased watermark for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 20587–20601, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-long.1005. URL https://aclanthology.org/2025.acl-long.1005/.
• Dathathri et al. (2024) Sumanth Dathathri, Abigail See, Sumedh Ghaisas, Po-Sen Huang, Rob McAdam, Johannes Welbl, Vandana Bachani, Alex Kaskasoli, Robert Stanforth, Tatiana Matejovicova, Jamie Hayes, Nidhi Vyas, Majd Al Merey, Jonah Brown-Cohen, Rudy Bunel, Borja Balle, Taylan Cemgil, Zahra Ahmed, Kitty Stacpoole, Ilia Shumailov, Ciprian Baetu, Sven Gowal, Demis Hassabis, and Pushmeet Kohli. Scalable watermarking for identifying large language model outputs. Nature, 634(8035):818–823, October 2024. doi: 10.1038/s41586-024-08025-4. URL https://doi.org/10.1038/s41586-024-08025-4.
• Davies (1987) Robert B. Davies. Hypothesis testing when a nuisance parameter is present only under the alternative. Biometrika, 74(1):33–43, 1987. doi: 10.1093/biomet/74.1.33.
• European Parliament and Council of the European Union (2024) European Parliament and Council of the European Union. Regulation (EU) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence and amending regulations (EC) no 300/2008, (EU) no 167/2013, (EU) no 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act). Official Journal of the European Union, June 2024. URL https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng.
• Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558–3567, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1346. URL https://aclanthology.org/P19-1346/.
• Fernandez et al. (2023) Pierre Fernandez, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. Three bricks to consolidate watermarks for large language models. In 2023 IEEE International Workshop on Information Forensics and Security (WIFS), pp. 1–6. IEEE, December 2023. doi: 10.1109/WIFS58808.2023.10374576. URL https://doi.org/10.1109/WIFS58808.2023.10374576.
• Gloaguen et al. (2026) Thibaud Gloaguen, Robin Staab, Nikola Jovanović, and Martin Vechev. A unified framework for LLM watermarks. arXiv preprint arXiv:2602.06754, February 2026. doi: 10.48550/arXiv.2602.06754. URL https://arxiv.org/abs/2602.06754.
• Jovanović et al. (2024) Nikola Jovanović, Robin Staab, and Martin Vechev. Watermark stealing in large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 22570–22593. PMLR, 2024. URL https://proceedings.mlr.press/v235/jovanovic24a.html.
• Karpathy (2026) Andrej Karpathy. autoresearch: AI agents running research on single-GPU nanochat training automatically. GitHub repository, March 2026. URL https://github.com/karpathy/autoresearch.
• Kirchenbauer et al. (2023) John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 17061–17084. PMLR, 2023. URL https://proceedings.mlr.press/v202/kirchenbauer23a.html.
• Kirchenbauer et al. (2024) John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. On the reliability of watermarks for large language models. In The Twelfth International Conference on Learning Representations, pp. 49660–49704, 2024. URL https://openreview.net/forum?id=DEJIDCmWOz.
• Kuditipudi et al. (2024) Rohith Kuditipudi, John Thickstun, Tatsunori Hashimoto, and Percy Liang. Robust distortion-free watermarks for language models. Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=FpaCL1MO2C.
• Lange et al. (2026) Robert Tjarko Lange, Yuki Imajuku, and Edoardo Cetin. ShinkaEvolve: Towards open-ended and sample-efficient program evolution. In The Fourteenth International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2509.19349.
• Lee et al. (2024) Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking for code generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4890–4911, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.268. URL https://aclanthology.org/2024.acl-long.268/.
• Lu et al. (2024) Chris Lu, Samuel Holt, Claudio Fanconi, Alex J. Chan, Jakob Foerster, Mihaela van der Schaar, and Robert Tjarko Lange. Discovering preference optimization algorithms with and for large language models. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://arxiv.org/abs/2406.08414.
• Lu et al. (2026) Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of AI research. Nature, 651:914–919, March 2026. doi: 10.1038/s41586-026-10265-5. URL https://doi.org/10.1038/s41586-026-10265-5.
• Novikov et al. (2025) Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmeet Kohli, and Matej Balog. Alphaevolve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131, 2025.
• OpenAI (2026) OpenAI. Finite time blowup for the euler equation, September 2026. URL https://cdn.openai.com/pdf/315b36cd-ec98-4023-8342-93345194ece1/euler.pdf. Accessed: 2026-09-22.
• Pan et al. (2024) Leyi Pan, Aiwei Liu, Zhiwei He, Zitian Gao, Xuandong Zhao, Yijian Lu, Binglin Zhou, Shuliang Liu, Xuming Hu, Lijie Wen, Irwin King, and Philip S. Yu. MarkLLM: An open-source toolkit for LLM watermarking. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 61–71, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-demo.7. URL https://aclanthology.org/2024.emnlp-demo.7/.
• Piet et al. (2025) Julien Piet, Chawin Sitawarin, Vivian Fang, Norman Mu, and David Wagner. Mark my words: Analyzing and evaluating language model watermarks. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 68–91. IEEE, April 2025. doi: 10.1109/SaTML64287.2025.00012. URL https://doi.org/10.1109/SaTML64287.2025.00012.
• Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67, 2020.
• Romera-Paredes et al. (2024) Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 625(7995):468–475, 2024. doi: 10.1038/s41586-023-06924-6. URL https://doi.org/10.1038/s41586-023-06924-6.
• Salmon et al. (2011) John K. Salmon, Mark A. Moraes, Ron O. Dror, and David E. Shaw. Parallel random numbers: As easy as 1, 2, 3. In Proceedings of the 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’11, pp. 1–12. ACM, November 2011. doi: 10.1145/2063384.2063405. URL https://doi.org/10.1145/2063384.2063405.
• Sander et al. (2026) Tom Sander, Hongyan Chang, Tomáš Souček, Tuan Tran, Valeriu Lacatusu, Sylvestre-Alvise Rebuffi, Alexandre Mourachko, Surya Parimi, Christophe Ropers, Rashel Moritz, Vanessa Stark, Hady Elsahar, and Pierre Fernandez. TextSeal: A localized LLM watermark for provenance & distillation protection. arXiv preprint arXiv:2605.12456, May 2026. doi: 10.48550/arXiv.2605.12456. URL https://arxiv.org/abs/2605.12456.
• Tu et al. (2024) Shangqing Tu, Yuliang Sun, Yushi Bai, Jifan Yu, Lei Hou, and Juanzi Li. WaterBench: Towards holistic evaluation of watermarks for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1517–1542, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.83. URL https://aclanthology.org/2024.acl-long.83/.
• Wijk et al. (2025) Hjalmar Wijk, Tao Roa Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Joshua M. Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Jun Koba Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. RE-bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 66772–66832. PMLR, 2025. URL https://proceedings.mlr.press/v267/wijk25a.html.
• Wu et al. (2024) Yihan Wu, Zhengmian Hu, Junfeng Guo, Hongyang Zhang, and Heng Huang. A resilient and accessible distribution-preserving watermark for large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 53443–53470. PMLR, 2024. URL https://proceedings.mlr.press/v235/wu24h.html.
• Zhao et al. (2024a) Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=Bl8u7ZRlbM.
• Zhao et al. (2024b) Xuandong Zhao, Prabhanjan Ananth, Lei Li, and Yu-Xiang Wang. Provable robust watermarking for AI-generated text. In International Conference on Learning Representations, volume 2024, pp. 43738–43772, 2024b. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/beae9ed5316bcc48e616754c06c11875-Paper-Conference.pdf.
• Zhong et al. (2026) Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. ImpossibleBench: Measuring LLMs’ propensity of exploiting test cases. In The Fourteenth International Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2510.20270.

Appendix A Breakdown of All Generated Components

In this section, we explain each of the watermarking components proposed by the research agents and visualized in Figure 3. In Section A.1, we explain in depth the decomposition of watermarks in 5 components (watermark unit, context seeding, scores, logits transformation, and detectors), and detail the notation used throughout. In the remaining subsections, we follow the architecture of Figure 3 and go through each component one by one: the watermark unit (Section A.2), context seeding (Section A.3), the watermark score (Section A.4), the logits transformation (Section A.5), and detection (Section A.6).

A.1 Preliminaries and Notation

Four components

We decompose a watermarking scheme into the four components introduced in Section 3.1, with the additional concept of watermark unit.

1. 1.

   Watermark unit is the component on which the watermark is applied: in prior works, it is usually tokens, but agents also proposed to apply watermarks on equivalence classes of tokens.

2. 2.

   Context seeding hashes the previously generated tokens into a seed that is used, along with the watermark private key, to pseudo-randomly sample the watermarking scores.

3. 3.

   The watermark score is the pseudo-random score that can be combined with true randomness before being used by the logits transformation. At detection, only the pseudo-random score can be recovered and not the true randomness.

4. 4.

   The logits transformation combines the score with the model’s next-token distribution to pick the next token, in a way that is distortion-free (Equation 1).

5. 5.

   The detector recomputes the pseudo-random scores of a candidate text, aggregates them into a statistic, and returns a p-value.

Notation

Let Σ be the vocabulary and ω∈Σ∗ a text, with ωt its tth token and pt∈Δ⁡(Σ) the model’s next-token distribution given ω<t. A grouping 𝒢:Σ→ℕ maps tokens to units. We write u=𝒢⁡(v) for the group of a candidate token v and

for its mass under pt. Context seeding maps the context ct (a function of ω<t) and the private key ξ to a seed at. The seed is then used to pseudo-randomly sample a multivariate uniform random variable Uat,u∼𝒰⁡([0,1]) for every token unit u. Because the watermark score may add true randomness that is not computable at detection, we use the notation U~at,u for the watermark score mixed with true randomness (and thus used by the logits transformation). The logits transformation selects a group u∗ from (U~at,pt) and then samples the emitted token inside that group using the residual probability pt​(v)/Mt​(u∗). At detection, the scheme reconstructs the set D of retained (seed, unit) pairs of ω (e.g., after de-duplication of repeated (at,u) tuples as in Fernandez et al. (2023)), with N:⁣=|D|, aggregates them into a statistic S⁡(ω), and reports a p-value P⁡(ω). We write Q(N,s):⁣=Pr[Gamma(N,1)≥s] for the Gamma upper tail.

Experimental setup

We ablate every component against a simple baseline AAR scheme: identity grouping, a two-token context, no private randomness in the score (U~=U), the Gumbel race, and the scalar Gamma detector of Section 4.2. For each ablation, we swap one component (e.g., the Gumbel race with the ResidualRace) and report the difference in percentage points for our evaluation metrics (Section 4). Quality is one minus the average relative distance to the unwatermarked model in perplexity and both Self-BLEU scores. In the bar plots, positive values indicate an improvement, and values in parentheses give the reference quality or TPR@1%FPR. For ablating some component hyperparameters, we may sometime use a different reference (instead of the simple AAR baseline) and specify in the caption when the reference differs from the baseline.

A.2 Watermark Unit

Prior works (Aaronson, 2023; Kirchenbauer et al., 2023; Dathathri et al., 2024; Sander et al., 2026) used a single token as the watermark: 𝒢 is the identity and Mt=pt.

Lexical group

Agents replace tokens by classes of a public map 𝒢 built from the tokenizer and WordNet 3.0: a token is decoded, stripped and lower-cased, and if what remains is an ASCII word of at least three characters it is mapped to the representative of its first WordNet synset. Hence synonyms share a class, which is what makes it robust to substitution: a replacement inside a class leaves both the seed and the observed "token" unchanged. Figure 8 confirms this intuition: Lexical group indeed significantly improves robustness to substitution, and slightly improves backtranslation and paraphrasing robustness as well. However, it significantly lowers deletion robustness, and slightly lowers quality.

A.3 Context Seeding

For context seeding, unlike some prior works (Kirchenbauer et al., 2023), we assume the hashing function is without collision: the hash of a context is unique. Because the set of possible contexts is finite, this assumption is achievable in practice. We defer the details of the hashing functions, as well as an analysis of the different hashing functions, to App. G. We also use a per-request watermark cache: if a seed at hashed from the context is repeated, the next token is sampled without watermarking. This is required to guarantee that each individual request is distortion-free.

Fixed-length

Prior work hashes the tuple of the k preceding tokens (Kirchenbauer et al., 2023; Sander et al., 2026). In particular, k=0 is the unigram scheme of Zhao et al. (2024b). As we show in Figure 8, the context size k controls a robustness-quality trade-off: shorter contexts survive more edits but repeat more often which leads to correlation across a corpus of generated texts. Furthermore, at low context, i.e., k=1 or k=0, because we need to de-duplicate (seed, token) pairs at detection, the lack of context diversity lowers the detectability and therefore can counteract the gain in robustness.

Unordered

It is the same as Fixed-length context seeding, except that it hashes the set of the k preceding tokens rather than their tuple (the candidate v is kept separate), so that a transposition of preceding tokens does not change the context. For k≤1, it is identical to Fixed-length. As we show in Figure 10, for k=2 and k=3 this change barely affects robustness and slightly improves quality. The agents also proposed an unordered pair variant (Pair in Table 6) that hashes the set {ωt−1,v}, which includes the candidate token itself, rather than the tuple (ωt−1,v). Compared to the core scheme, it significantly increases robustness but significantly lowers quality (Figure 10).

Adaptive and global length

Rather than fixing k, these variants shorten the context for the tokens that are unlikely to repeat anyway, so that only the positions that need a long context pay for one.

1. 1.

   Adaptive length: given a classifier rare:Σ→{0,1} (here that a token frequency in a fixed corpus is below 0.1%), if the previous token is rare, hash only that token (k=1), otherwise hash the full two-token window. In Table 6, we also evaluate it with the Excess detector of Section A.6.

2. 2.

   Global + local: fix a public set E⊂Σ of eligible tokens (the agents used as E the tokens that have a WordNet entry), use unigram (i.e., k=0) if the candidate is in E, and a k-token context otherwise, with k=2 unless stated otherwise.

3. 3.

   Global first use refines it by applying unigram to a candidate in E only the first time it appears in the completion, and then uses a two-token context if it appears again.

We show in Figure 10 that all those approaches significantly increase the robustness compared to the baseline, but both global variants also significantly degrade quality.

Anchored

Let E again be a public token set and let s be the distance from the current token back to the most recent token of E. If s≤Smax=4, use as context the tuple of that anchor token and the offset s. Otherwise, if s>Smax skip the watermark. For the detector, at every token ωt it finds the closest previous anchor and sweeps all permitted distances s∈{0,…,Smax} adjusting the p-values with Bonferroni correction. Yet, as we show in Figure 12, while intuitively the slot search should buy robustness (editing tokens in between anchor tokens does not change the context), the search is too expensive and this scheme is worse than the baseline on every direction.

Suffix cascade and Tally & ladder

These approaches share the same goal: when a context has already been used in the completion, derive a new seed for it instead of skipping the position, so that a short context keeps its robustness without losing detectability. They only differ in how that new seed is built.

1. 1.

   Suffix cascade: try suffix lengths in increasing order and use the shortest one that has not already been used in this completion. If every allowed length has been used, skip the watermark.

2. 2.

   Tally: keep a one-token suffix and hash the previous token, and the number of its earlier occurrences, so that a repeated context yields a fresh seed.

3. 3.

   Ladder generalizes both: count every suffix width 1≤w≤K, select the shortest width whose count is below a threshold, and hash the width, the suffix, and the count. Counts are capped at 8192, beyond which the position falls back to ordinary sampling due to the specifics of the hash function (App. G). Optionally, a floor forces a width of at least 2 when the previous token ID is below 200 (as in ConcordMark, Section H.1), as for most tokenizers low ID tokens tend to be very common (e.g., letters, numers, or punctuations). Unless stated otherwise, we use K=4, a threshold of 3, and the floor, and we report all four combinations of K∈{3,4} with and without the floor in Table 6. Ladder makes almost every position usable at a one- or two-token context, which is why the Ladder is used in ConcordMark (Section 4.3).

As we find in Figure 12, the Suffix cascade indeed keeps most of the robustness of a short context, with a small quality cost, and unlike k=1 it improves on all robustness metrics.

Last important and Interleaved

Those two schemes belong to the skeleton family (Figure 3). The skeleton family splits the vocabulary using a fixed important/filler tokens classifier and gives the two streams separate histories.

1. 1.

   Last important ignores filler tokens when computing the context and uses the last important token as context, so that edits to filler tokens do not move the context of the next content word.

2. 2.

   Interleaved uses when sampling an important token the last important token as context, and when sampling a filler token uses the last two previous filler tokens as context.

Both context seedings must be changed together with their logits transformation which is why we defer the evaluation to Section A.5.

A.4 Watermark Score

Prelude

Prelude is the only component that can be composed with any other. It consists of sampling ordinarily at the start of the completion until the accumulated entropy H2(pt)=−log∑vpt(v)2 of the first generated tokens reaches h nats, or after T tokens, whichever comes first. In practice, agents use h=14 and T=24. For detection, we keep all tokens, including those of the prelude, as the detector cannot recompute the entropy without the model. We compare Prelude to the no-Prelude baseline and find in Figure 13 that it indeed improves quality with a mild robustness cost.

Gaussian copula

Gaussian copula draws a fresh private normal ηt,u∼𝒩⁡(0,1) at every position and sets U~t,u=Φ⁡(ρ​Φ−1​(Uat,u)+1−ρ2​ηt,u). The goal is to inject true randomness into the watermark score to increase quality at the cost of detectability. At ρ=0 the scheme is unwatermarked, at ρ=1 it uses only the watermarked pseudo-random scores. As we show in Section 4.2, it is a more effective way of improving quality via true randomness than prior works.

The Gaussian copula can be further refined with a common pair filter: we replace ρ with a function of the current context, ρt,u=ρ 1{(at,u)∉ℱ}, where ℱ is the set of the 10,000 most frequent (seed, token) pairs of a public corpus (in this case, the first 20,000 texts of the public ELI5 corpus, counting each pair at most once per text). The goal is to further increase the quality with a better detectability trade-off. Yet, as we show in Figure 15, using the simpler ρ∈[0,1] is better in practice.

Tail indicator

For each unit u, set Bu=max⁡(1,⌊1/max⁡(Mt​(u),2−40)⌋) and bu=1−1/Bu, draw a private Vu∼𝒰⁡([0,1]), and let U~t,u=bu+Vu/Bu if Uat,u≥bu and bu​Vu otherwise. The construction keeps U~ uniform, so the scheme stays distortion-free, and the detector still reads the full U. However, as we show in Figure 14, it has a significantly worse quality-robustness trade-off than e.g., the Gaussian copula.

Private sampling

This is the naive idea: with probability r, ignore the watermark at this position and sample from pt and otherwise use the keyed sampler. However, as we show in Figure 14, it has a significantly worse quality-robustness trade-off than e.g., the Gaussian copula.

Channels

Channels maintain L independent keyed fields Uj,a,u and draw a single channel J∼𝒰​{0,…,L−1} per request, using U~t,u=UJ,at,u throughout the completion. The goal is the same as the Gaussian copula, spending private randomness on quality, except that here the randomness is discrete: two requests that share a prompt and a context take different fields. The cost is paid at detection, where the detector runs all L fields and corrects its p-value by a Bonferroni factor L. However, as we show in Figure 14, that factor dominates quickly: quality saturates around L=32 while the robustness to every attack keeps falling, so only small L are useful.

For adjusting the p-values to account for the L tests, we can either use Bonferroni or Šidák Indeed, the L per-channel p-values are independent under the null, so the Šidák correction 1−(1−minj⁡Pj)L is valid and strictly tighter than the Bonferroni factor L, and the submissions use both. Yet, we observe no difference in practice (Table 9).

Channels can be further refined by correlating the L fields, so that the request-level choice costs less detectability.

1. 1.

   Latin channels derive them from a single base uniform by rotation, Uj,a,u=(Ua,u+j/L)mod1.

2. 2.

   Antithetic channels work only for schemes that use normal scores X=Φ−1​(Uj,a,u), and pair each keyed normal X with −X.

3. 3.

   Stratified channels use a keyed permutation σa,u and keyed jitters, Uj,a,u=(σa,u​(j)+υj,a,u)/L.

All three keep the same detector and use only Bonferroni correction (Šidák is no longer valid due to the dependence between tests), so Figure 16 compares each against independent channels at the same L. Yet, none of the three is a reliable improvement: the differences are within a few points everywhere and change sign across attacks. We therefore suggest using the simpler version with independent keys, which also benefits from the more powerful Šidák correction.

Gaussian direction

This is the first of the private-geometry scores (Figure 3), and it generalizes Channels from a discrete choice to a continuous one. We use the context seed and watermark private key to sample a multivariate Gaussian Zat,u∼𝒩⁡(0,Id), and per request we draw a private unit direction A∈Sd−1 once, used for the whole completion:

Hence, two different requests (even with the same prompt) will have different directions, hence different watermark scores and a higher diversity. As we show in Figure 18, increasing the dimension d significantly improves the quality at the cost of detectability, and thus robustness.

Spherical

This is a similar idea to the Gaussian direction, but on compact spaces.

1. 1.

   Signed sphere draws Zat,u∼𝒰⁡(S2) and A∈S2, with U~at,u=(1+A⋅Zat,u)/2.

2. 2.

   Unoriented axis draws Zat,u∼𝒰⁡(S2) and A∈S2, with U~at,u=|A⋅Zat,u|.

3. 3.

   Complex ray draws Zat,u∼𝒰⁡(S3) and A unit vectors of ℂ3 modulo phase with U~at,u=|A∗​Zat,u|2​(2−|A∗​Zat,u|2).

4. 4.

   Rotation draws Zat,u∼𝒰⁡(S3) and A is a unit quaternion with U~at,u=F⁡(|A⋅Zat,u|), where F⁡(r)=2π​[arcsin⁡r+r​1−r2].

Figure 18 evaluates all four (using their corresponding detectors) compared to the simple baseline. We find that they all trade off quality for detectability and that Rotation has the most favorable trade-offs. Because the Gaussian direction and the spherical geometries only change the score U~, they can be combined with any logits transformation: in Table 7, we evaluate each of them both with the Gumbel race and with the ResidualRace (Section A.5).

Wheel

Draw a keyed offset Uat and a per-request random rotation φ∼𝒰⁡([0,1]), and feed U~at,u=(Uat,u+φ)mod1 to the QuantileRace transformation of Section A.5. It is the continuous limit of the Latin channels of Figure 16: the rotation φ is drawn from the continuous circle rather than from L evenly spaced angles. It requires the rotation-invariant detector of Section A.6.

A.5 Logits Transformation

Gumbel race

This is the transformation of Aaronson (2023), which we rewrite here with the group notation for clarity:

after which the token is drawn inside u∗ with the residual probability pt​(v)/Mt​(u∗). By the Gumbel-max trick, u∗ has distribution Mt in expectation over the key, so the scheme is distortion-free. Equivalently, it is an exponential race: u∗ minimizes −logU~t,u/Mt(u).

BandRace

Partition the groups into bands whose masses differ by less than a factor of four,

pick a band randomly with probability equal to its total mass, and run the Gumbel race inside it. This increases quality as the choice of the band is truly random, but lowers the detectability (Figure 19).

PoolRace

It is the black-box equivalent of the Gumbel race (i.e., it is an approximation of the Gumbel race that does not require access to the logits), similar to what has been proposed in Bahri & Wieting (2026): draw K=16 tokens Vi independently from pt, let Nu count the proposals in group u, and pick

As we see in Figure 19, this variant slightly improves quality because the candidates are drawn with true randomness, but hurts detectability.

MomentBridge

The closest equivalent to this scheme in the literature is SynthID (Dathathri et al., 2024), where the watermark distribution is refined iteratively using the watermark keyed scores. Let K∈ℕ be a scheme parameter. Start with the weights wu(0)=Mt​(u) and the fixed costs cu=−log⁡Mt​(u). Let Xt,u=Φ−1​(U~t,u) be the normal watermark score. For i=1,…,K, sample random Gaussians ηi,u and compute:

Let

we update the weights iteratively with

where Ai=maxu:wu(i−1)>0|ri,u|,

The next group is sampled randomly according to the w(K) distribution. Because of the true randomness in ηi,u, this logits transformation trades off quality for detectability. However, as we show in Figure 19, it achieves a relatively poor trade-off.

QuantileRace

It replaces the Gumbel race with inverse-CDF selection: order each group (or token) in the vocabulary according to a keyed pseudo-random permutation Rat,u with an interval of length Mt​(u), and emit the group whose interval contains the watermark score U~at. See Figure 19.

TransportRace and QuotaTransport

This is the second black-box family, and the only one that couples the transformation to the Channels. Draw L tokens from the model, compute the keyed score Ej,at,𝒢⁡(Vi) of every (channel, proposal) pair, and choose the permutation σ∗ maximizing ∑jEj,at,𝒢⁡(Vσ⁡(j)). QuotaTransport stratifies the proposal pool before the same assignment, drawing one proposal from each of the L mass intervals (i+W)/L for a fresh W∼𝒰⁡([0,1]), so that the pool covers the distribution rather than concentrating on its mode. Both were evaluated at L∈{32,128} against the Gamma detector with Bonferroni correction. As we see in Figure 19, all four configurations substantially improve quality, but they are less robust than the Gumbel race.

ResidualRace

This is the extension of the Gumbel race described in Section 4.2. We keep one residual clock Rat,u per (context, group) pair, initialized at −log⁡U~at,u on first use. At each step, we sample the next token according to the exponential race:

and replace the winning token’s score with a fresh exponential random variable: Rat,u∗∼Exp⁡(1). Because of the memorylessness property of the exponential, all losing residual exponentials are independent given the past, so the next winning probability is still Mt​(u) even when a context recurs and its masses have changed. This is what makes it sound to re-use previous clocks in the exponential race when a context re-occurs (Proposition 5). As we show in Figure 19, it increases detectability at the cost of quality. Yet, as we have shown in Figure 5, adding Gaussian copula noise to the initial clocks recovers the quality cost while keeping the detectability gains.

Content gate and Interleaved gate

This is the skeleton family logits transformation, which relies on either the Last important or Interleaved context seeding. Let 𝒞 be the important groups, and rt=∑u∈𝒞Mt​(u). At each step, draw a fresh Bt∼Bernoulli⁡(rt).

1. 1.

   The Content gate uses the Gumbel race within the important groups when Bt=1 and samples conditionally from the fillers otherwise.

2. 2.

   The Interleaved gate uses a Gumbel race inside whichever group the gate selects (e.g., either the important or filler group), seeding the Gumbel race with the corresponding group context.

As we see in Figure 19, the Interleaved gate strictly outperforms the Content gate, though both show limited robustness.

A.6 Detection

Scalar Gamma and Gaussian

The Gamma detector sums exponential scores,

The Gaussian detector sums the normal scores instead,

Both detectors weight the evidence differently: unlike the Gaussian detector, the Gamma detector gives more weight to the tail of the distribution, i.e., to the tokens with very high watermarking scores. As we see in Figure 20, the global Gamma detector significantly outperforms the Gaussian detector without changing the generation algorithm.

Gamma Clustering and Excess threshold

Both detectors are refinements over the Gamma detector that group the evidence by context before aggregating. Let ℋ⁡(ω) be the set of unique contexts in ω and Da​(ω) the distinct successors of context a, with ma=|Da​(ω)|.

1. 1.

   Clustering calibrates each context separately, and then aggregates the calibrated values,

   	Sa​(ω)	=∑v∈Da​(ω)−log(1−Ua,v),
   	S⁡(ω)	=∑a∈ℋ⁡(ω)−logQ(ma,Sa(ω))∼Gamma(|ℋ(ω)|,1),

   so that a context with many successors cannot dominate the statistic on its own. Indeed, during generation, all successors of a given context take part in the same race, which means that two observations of the same race (i.e., context) carry less information than two observations of distinct races. Hence, even though the detection is sound as long as no (context, candidate) pair is repeated, it should account for this loss of information whenever a context is repeated.

2. 2.

   Excess detection keeps only the part of each −log⁡Q⁡(ma,Sa​(ω)) above the Exp⁡(1) median,

   S⁡(ω)=∑amax⁡(−log⁡Q⁡(ma,Sa​(ω))−log⁡2,0),

   which is zero with probability one half under the null and Exp⁡(1) otherwise, giving the exact tail RH​(s)=∑b=1H(Hb)​2−H​Q​(b,s) for H=|ℋ|. This pushes the idea of the Gamma detector even further: only the tail of the distribution carries reliable watermarking signal.

As we see in Figure 20, both approaches indeed slightly increase detection power without changing the generation algorithm.

Weighted Gamma and Position evidence

Those two detectors are additions on top of the Gamma detector that require a model forward pass at detection, and hence are considerably more costly. Let p^t be the estimated next-token distribution given ω<t.

1. 1.

   Weighted Gamma weights each exponential score by the observed token’s surprisal quantized to quarter-unit,

   ηt=1+14​round⁡(4​min⁡(max⁡(−log⁡p^t​(ωt),0),2.5))

   and uses as a detection statistic

   S(ω)=∑t∈D−ηtlog(1−Uat,ωt).

   The intuition is that a high-entropy position is where the race had a real choice, so its score carries more information than a position where the model was nearly deterministic.

2. 2.

   Position evidence goes further and reconstructs the race itself from the observed token’s log-probability and those of up to eight competitors. At position t, let Ct contain the retained competitors other than ωt. It then reconstructs the Gumbel race outcome:

   αt,v=min(104,p^t​(v)p^t​(ωt)),λt=∑v∈Ctαt,v,Wt={Uat,v<Uat,ωtαt,v for all v∈Ct},

   where Wt indicates whether the observed token wins the reconstructed race. Its exact race-win tail is

   Rt={1−Uat,ωt1+λt1+λt,Wt=1,1−Uat,ωt+Uat,ωt1+λt1+λt,Wt=0.

   Under the null, Pr⁡(Wt=1∣Uat,ωt=u)=uλt and Rt∼Uniform⁡(0,1) (Proposition 7). When no competitor survives, λt=0 and Wt=1, giving Rt=1−Uat,ωt. Keeping only the first retained candidate for each context (i.e., deduplicating occurrences of the context), denoted by Dpos, avoids reusing the same race and gives the statistic

   S(ω)=∑t∈Dpos−logRt∼Gamma(|Dpos|,1).

   It can also be combined with the weighted variant.

As we have shown in Section 4.2, and see in Figure 20, the weighted Position evidence detector is significantly more powerful than the versions that do not require a forward pass.

Quantile detector

This is the detector matched to QuantileRace. Let Rat,u be the keyed ordering score and U~at the keyed offset, it computes S⁡(ω)=∑(a,u)∈Dcos⁡(2​π​(Rat,u−Uat)) and calibrates it by Monte Carlo, since the statistic has no closed-form null.

Quantile + wheel

This is the rotation-invariant detector required by the Wheel. Form the angles Aat,u=2​π​((Rat,u−Uat)mod1) and their resultant S=∥(∑Dcos⁡Aat,u,∑Dsin⁡Aat,u)∥, then report an e-value rather than a p-value. With κ=2​log⁡(2000)/N, the quantity ℰ=I0​(κ​S)/I0​(κ)N has unit expectation under the null, so P=min⁡(1,1/ℰ) is valid. As we see in Figure 21, the Wheel slightly improves quality but almost entirely removes the robustness of QuantileRace.

Predictive Gamma

This variant of the Gamma detector applies to all the private-geometry scores.

Enumerate the retained pairs in D as (ai,ui)i=1N and write Zi=Zai,ui. Let A0,…,AK−1 be a fixed public dictionary of candidate geometries (K=128 for the Signed sphere and Unoriented axis, and K=512 for Complex ray and Rotation), with weights wi,j measuring how well Aj explains the scores observed before step i. Let Bi denote the estimate of A formed from the weights wi,j before observing Zi. Write U~​(B,Z) for the corresponding geometric score from Section A.4, evaluated with geometry B in place of A, and define d​(B,Z)=1−U~​(B,Z). At step i, compute di=d⁡(Bi,Zi).

After scoring Zi, update and normalize the weights as wi+1,j∝wi,jd(Aj,Zi)−1/2, then compute

where eigmax returns a unit eigenvector associated with the largest eigenvalue and ∗ denotes the conjugate transpose. Under the null, each fresh score Zi is independent of the earlier scores, and the matching geometric transform makes di uniform on [0,1] conditional on them. Thus,

We prove this in Proposition 6.

Resultant length, Spherical caps, and Anchor distance

These detectors test whether the Signed sphere scores concentrate around a common direction, without knowing the private direction A. As above, write Zi=Zai,ui for the scores associated with the N retained pairs in D.

1. 1.

   Resultant length measures the alignment of the scores through R=∥∑i=1NZi∥. Aligned scores produce a larger resultant, whereas scores pointing in different directions tend to cancel. The detector compares R with its null distribution for N independent uniform points on S2, making it the spherical analogue of the Gaussian norm detector below.

2. 2.

   Spherical caps measures local concentration around m=min⁡(N,16) anchor scores. For each anchor Zj and area fraction q∈{1/16,1/8,1/4}, it counts Cj,q=∑i≠j𝟙{Zj⋅Zi≥1−2q}. Under the null, conditional on Zj, this count follows Bin⁡(N−1,q). The detector takes the smallest upper-tail p-value and applies a Bonferroni correction for the 3​m anchor–cap combinations.

3. 3.

   Anchor distance replaces cap membership with a continuous measure of alignment. For each anchor Zj, it computes Sj=∑i≠j−log((1−Zj⋅Zi)/2), so that scores closer to the anchor contribute more evidence. Under the null, conditional on Zj, the transformed inner products are independent uniforms, giving Sj∼Gamma⁡(N−1,1). Searching over all N anchors gives the corrected p-value P⁡(ω)=min⁡{1,N​minj⁡Q⁡(N−1,Sj)}.

Figure 23 compares these detectors to the Predictive Gamma detector. We find that the Predictive Gamma detector outperforms each of those geometry-specific detectors.

Gaussian norm, Circle search, and Residual energy

These detectors apply to the Gaussian direction scores. We write Zi=Zai,ui for the scores associated with the N retained pairs in D.

1. 1.

   Gaussian norm: rather than searching for the private direction A, it integrates it out through the norm of the summed scores,

   S⁡(ω)=1N​‖∑i∈DZi‖2∼χd2.

   (13)

   Circle search and Residual energy extend this detector.

2. 2.

   Circle search searches explicitly for the private direction A in dimension d=2. For each candidate direction B⁡(θ)=(cos⁡θ,sin⁡θ), it computes the Gamma statistic

   	S⁡(θ)	=∑i=1N−logΦ(−B(θ)⋅Zi),
   	T	=maxθ∈[0,2​π)⁡S⁡(θ).

   Under the null, S⁡(θ)∼Gamma⁡(N,1) for each fixed θ, but maximizing over directions requires a correction. The detector uses the continuous-search bound of Davies (1987) (Section I.5)

   P0=min⁡{1,Q⁡(N,T)+2​π​T​fN​(T)},

   where fN is the density of Gamma⁡(N,1).

3. 3.

   Residual energy measures the dispersion of the scores around their mean, in addition to their alignment. Let Z¯=N−1​∑i=1NZi. The two statistics are

   	Smean	=N​∥Z¯∥2,
   	Sres	=∑i=1N∥Zi−Z¯∥2.

   Under the null, they are independent, with Smean∼χd2 and Sres∼χd⁡(N−1)2. For N>1, let pmean and pres be their respective upper-tail p-values. The detector combines them through S⁡(ω)=−log⁡pmean−log⁡pres∼Gamma⁡(2,1) under the null, giving P0=Q⁡(2,S⁡(ω)). For N=1, it uses only the Gaussian norm statistic.

As we see in Figure 23, both extensions improve over Gaussian norm, with Circle search giving the largest gains at d=2, but the gains of Residual energy shrink as d grows and vanish at d=8.

Appendix B Examples of Invalid Autoresearch Schemes

In this section, we present 4 schemes that research agents designed using earlier iterations of our harness, whose verification was weaker than that described in Section 3.3. All four schemes appear to outperform prior work, yet they either exploit gaps in the evaluation process or have an abnormally high false positive rate. We re-evaluate them with our harness in Table 2: they show impressive robustness, but none passes our verification. Nevertheless, we believe that some of their ideas are interesting and could be explored in future work.

Prior harnesses

ConcordMark-v0, CathedralMark, OverlayMark, and MeridianMark come from a harness with neither private verification nor private evaluation (i.e., the agent could see the whole evaluation dataset). Furhtermore, the verifier only checked distortion-freeness and the false positive rate on 5,000 English C4 (Raffel et al., 2020) human texts, using a single fixed watermark key. In particular, it had no check for soundness over the key. Schemes that passed were then evaluated once using the watermark key set in their configuration, i.e., chosen by the agent.

ConcordMark-v0

Despite its name, ConcordMark-v0 is unrelated to ConcordMark (Section H.1); we simply kept the independently submitted names for both. Its generation is a Gumbel race with a one-token context (i.e., Fixed-length context seeding with k=1). Its detector, however, combines two p-values: Pw, the rank of the statistic under the private key among 1023 decoy keys (i.e., a Monte Carlo estimate of the p-value), and Pc, a p-value derived from a zero-shot LLM-text classifier, specifically an n-gram classifier. The agent trained this classifier to separate its own watermarked texts (clean and attacked) from 2,500 human texts and calibrated it on 2,500 other human texts. A text is flagged as watermarked if Pw≤0.002 or Pc≤0.008, so that, by the union bound, the false positive rate on human texts is at most 1%. This significantly improved the scheme’s robustness (Table 2).

Yet, the classifier learned to recognize LLM-generated text in general rather than watermarked LLM-generated text, e.g., on unwatermarked replies from WildChat, it wrongly flags 31% of the texts as watermarked at a supposedly 1% FPR. This shows the importance of soundness over the key. Indeed, Pc does not depend on the key, so the scheme violates soundness over the key (Equation 2) (a text flagged by the classifier is flagged under every key).

CathedralMark

CathedralMark is a later scheme from the same run that keeps the detector of ConcordMark-v0 but also steers the generation toward its classifier. Depending on a keyed latent state, it either tilts the next-token distribution toward the tokens that increase the classifier score or applies a compensating tilt, so that, in expectation over the key, the next-token distribution remains pt. Hence, the scheme passes the distortion-freeness test but, like ConcordMark-v0, fails the soundness over the key check. Nonetheless, the idea of steering watermark generation toward easily detectable LLM patterns is particularly robust: it achieves more than 90% robustness to paraphrasing.

OverlayMark

For OverlayMark, a scheme similar to CathedralMark, the agent carefully selected a watermark key that maximized TPR@1%FPR on the evaluation dataset because the evaluation was public. Hence, as shown in Table 2, it achieves 100% paraphrasing robustness in the original harness’s evaluation, but this falls to 92% in our updated harness, which varies the key (we take the worst TPR@1%FPR out of 5 random keys).

MeridianMark

MeridianMark uses a Gumbel race seeded by the last 4 tokens mapped to 32 semantic clusters. Its detector combines an entropy-weighted Gamma statistic and a tail statistic, and computes a p-value by fitting a Cauchy distribution to the statistics of 1024 decoy keys. It then remaps the p-values: every p-value below 0.0066 is linearly mapped to [0,0.001]. The agent chose this threshold just below the fifth-smallest p-value of the public human texts (0.0069), and documented it explicitly in its submission: “evaluate never measures human FPR […] exactly 4 humans fall under it (one-text margin)”. This shows that, if verification is public, the agent can exploit the metric to pass verification incorrectly. This is why our final harness uses private verification.

Takeaways

Each of the above schemes exploits a gap in the harness: a human null corpus that does not cover LLM-generated text (ConcordMark-v0 and CathedralMark), a watermark key chosen by the agent (OverlayMark), or the ability to exploit public verification (MeridianMark). Our harness closes these gaps: our private verification corpus includes LLM-generated replies (Section D.1), we evaluate each scheme with 5 private keys and report the worst result (Section D.2), and our soundness over the key test only requires a single text on which the detector is miscalibrated over the key to reject a scheme, independently of how well the detector is calibrated on any specific human corpus.

Appendix C Studying the Soundness Given a Key

In this section, we investigate the effect of enforcing soundness given a key (Section 3.2). In Figure 2, we plotted the autoresearch trajectories of each agent independently, and as we explain they tackled soundness given a key differently which makes the comparison uneven. Namely, GPT-6 Astra multiplies its p-values by 1/β=20 to be provably sound given a key, Opus 5 applies smaller safety margins that it calibrated locally, and Gemini-3.8 Flash and the baselines apply none. For the baselines in particular, in the settings of Figure 2, neither AAR nor SynthID passes our verification of being sound given a key.

Experimental setup

We rescore the submissions of the four agents from Figure 2 and the three baselines. For each scheme, we remove the explicit outer p-value multiplier chosen by the agent (if any), and keep the rest of the detector unchanged (e.g., the Bonferroni correction over Channels) so it remains sound over the keys. We then evaluate three settings:

• •

  Guaranteed: we multiply the p-value by 1/β=20 as in Equation 14. By Proposition 1, any scheme that is sound over the key becomes sound given a key.

• •

  Empirical: we multiply the p-value by the smallest multiplier κ≥1 such that the scheme passes both soundness tests (Algorithms 2 and 3), with the same thresholds, sample sizes and significance levels as in Section 3.3.

• •

  No correction: we use the raw p-values (κ=1) and do not enforce the soundness given a key test.

In all settings, the scheme must still be distortion-free and sound over the key. We then measure the paraphrasing TPR@1%FPR with the protocol of Section 4.

Baselines fail without correction

In Figure 24, we show the smallest multiplier each scheme requires to pass our soundness tests. Among the baselines, only TextSeal passes without correction. With raw p-values, both AAR and SynthID fail the soundness given a key test. Similarly, 23 of the 52 agent submissions would fail without their outer multiplier. In fact, both GPT-6 Astra (low) and Opus 5 only started adding such multipliers after their first submissions were rejected by our soundness given a key test.

The guaranteed correction is overly conservative

Yet, the multiplier required in practice is much smaller than 1/β: at most 1.08 for all but one submission. This is expected: if the watermark pseudo-random scores are diverse enough over a corpus given a fixed key, by soundness over the key the expected false-positive rate will converge to be below α which satisfies soundness given a key. This means that correcting with 1/β is overly conservative in many cases, costing significant power. This is why we opted to not correct the schemes in our main evaluation in Section 4.1 unless the agents did so in their submissions.

Impact on robustness

We show in Figure 25 the paraphrasing robustness trajectories in each setting. The guaranteed correction costs on average 16.3 percentage points of paraphrasing TPR compared to the empirical correction, whereas the empirical correction costs at most 2.5 percentage points compared to no correction. Importantly, the conclusions of Section 4.1 hold in all three settings: all agents improve over time and outperform all baselines, with Opus 5 and GPT-6 Astra (xhigh) reaching the highest robustness. In particular, in the empirical setting, the best GPT-6 Astra (xhigh) submission reaches 83% paraphrasing TPR, compared to 70% in the guaranteed setting, closing most of the gap with Opus 5 (86%).

Appendix D Experimental Setup

In this section, we detail the experimental setup used in our evaluation (Section 4). Our harness distinguishes two configurations: the public configuration, which is given to the research agent and which it can run locally, and the private configuration, which is only used by the grader when the agent submits a scheme (Section 3.4).

D.1 Watermark Verification

To verify each watermarking scheme, we run the three statistical tests from Section 3.3 on a corpus of non-watermarked texts. A scheme is valid only if it passes all three tests.

Corpora

The public corpus consists of the answers of the ELI5 (Fan et al., 2019) dataset. The private corpus is a mixture where half of the texts come from WildChat (Zhao et al., 2024a) and the other half from Wikipedia articles, evenly split across 10 languages (English, German, French, Spanish, Chinese, Russian, Arabic, Hindi, Japanese, and Swahili). For WildChat, we take in each conversation the first assistant reply that has at least 512 tokens (i.e., non-watermarked LLM-generated text), across all 86 shards of WildChat-4.8M. For Wikipedia, we use the first shard of the 2023-11-01 dump of each language. In both corpora, we tokenize the texts with the Llama-3.1-8B-Instruct tokenizer, discard texts shorter than 512 tokens, and truncate the others to their first 512 tokens. Whenever a test samples texts from the corpus, it samples them from this mixture, according to the weights of each component.

Soundness tests

For soundness over the key (Algorithm 2), we sample 200 fixed reference texts from the corpus and draw m keys per text, with m=10,000 in the public configuration and m=20,000 in the private one. For soundness given a key (Algorithm 3), we draw 1000 keys and, for each key, sample 20,000 fresh texts from the corpus (with replacement). We set the allowed bad-key fraction to β=5% with a single screening level of 1%. In the private configuration, both tests check the thresholds α∈{0.001,0.01,0.05}, whereas the public configuration only checks α∈{0.01,0.05}. The two soundness tests share a significance level of 0.001, split equally between them (i.e., δ=0.0005 in Algorithms 2 and 3).

Distortion-freeness test

For distortion-freeness (Algorithm 1), we sample 4 texts from the corpus and use their first 128 tokens as contexts. We compute the next-token distributions of Llama-3.2-1B-Instruct (temperature 1, top-k 50) on each context, and average the watermarked distributions over 2048 keys, testing the 16 most likely tokens (and the remaining mass). We set the significance level to δ=0.001. As in Algorithm 1, if the scheme leaves all 4 distributions unchanged, the test is inconclusive and the scheme is rejected.

D.2 Watermark Evaluation

The public and private evaluations share the same protocol, which we describe first, and differ in the watermark keys, prompts, and sampling seeds.

Generation

We sample 1000 prompts from the questions of the train split of ELI5 (Fan et al., 2019), keeping only questions of more than 100 characters and removing duplicates. We generate the replies with Llama-3.1-8B-Instruct, prompting it with the raw question. We sample with temperature 1, top-k 50, and top-p 1, and force replies of 200 to 300 tokens.

Detection and robustness

To measure watermark strength (either before or after the attacks), we report the TPR@1%FPR, i.e., the fraction of the 1000 replies for which the p-value returned by the detector is below 0.01. Because the verification (Section D.1) screens the p-values for soundness, we do not calibrate the threshold empirically. For robustness, we use four attacks. For 50% word deletion, each whitespace-separated word is deleted independently with probability 0.5. For 50% synonym substitution, we randomly replace half of the words (or all replaceable words if there are fewer) with a random WordNet synonym. For back-translation through French and paraphrasing, we use GPT-5.4-mini (gpt-5.4-mini-2026-03-17) with the default sampling parameters of the OpenAI API and the prompts from Section J.2.

Quality

To measure the impact on quality, we use three different metrics. We compute the perplexity of each of the 1000 replies with Llama-3.1-8B-Instruct, on the reply alone (i.e., not conditioned on the prompt), and average the perplexities over the replies. We also measure the diversity of replies to a single prompt: we sample 10 additional prompts from ELI5 and generate 100 replies for each prompt, following the protocol established in Gloaguen et al. (2026). For each reply, we compute the Self-BLEU (bigram and trigram) score on token IDs, using the 99 other replies to the same prompt as references, and average the scores over the 1000 replies. We then compare each quality metric to that of the unwatermarked model, evaluated with the same protocol. Then, we report the relative distance to the unwatermarked model (in percent), and, when we report an aggregated quality metric, we average the relative distances of the three metrics.

Public and private evaluation

In the public configuration, the agent evaluates its scheme locally with its own watermark key, and the prompts are always the first 1000 (and 10 for diversity) ELI5 questions after shuffling the dataset with seed 0. In the private configuration, the grader evaluates each submission with 5 private watermark keys. For each key, the grader derives from the key the seeds used to shuffle ELI5 (separately for the detection and diversity prompts) and the generation seed, so that the private prompts differ from the public ones and across keys. The grader then reports, for each metric, the worst result over the 5 keys: the minimum TPR and the maximum distance to the unwatermarked model. Only the aggregated metrics and the verification verdicts are returned to the agent. The private evaluation runs in an isolated container.

D.3 Research Agents

We summarize the research agents in Table 3. We run all agents with the prompt of Section J.1, in their own command-line interface and with subagents enabled. When an agent exits, the harness starts a new agent in the same container and Git checkout, and the loop stops once the submission budget is exhausted and all submissions are graded.

Environment

Each agent runs in a persistent Docker container with one NVIDIA RTX PRO 6000 Blackwell GPU (96 GB), 48 CPU cores, and full Internet access. The container holds the watermarking library (with the baselines of Section 3.4), the public configurations, and API keys for third-party AI providers (intended to be used for e.g., paraphrasing attacks). The grader runs in a separate container, which does not share any storage with the agent container, and on a different GPU.

D.4 Baselines

Here, we first detail how the three baselines (AAR, SynthID, and TextSeal) work using the notation from Section 3.1 and then go through their hyperparameters. Recall that ω∈Σ∗ is a sequence of tokens, pt∈Δ⁡(Σ) the conditional probability distribution of the next token given ω<t. We assume there exists a sequence (Uat) of pseudo-random vectors in [0,1]|Σ| seeded by the context hash at whose entries are i.i.d. uniform variables. The watermark is defined by f:[0,1]|Σ|×Δ⁡(Σ)→Δ⁡(Σ), and S⁡(ω) is the detector statistic.

AAR

With AAR, the logits transformation is simply a Gumbel race, and the next token is sampled according to

Because −log⁡(−log⁡(Uat,v)) are i.i.d. Gumbel random variables, using the Gumbel-max trick we can show that in expectation over Uat the next token ωt is sampled according to pt. For detection, we use the one-sided Kolmogorov-Smirnov test against the uniform null, with the statistic

where U(i) are the ordered scores (Uat,ωt).

SynthID

With SynthID, the logits transformation corresponds to a repeated tournament sampling with L layers and C leaves. Instead of a single watermark score, SynthID uses L independent watermark scores Uat(1),…,Uat(L), one per layer, which it binarizes into gat,v(l):⁣=𝟙{Uat,v(l)>1/2}. Let pt(l) be the probability distribution at layer l with pt(0):⁣=pt. Then, at each layer we apply

and the logits transformation is f(Uat,pt):⁣=pt(L). For detection, we use the weighted mean detector over the retained positions t∈D, with statistic

where w1,…,wL≥0 are layer weights summing to 1. Under the null hypothesis, the per-token scores ∑lwl​gat,ωt(l) are i.i.d. with mean 1/2 and variance σw2:⁣=14∑lwl2. We thus use a normal approximation and return the p-value 1−Φ⁡(N​(S⁡(ω)−1/2)/σw), where Φ is the standard normal CDF.

TextSeal

With TextSeal, the logits transformation is the same Gumbel race as AAR, but with two keys: a primary and a secondary key, yielding two watermark scores Uat and Uat′. At each step, the watermark score used by the logits transformation is U~at=Uat′ with probability λ and U~at=Uat otherwise, where the choice is drawn with true randomness. For detection, we fuse the two scores of each retained position and use the weighted statistic

where the weights wt∈[wmin,wmax] increase linearly with the entropy Ht of the next-token distribution of an auxiliary LLM at position t, i.e., wt=wmin+(wmax−wmin)​(Ht−mint′⁡Ht′)/(maxt′⁡Ht′−mint′⁡Ht′).

Watermark hyperparameters

All three baselines seed their watermark with a hash of the 3 preceding tokens and the private key, and do not apply the watermark when the context was already seen in the same request. All sample from the top-50 tokens. For AAR (Aaronson, 2023), we use the Gumbel-max sampling, and its detector, a one-sided Kolmogorov–Smirnov test on the de-duplicated Gumbel scores. For SynthID (Dathathri et al., 2024), we use the tournament sampling with 30 layers and 2 leaves, restricted to the top-50 tokens of the unwatermarked distribution, and the weighted mean detector (with weights decreasing linearly from 10 to 1 across layers) with a normal approximation. For TextSeal (Sander et al., 2026), we use a secondary key with probability λ=0.1, and weights between wmin=0.1 and wmax=1 derived from the entropy of Gemma-3-270M-it.

D.5 Held-Out Evaluation

For the held-out evaluation of Section 4.3, we use a different protocol, which neither the agents nor our harness used.

Prompts

We sample prompts from WildChat-4.8M (Zhao et al., 2024a) using its language labels, English and Chinese. We use the first user turn of conversations whose second turn is an assistant reply, remove duplicate prompts, and discard prompts longer than 3500 tokens. We keep a prompt only if its original WildChat reply has at least 200 tokens with both the Llama-3.1-8B-Instruct and the Qwen3.8-27B tokenizers. We then take the first 1000 qualifying prompts of each language after shuffling the dataset shards with a fixed seed. The prompts are the same for all schemes, models, and keys.

Generation

We generate the replies with Llama-3.1-8B-Instruct and Qwen3.8-27B (with thinking disabled), using their chat template with a single user message and no system prompt. We sample with temperature 1, top-k 50, and top-p 1, and force replies of 200 to 300 tokens.

Metrics

We report the TPR@1%FPR as in Section D.2. For word deletion and synonym substitution, we segment the English texts with a regular expression and the Chinese texts with Jieba. Deletion removes exactly half of the words, chosen at random, and substitution replaces half of the words with a random synonym from WordNet (English) or the Open Multilingual WordNet (Chinese); if fewer words are replaceable, we replace all of them. For back-translation and paraphrasing, we use DeepSeek-v4 Flash with temperature 1, reasoning disabled, and the prompts of Section J.2.

We compute the perplexity of the replies with the model that generated them, on the reply alone. For diversity, we use the first 10 prompts of each language and generate 100 replies per prompt, and compute the Self-BLEU scores as in Section D.2. We report the relative distance (in percent) to the unwatermarked model with the same model and language. We use 5 new watermark keys, different from all keys used in our harness, and report for each metric the worst result over the 5 keys.

Lastly, for RotationFirst, we rebuild the Lexical group map for each tokenizer and extend it to Chinese tokens with the Open Multilingual WordNet.

Appendix E Our Autoresearch Harness

In this section, we detail our watermarking library (Section E.1) and motivate the design of the research environment (Section E.2).

E.1 Our Watermarking Library

Library design

We design our watermarking library around four primary components: the context seeding, the watermark score, the logits transformation, and the detector, following the formalization described in Section 3.1 and the work of Gloaguen et al. (2026). At generation time, the watermarking scheme is defined as a vLLM logit processor. It takes as input the list of tokens already generated and the next-token logits (i.e., the unnormalized next-token probability distribution) and returns the watermarked next-token logits. It is stateful per prompt but stateless across prompts (i.e., each prompt is treated independently). For detection, it takes as input a list of tokens and returns a p-value. This simple design, essentially a class with two methods, ensures that the proposed schemes can be easily evaluated and, following Gloaguen et al. (2026), is expressive enough to represent most prior watermarking schemes. In fact, alongside the library boilerplate, we provide the research agents with implementations of the following watermarks: AAR (Aaronson, 2023), SynthID (Dathathri et al., 2024), DiPMark (Wu et al., 2024), TextSeal (Sander et al., 2026), MCMark (Chen et al., 2025), SWEET (Lee et al., 2024), KGW (Kirchenbauer et al., 2023), unigram (Zhao et al., 2024b), and the schemes from Gloaguen et al. (2026).

Throughput

We compare our vLLM library with the reference implementations of AAR, SynthID, and TextSeal on NVIDIA RTX PRO 6000 Blackwell GPUs. For each scheme, we generate 128 tokens for each of 100 ELI5 prompts with Llama-3.2-1B-Instruct, temperature 0.7, and top-p 1. Our runs and the SynthID reference use top-k 50; the AAR and TextSeal references expose no top-k option and sample from the full vocabulary. We exclude model loading and compute throughput as total generated tokens divided by total generation time. Our unbatched generation is 1.56–1.60× faster than the references; batching eight prompts increases this to 4.12–7.27×.

E.2 Research Environment

Here, we motivate the two-phase submission budget of Section 3.4 with two runs from Table 5. We find that, when several submissions are available, agents mostly spend them on variants of a single scheme, whereas when a single submission is left, they explore distinct ideas locally before committing to one.

GPT-6 Astra (low)

In this run, the first agent was given all 11 submissions at once. It spent 10 of them in about four hours on FreshRace and 8 of its variants (FreshRace-lexical was submitted twice). Then, with a single submission left, the 8 subsequent agents implemented and evaluated locally 15 new schemes before submitting RenewalRace. Moreover, when we retrospectively submit the 15 unsubmitted schemes to our private grader, all of them pass our verification, i.e., they are not merely failed attempts.

Opus 5

This run uses the two-phase budget. In the first phase, the agent spent its 4 submissions on a single scheme, TallyMark. After its first submission failed our soundness given a key test, it changed the context seeding (also rejected), then reverted this change and multiplied the p-values by 4, and finally resubmitted the same code with an updated description. In the second phase, where each agent has a single submission, all 12 submissions are distinct schemes, which significantly improve robustness.

Appendix F Statistical Tests

In this section, we present the algorithmic description of the three statistical tests from Section 3.3. We prove in Section I.3 that each test rejects a valid scheme with probability at most δ. If a scheme is rejected, it means that with confidence 1−δ it violates the property. If it is not rejected, as with every statistical tests, it means we can’t conclude whether the scheme violates the property. In practice, however we find that the tests are powerful e.g., in App. B they rejected all bad schemes and even rejected some of the baselines.

Appendix G Hash Function and PRNG

In this part, we explain the hash functions used by the agents to turn the context into a seed (i.e., an integer), and detail the PRNG algorithm we use in our library.

Hash functions

As stated in Section A.3, we assume that the hashing function is without collision, i.e., two contexts that are different should return a different hash. In particular, prior works used weaker hash functions that had a high probability of collisions. We assume we have a tuple (x1,…,xk)∈ℕk of integers (token IDs or lexical classes 𝒢⁡(ωt)). All arithmetic is on unsigned 64-bit integers (i.e., modulo 264) unless stated otherwise, ⊕ is the bitwise XOR, and ≫ the right shift.

SplitMix64 hash: We chain the SplitMix64 finalizer m⁡(x)=y2⊕(y2≫31), with y1=(x⊕(x≫30))⋅0xbf58476d1ce4e5b9 and y2=(y1⊕(y1≫27))⋅0x94d049bb133111eb. The context hash is

where the offset +j makes the hash order-sensitive. The hash is not injective, but with 64-bit outputs a collision among n contexts has probability at most n2/265.

Rotation first hash: packs the context injectively, so the no-collision assumption holds exactly. However, it is only valid for context sizes up to two. A context of length one is encoded as at=x1+1, and one of length two as at=241+220​x1+x2, which is injective for vocabularies below 220.

Rolling polynomial hash: a rolling polynomial hash with a counter. This is the hash used with Ladder context seeding (Section A.3). Given the lexical classes gt=𝒢⁡(ωt), a context of width w≤4 ending at t is hashed as

where the XOR with a width-dependent constant separates contexts of different lengths. Let n be the number of earlier occurrences of Kw in the text, and w⋆ the width selected by the ladder. The seed is then

where fmix64 is the MurmurHash3 finalizer and 8192 is the maximum tally, so that repeated contexts with different counts get different seeds. Unlike the two previous hashes, the seed has only 32 bits and collides after roughly 216 distinct contexts.

PRNG

Given a seed at, a candidate token v, and the 64-bit private key ξ, the PRNG returns the score Uat,v∈[0,1). We require it to be stateless, i.e., Uat,v depends only on (ξ,at,v), so that the detector recovers exactly the scores used at generation.

Philox: this is the PRNG we implemented as part of our library and used for all baselines. It uses the counter-based Philox4x32-10 block cipher (Salmon et al., 2011), which maps a 128-bit counter and a 64-bit key to 4 pseudorandom 32-bit words (y0,y1,y2,y3). With A a bijective 32-bit avalanche (x↦x⊕(x≫16), multiply by 0x7FEB352D, x⊕(x≫15), multiply by 0x846CA68B, x⊕(x≫16)), the score is

where the avalanche disperses adjacent token IDs before Philox, and 23 bits fill the float32 mantissa. Independent Channels c use the candidate v+218​c, which never collides with a real token ID.

High-precision Philox: similar to Philox, but the 64-bit seed and the token are packed injectively into the counter:

Since the counter is injective in (at,v), distinct (seed, token) pairs always map to distinct Philox counters, hence to pseudorandomly independent scores. When several scores are needed per (seed, token) pair (e.g., the three coordinates for RotationFirst), the seed is replaced by 26​at+d, reserving the low 6 bits for the index d.

SplitMix64 PRF: reuses the SplitMix64 finalizer m from the SplitMix64 hash, with the domain-separation constants c1=0xa4093822299f31d0 and c2=0x082efa98ec4e6c89:

where 52 bits fill the float64 mantissa, and the midpoint +12 keeps Uat,v∈(0,1) so that Φ−1​(Uat,v) is always finite. Independent Channels c use the key ξ⊕(c⋅0x9e3779b97f4a7c15).

Appendix H ConcordMark and RotationFirst

We describe the schemes used in Section 4.3. For low and high variants, the generation is shared and only the detector differs.

H.1 ConcordMark

Key components

We describe precisely the ConcordMark generation algorithm in Algorithm 4, and both detector variants in Algorithms 5 and 6. ConcordMark uses Lexical group only for context seeding. Let 𝒢:Σ→ℕ be the corresponding lexical map. Given a word, its spelling is first normalized and then gets attributed a synonym class of at most eight members (using WordNet). It then uses Ladder context seeding starting at width 1, or width 2 when the last mapped token ID is below 200. For the hash function, it uses the rolling polynomial hash (App. G).

For the watermark score, it uses both Prelude with a cap of 14 nats of entropy or 24 tokens and 6 independent Channels. It uses the Gumbel race as a logits transformation. For detection, the low variant uses the Excess Gamma detection and the high variant the weighted Position evidence.

H.2 RotationFirst

Key components

We describe precisely the RotationFirst generation algorithm and its detector in Algorithms 7 and 8. RotationFirst uses Lexical group throughout the scheme. Let 𝒢:Σ→ℕ be the corresponding lexical map. Each token is decoded, stripped of surrounding whitespace, and lowercased. If the resulting word appears in WordNet, it is assigned to the synonym class given by its first WordNet synset. Let ℰ be the set of the resulting synonym classes. Tokens sharing this synset are grouped together, whereas all other tokens remain in singleton classes. It then uses Global first use context seeding with the rotation first hash (App. G). For a class c, it uses address (0,c) if it has not been used before and c∈ℰ, and otherwise uses (𝒢⁡(ωt−1)+1,c).

For the watermark score, it uses a private geometry approach, namely the Rotation spherical geometry (Spherical). For each request, sample randomly a uniform quaternion A. Then, at each step draw a uniform keyed Zat,u∈S3 and derive the score:

where F⁡(r)=2π​(arcsin⁡r+r​1−r2). Then for detection, it uses the corresponding Predictive Gamma detector.

H.3 Additional Evaluation

Here, we complete the evaluation from Section 4.3 with additional plots (Figure 26). Namely, we plot the TPR as a function of the significance level α, from 10−6 to 10−1, for clean detection and paraphrasing. At each α, the TPR is the fraction of samples with p-value below α. As in the main evaluation, we report the minimum TPR across the five watermarking seeds.

Results

Across models, languages, and significance levels, these curves confirm the conclusions of Section 4.3. ConcordMark (high) provides the strongest detection, and especially after paraphrasing. ConcordMark (low) also retains a strong paraphrasing robustness while being more efficient at detection, as it does not require a forward pass. RotationFirst is less robust than ConcordMark, as expected as it focuses mainly on quality, yet remains comparable to the baselines.

Appendix I Proofs

In this section, we prove the theoretical claims made throughout the paper. In Section I.1, we state our assumptions and two lemmas used in the rest of the section. In Section I.2, we prove that a detector that is sound over the key can be turned into a detector that is sound given a key (Section 3.2). In Section I.3, we prove that the three statistical tests of Section 3.3 are valid. In Section I.4, we prove that ResidualRace is distortion-free (Section A.5). Lastly, in Section I.5, we derive the null distribution of the Predictive Gamma and Position evidence detectors (Section A.6). For all other mathematical results, we consider the proof sufficiently trivial and do not derive them here.

I.1 Preliminaries

Setup

Let Ξ be the key space, and let ξ be drawn from Ξ as in Equation 2. For every text ω∈Σ∗, the detector returns a p-value Pξ​(ω)∈[0,1], and we assume that (ξ,ω)↦Pξ​(ω) is measurable (which holds trivially for a finite key space). Human texts are written without knowledge of the private key: for any human text distribution ℋ0, the text Ω∼ℋ0 is independent of ξ. As in Section 3.1, we model the PRNG as ideal: the seed at is a function of the context only, and, over a uniformly drawn key, the scores of distinct (seed, unit) pairs are independent, with Ua,u∼𝒰⁡([0,1]) (and Za,u distributed according to its geometry for private-geometry scores). We also assume that the hash function has no collision (App. G). Finally, we write T(s,p,c):⁣=PrZ∼Binomial⁡(s,p)[Z≥c] for the binomial upper tail, as in Algorithm 3.

Lemma 1.

Let X be a real random variable and φ:ℝ→[0,∞) a continuous nonincreasing function such that Pr[X≥s]≤φ(s) for all s∈ℝ. Then min⁡{1,φ⁡(X)} is a valid p-value, i.e., Pr[min{1,φ(X)}≤x]≤x for all x∈[0,1].

Proof.

The case x=1 is trivial, and the case x=0 follows from the case x>0, since Pr[min{1,φ(X)}≤0]≤Pr[min{1,φ(X)}≤x] for every x>0. Let x∈(0,1) and I:⁣={s∈ℝ:φ(s)≤x}. If I=∅, the event is empty. Otherwise, I is an interval unbounded above, and it is bounded below, since otherwise Pr[X≥s]≤x<1 for all s, which contradicts Pr[X≥s]→1 as s→−∞. Let s∗:⁣=infI. By continuity, φ⁡(s∗)≤x, and since {φ(X)≤x}⊆{X≥s∗},

Lemma 2.

Let X be a sum of s independent Bernoulli random variables with parameters at most p∈[0,1]. Then Pr[X≥c]≤T(s,p,c) for all c∈ℕ, and Pr[T(s,p,X)≤η]≤η for all η∈[0,1].

Proof.

Let p1,…,ps≤p be the Bernoulli parameters and V1,…,Vs i.i.d. uniform random variables on [0,1]. Then X has the same distribution as ∑i𝟙{Vi≤pi}≤∑i𝟙{Vi≤p}∼Binomial(s,p), which gives the first claim. For the second claim, since c↦T⁡(s,p,c) is nonincreasing and T⁡(s,p,s+1)=0, let c∗:⁣=min{c∈{0,…,s+1}:T(s,p,c)≤η}. Then {T(s,p,X)≤η}={X≥c∗}, and by the first claim, Pr[X≥c∗]≤T(s,p,c∗)≤η. ∎

I.2 Soundness Given a Key

We prove the claim of Section 3.2: multiplying by 1/β the p-value of a detector that is sound over the key (Equation 2) gives a detector that is sound given a key (Equation 3).

Proposition 1.

Let β∈(0,1] and let Pξ be a detector that satisfies Equation 2. Define the calibrated detector

Then Pξβ satisfies Equation 2, and for every human text distribution ℋ0 and every α∈[0,1],

i.e., Pξβ also satisfies Equation 3.

Proof.

We first relate the rejection regions of both detectors. For α∈[0,1), since min⁡{1,x}≤α<1 holds if and only if x≤α, we have for every key ξ and text ω

For α=1, both properties hold trivially, as all probabilities are bounded by 1. We therefore assume α∈[0,1) in the rest of the proof.

Soundness over the key. Let ω∈Σ∗. By Equation 15 and Equation 2 applied at the threshold α​β∈[0,1],

Soundness given a key. Let ℋ0 be a human text distribution, and define for each key the false positive rate

where the equality follows from Equation 15. Because Ω is independent of ξ, Fubini’s theorem allows us to swap the two expectations, and applying Equation 2 to each fixed text gives

If α=0, then g≥0 and 𝔼ξ​[g​(ξ)]≤0, so g⁡(ξ)=0 almost surely and Prξ[g(ξ)>0]=0≤β. Otherwise, α>0 and Markov’s inequality gives

which concludes the proof. ∎

I.3 Validity of the Verification Tests

We prove that each of the three tests of Section 3.3 wrongly rejects a valid scheme with probability at most its significance level δ.

Proposition 2.

Let f be a distortion-free logits transformation (Equation 1). For each context t, let v1,…,vk be the k most likely tokens under pt and let

map a distribution q∈Δ⁡(Σ) to a probability vector over the head tokens and the pooled remaining tokens. For m independent keys, let dj:⁣=Gt(f(ζt(j),pt))−Gt(pt), d¯ their mean, h:⁣=⌊m/2⌋, u:⁣=sign(h−1∑j=1hdj) and b:⁣=max{0,(m−h)−1∑j=h+1mu⊤dj}. Algorithm 1 combines the p-values

into qt:⁣=min{1,2min(pcoord,psplit)}, and rejects f if Q:⁣=min{1,Nmintqt}≤δ. Then Algorithm 1 rejects f with probability at most δ.

Proof.

Fix a context t. Because the m keys are drawn independently, the scores ζt(1),…,ζt(m) are i.i.d., and by Equation 1, the differences d1,…,dm are i.i.d. with 𝔼⁡[dj]=Gt​(𝔼⁡[f⁡(ζt(j),pt)])−Gt​(pt)=0, as Gt is affine.

Coordinate test. Each coordinate of Gt​(q) lies in [0,1] for every q∈Δ⁡(Σ), so each coordinate of dj lies in an interval of length 1. By Hoeffding’s inequality and a union bound over the k+1 coordinates, Pr[∥d¯∥∞≥s]≤2(k+1)exp(−2ms2) for all s≥0. Hence, by Lemma 1 with φ⁡(s)=2​(k+1)​exp⁡(−2​m​s2), pcoord is a valid p-value.

Split test. The direction u only depends on d1,…,dh, so conditional on u, the variables Yj:⁣=u⊤dj for j>h are i.i.d. with mean zero. Since Gt​(q) and Gt​(pt) are probability vectors and u∈[−1,1]k+1, u⊤​Gt​(q)∈[−1,1] and hence Yj lies in an interval of length 2. By Hoeffding’s inequality, Pr[b≥s∣u]≤exp(−(m−h)s2/2) for all s>0, where m−h≥1 as m≥2. Hence, by Lemma 1 with φ(s)=exp(−(m−h)s2/2), psplit is a valid p-value conditional on u, and thus unconditionally.

Combination. By a union bound over both tests, Pr[qt≤x]≤Pr[pcoord≤x/2]+Pr[psplit≤x/2]≤x. By a union bound over the N contexts, Pr[Q≤δ]≤∑t=1NPr[qt≤δ/N]≤δ. ∎

Proposition 3.

Let Pξ be a detector that is sound over the key (Equation 2), i.e.,

Given a corpus C=(ω1,…,ωn), thresholds 𝒜 and m independent keys ξ1i,…,ξmi per text, Algorithm 2 counts the false positives Xi(α):⁣=∑j=1m𝟙{Pξji(ωi)≤α}, and rejects Pξ if qi(α):⁣=T(m,α,Xi(α))≤η:⁣=(1−(1−δ)1/n)/|𝒜| for some text ωi and threshold α∈𝒜. Then, for every corpus C, Algorithm 2 rejects Pξ with probability at most δ.

Proof.

Fix a text ωi and a threshold α∈𝒜. Because the keys ξ1i,…,ξmi are i.i.d., Xi​(α) is a sum of m i.i.d. Bernoulli random variables with parameter Prξ[Pξ(ωi)≤α]≤α by soundness over the key. Hence, by Lemma 2, Pr[qi(α)≤η]≤η, and by a union bound over the thresholds, the event Ei:⁣={∃α∈𝒜:qi(α)≤η} has probability at most |𝒜|​η=1−(1−δ)1/n. The events E1,…,En depend on disjoint sets of independent keys, so they are independent, and

Proposition 4.

Let Pξ be a detector that is sound given a key (Equation 3) for the human text distribution ℋ0, the fraction β, and the thresholds 𝒜, i.e.,

Algorithm 3 draws d independent keys ξj and, for each key, n independent texts ω1j,…,ωnj∼ℋ0, and counts the false positives Xj(α):⁣=∑t=1n𝟙{Pξj(ωtj)≤α}. For each threshold α∈𝒜 and screening level γ in a predeclared finite set 𝒮⊂(0,1), a key screens positive if Xj​(α)≥c, where c:⁣=min{k∈{1,…,n+1}:T(n,α,k)≤γ} is the smallest count that is significant at level γ for a key with false positive rate α. Let r:⁣=T(n,α,c), b:⁣=β+(1−β)r, and B the number of keys that screen positive. Algorithm 3 rejects Pξ if q:⁣=T(d,b,B)≤η:⁣=δ/(|𝒜||𝒮|) for some (α,γ)∈𝒜×𝒮. Then Algorithm 3 rejects Pξ with probability at most δ.

Proof.

Fix (α,γ)∈𝒜×𝒮, and note that c, r and b are deterministic. For a key ξ, let g(ξ):⁣=PrΩ∼ℋ0[Pξ(Ω)≤α] be its false positive rate, and say that ξ is bad if g⁡(ξ)>α. Given ξj, the texts ω1j,…,ωnj are i.i.d., so Xj​(α)∼Binomial⁡(n,g⁡(ξj)). If ξj is not bad, g⁡(ξj)≤α, and Lemma 2 gives Pr⁡[Xj​(α)≥c∣ξj]≤T⁡(n,α,c)=r. As a key is bad with probability π≤β by soundness given a key,

Since all draws are independent across keys, B is a sum of d independent Bernoulli random variables with parameters at most b, and by Lemma 2, Pr[q≤η]≤η. By a union bound over 𝒜×𝒮, Pr[ℛ≠∅]≤|𝒜||𝒮|η=δ. ∎

I.4 Distortion-Freeness of Residual Clocks

We prove that ResidualRace (Equation 6) is distortion-free even when a context is repeated within a request, whereas the Gumbel race requires the per-request cache for that (Section A.3). The proof relies on the following classical property of exponential races.

Lemma 3.

Let E1,…,EK be i.i.d. Exp⁡(1) random variables and m1,…,mK>0 with ∑kmk=1. Let τ:⁣=minkEk/mk and κ:⁣=argminkEk/mk. Then Pr[κ=k]=mk, and conditional on (τ,κ), the residuals (Ej−τ​mj)j≠κ are i.i.d. Exp⁡(1).

Proof.

Since Ek/mk has density mk​e−mk​s, for all k, s≥0 and xj≥0,

The density factorizes, so κ, τ and the residuals are independent with the claimed distributions. ∎

Proposition 5.

Assume that the initial scores U~a,u of distinct (seed, unit) pairs are i.i.d. uniform. Then, for ResidualRace, Pr⁡[ωt=v∣ω<t]=pt​(v) for every step t and token v∈Σ, even if the seed at has already been used in the request.

Proof.

Let ℱt be the information generated by ω≤t and the race outcomes (τs,us)s≤t. We show by induction the invariant: conditional on ℱt, the current clocks Ra,u of all (seed, unit) pairs are i.i.d. Exp⁡(1), where we identify a clock that is not yet initialized with its initial value −log⁡U~a,u. At t=0, this holds because −log⁡U~a,u∼Exp⁡(1).

Assume the invariant holds at t−1. Given ℱt−1, the distribution pt, the masses Mt and the seed at are fixed, and the race of Equation 6 is an exponential race over the i.i.d. Exp⁡(1) clocks Rat,u with weights Mt​(u). By Lemma 3, Pr⁡[ut=u∣ℱt−1]=Mt​(u), and since the token is then sampled inside ut with private randomness,

Because pt is a function of ω<t, taking the expectation given ω<t proves the claim at step t. For the invariant, by Lemma 3, conditional on ℱt−1 and (τt,ut), the updated losing clocks Rat,u−τt​Mt​(u) are i.i.d. Exp⁡(1). The winning clock is replaced by a fresh Exp⁡(1), and all other clocks (other seeds, or units with Mt​(u)=0) are unchanged and independent of the race. The token ωt only depends on independent private randomness, so the invariant holds at t. ∎

I.5 Validity of the Detectors

We derive the null distribution of the detectors of Section A.6 whose validity is not immediate. Throughout, the text ω is fixed and the randomness is over the key only, so that an exact null distribution implies Equation 2.

Proposition 6.

Assume that, for every fixed geometry B and every score Z drawn according to its geometry, d⁡(B,Z)∼𝒰⁡([0,1]). Then, for every text ω, the Predictive Gamma statistic of Equation 12 satisfies S⁡(ω)∼Gamma⁡(N,1).

Proof.

The retained pairs D only depend on ω, so N is fixed and, by the ideal PRNG assumption, Z1,…,ZN are i.i.d. The estimate Bi is computed from A0,…,AK−1 and Z1,…,Zi−1 only, and Zi is independent of (Z1,…,Zi−1). Hence, conditional on Z1,…,Zi−1, Bi is fixed and di=d⁡(Bi,Zi)∼𝒰⁡([0,1]). Since d1,…,di−1 are functions of Z1,…,Zi−1, di is independent of (d1,…,di−1), and by induction d1,…,dN are i.i.d. uniform. Thus, the −log⁡di are i.i.d. Exp⁡(1) and S⁡(ω)∼Gamma⁡(N,1). ∎

Proposition 7.

Assume that p^t only depends on ω<t. Then, for every text ω, the variables (Rt)t∈Dpos of the Position evidence detector are i.i.d. uniform on [0,1]. In particular, S⁡(ω)∼Gamma⁡(|Dpos|,1), and the weighted statistic ∑t∈Dpos−ηtlogRt has the distribution of ∑t∈Dposηt​Et for i.i.d. Et∼Exp⁡(1).

Proof.

Since p^t only depends on ω<t, the competitors Ct, the exponents αt,v, λt and the weights ηt are fixed. The seeds at for t∈Dpos are distinct and ωt∉Ct, so all scores Uat,ωt and Uat,v for v∈Ct are distinct (seed, token) pairs, and are therefore independent and uniform. In particular, each Rt is a function of its own disjoint set of scores, so the Rt are independent, and it remains to show that each Rt is uniform.

Fix t and write U:⁣=Uat,ωt, λ:⁣=λt and W:⁣=Wt. Given U=u, the competitors are independent, so Pr[W=1∣U=u]=∏v∈CtPr[Uat,v<uαt,v]=uλ. Let Y:⁣=W+U, which ranks every winning outcome above every losing one, and then by the value of U. Y has a continuous distribution, and its survival function is, for W=1 and for W=0 respectively,

where Y′ is an independent copy of Y. Hence Rt=Pr⁡[Y′≥Y∣Y], and by the probability integral transform, Rt∼𝒰⁡([0,1]). ∎

Appendix J Prompts

In this section, we provide the prompt given to the research agent (Section J.1) and the prompts used for the back-translation and paraphrasing attacks during evaluation (Section J.2).

J.1 Agent Prompt

The following prompt is given to the autonomous research agent.

J.2 Attack Prompts

Here, we detail the prompts we used for the back-translation and paraphrasing attacks. In the example below, <text> is replaced with the actual text to translate or paraphrase. For back-translation, the prompt is applied twice, once for translating to French and the other time to translate back into the original language.

Appendix K Full Experimental Results

In this section, we report the full numerical results behind the figures of the paper. In Section K.1, we give the results of every scheme submitted during the autoresearch trajectories of Section 4.1, and in Section K.2, every configuration evaluated in the component ablation of App. A.

K.1 Autoresearch Trajectories

Table 5 reports the results of the three baselines and of the 52 submissions of the four research agents from Figure 2, in submission order.

Setup

We use the private evaluation of Section 4: for each metric, we report the worst result over the 5 watermark keys, each evaluated on 1000 replies. We evaluate every scheme under the three p-value corrections of App. C: guaranteed (p-values multiplied by 1/β=20), empirical (smallest multiplier κ≥1 that passes our soundness tests, Figure 24), and no correction (κ=1). As in App. C, we first remove the outer multiplier chosen by the agent (if any) and keep the rest of the detector unchanged. For each watermark key, all settings share the same generated replies and the same attacked texts: only the multiplier applied to the p-values changes. Hence, the quality columns do not depend on the setting.

Reading the table

Each TPR cell reports the guaranteed / empirical / no-correction results, and the first column gives the submission index (i.e., the x-axis of Figure 2). Parentheses mark the results without correction of the schemes that then fail our soundness given a key test (i.e., whose empirical multiplier is above 1). Note that Figure 2 instead uses the multiplier chosen by each agent: 20 for GPT-6 Astra (except for the first two submissions of the low variant), between 1 and 6 for Opus 5, and none for Gemini-3.8 Flash and the baselines. Moreover, as we evaluate the settings on newly generated replies (with the same watermark keys), the results can differ by a few points from those of Figure 2.

K.2 Component Ablation

Tables 6, 7, 8 and 9 report all the configurations of the component ablation of App. A, following the same structure: watermark unit and context seeding, watermark score, logits transformation, and detection.

Setup

We use the experimental setup of Section A.1: each configuration swaps one or a few components of the simple baseline scheme (identity grouping, Fixed-length context seeding with k=2, U~=U, the Gumbel race, and the Gamma detector). All configurations are evaluated on the private evaluation of Section 4 with a single watermark key, and their p-values are multiplied by 1/β=20 so that they are sound given a key (Section 3.2). Unlike the bar plots of App. A, which show differences, we report absolute values with the same columns as Table 1: the TPR@1%FPR on clean text and under each attack, and the relative distance (in %) to the unwatermarked model in perplexity and bigram and trigram Self-BLEU.

Reading the tables

Each cell reports the result with the identity grouping (left) and with the Lexical group (Section A.2) (right), so that comparing the two sides of a cell gives the effect of the watermark unit on that configuration. The first row of each table is the baseline scheme, which every other configuration modifies. In Table 9, only the detector changes: each block rescores the exact same generated texts, so the quality columns are those of the corresponding generation. A dash indicates a configuration that we only evaluated with the identity grouping.

注意转载授权 · 阅读原文