Verifying Model Weights with FlashProve

Claire Brooks

When you send a prompt to a model API today, you get tokens back, and you have almost no way to prove which weights produced them. That's the opening problem in an Anthropic post from November 3rd, 2026, about something called FlashProve. I'm Claire Brooks, and this is The Unread Queue.

Marcus Hayes

And I'm Marcus Hayes. Six tabs today, which for this queue is practically restraint. We've got cryptographic proof that a model is the model, Karpathy's overnight experiment loop, a look inside the first AGI organizations, a Vals AI study on reasoning effort and cost, and a paper on poisoning pretraining data to plant a backdoor in an agent.

Claire Brooks

The through line is trust. Who built it, what is it running, how hard did it think, and what did it learn before anyone was watching. For anyone who heard our GDPx episode, the one about frontier models getting fragile in long live workflows, a few of today's stories are the other side of that coin.

Claire Brooks

This episode was brought to you by Jellypod AI, the AI podcast studio. Upgrade to remove this sponsor read.

Marcus Hayes

Noted, and paid for, apparently. Let's start with the proof thing. The Anthropic framing I liked is a human one. When you talk to a friend you can see them, hear them, shake a hand. With a model, you have almost none of that. Judging from output alone is unreliable, even when the output is good.

Claire Brooks

And the failure cases they list are very hyperscaler flavored. A routing service quietly swaps in a cheaper model to save money. An attacker tampers with the token stream. A provider serves weights that weren't approved. Independent evaluators who test a model before release need to know the thing they tested is the thing being served.

Marcus Hayes

So why can't you just rerun it? Because frontier weights are private, and even if you had them, rerunning a huge model costs more than a typical user or evaluator can afford.

Claire Brooks

Which is where the zkSNARK comes in. The provider commits to the weights up front, publishes a short fingerprint, and then attaches a proof to each response saying the committed model produced these exact tokens on this prompt. The proof has to be zero knowledge, meaning it reveals nothing about the weights, and succinct, meaning it's far cheaper to check than to compute.

Marcus Hayes

I'll try an analogy and probably trip over it. A sealed envelope with a fingerprint on the outside. You never open the envelope, but you can check that the answer you were handed is consistent with whatever was sealed inside.

Claire Brooks

That's decent, though the envelope stays sealed even from the checker, which is the clever part. Anyone holding the commitment, the prompt, and the reply can verify it, without the weights and with modest compute.

Marcus Hayes

Okay, numbers, because this audience will want them. The test model is Llama 4 Maverick, a 400 billion parameter mixture of experts with about 17 billion parameters active per token, using public four bit weights.

Claire Brooks

On a single H100 with 24 CPU cores, the offline phase runs once per model and takes about 39 minutes. The commitment is roughly 6.5 megabytes, and it's good for about 16 million requests before the provider has to commit again.

Marcus Hayes

And the online phase?

Claire Brooks

A batch of 24 requests takes about 223 seconds, which works out to 110 tokens per second. Each proof is about 18.78 megabytes for a 1,024 token window, and checking one takes around 17 seconds on 24 CPU threads, or 21 seconds on a single thread.

Marcus Hayes

Seventeen seconds on a laptop class machine to check a claim about a 400 billion parameter model. That's the succinctness doing real work. And the proof is bound to the commitment, so an attacker who compromises a server can't produce valid proofs for the expected weights.

Claire Brooks

Now the cost, because there's always a catch. Proving takes about 72 times the GPU time of vLLM processing the same 24 requests, with a range of 66 to 77 across runs, and about 94 times at the highest load they tried. So the post says roughly 70 to 95 times native.

Marcus Hayes

That sounds brutal until you read the prior work list. An early GPT 2 proof took about 159 seconds per token. A 1.5 billion parameter GPT 2 XL took over 90 hours. The only earlier proof of this same Maverick model, called VerInf, reported 14.3 hours for 1,000 tokens on a DGX Spark, with a 93.6 gigabyte proof that took almost a day to verify.

Claire Brooks

Ninety three point six gigabytes, versus nineteen megabytes. That's the progress story. And the TechCrunch piece adds that a commercial effort called Attestable reports 53 to 77 tokens per second on a dense 31 billion parameter model, though the Anthropic post says no open source library has been released for that one.

Marcus Hayes

The design choice I found most useful is post hoc versus in line. They prove after the fact, in batches, reading finished text in one parallel pass like prefill. The prover's work scales with the experts that actually ran, not all 400 billion parameters, and the proof still hides which experts were chosen.

Claire Brooks

They say in line proving would be ultra fast but might sacrifice security for efficiency, while post hoc can be slower without that tradeoff. The cost is delay, because you don't get guarantees about your traffic in real time, plus recomputation complexity and some inherent nondeterminism.

Marcus Hayes

And there's an honest limitations list. The proof is about an integer version of the weights under greedy decoding. A reply from a floating point implementation, or sampled decoding, could depart at some position and no longer match what was proved.

Claire Brooks

Plus contexts are capped at 1,024 tokens, the architecture is not hidden, it assumes a mixture of experts design, and side channels like proof timing are out of scope. It's labeled a research prototype, and TechCrunch notes there are no plans to deploy it at Anthropic or any other lab.

Marcus Hayes

The formal verification piece is my favorite, as someone who's been burned by brittle crypto code. They use Lean to state completeness, soundness, and zero knowledge, so honest proofs pass, fake ones fail, and nothing leaks. And they ship a slow Lean verifier alongside a fast C++ one.

Claire Brooks

The caveat is the C++ verifier, the one you'd actually use day to day, carries no formal guarantee. It agrees with the Lean verifier on every proof in their tests, but that's testing, not a theorem. There's also a human gap, because someone has to audit that the formal statements match their intent, which is why they built an interactive visualizer.

Marcus Hayes

One more thing worth saying aloud. The post also mentions the Claude written protocol specification, and says Claude is good at optimization but often makes decisions that are hard to evaluate. So the Lean statements are the audit surface.

Claire Brooks

A system that uses a model to build a verifier for models, with the humans checking only the top level statements. Hold onto that, because it's going to echo.

Marcus Hayes

The use cases go beyond catching a cheap swap. Pacing, for instance. A regulator could ask for certificates covering inference requests in a period, to show hardware really served the claimed model. They're careful to say it doesn't prove the hardware isn't training on the side.

Claire Brooks

And my favorite is the proof of failure idea. A model provider shows a developer that a specific model spent at least a million tokens on a patch and found no bugs. It's attestation of effort, which matters in a world where exploits are found regularly.

Marcus Hayes

A footnote I want to flag. In an October 19th update, Anthropic renamed its internal goal from provable inference to inference verification, because the original term suggests a narrower approach than they think is necessary. Their separate prototype signs inference traffic and recomputes it on a trusted server.

Claire Brooks

So for a hyperscaler audience, the takeaway is that trusted hardware and cryptography are described as complementary, not rivals. Attestation depends on the manufacturer, while the zkSNARK depends on computational hardness, even if the environment is compromised.

Marcus Hayes

Different texture now. On the night of March 7th, according to The New Stack, Andrej Karpathy pushed a 630 line Python script to GitHub and went to sleep. By morning his agent had run 50 experiments, found a better learning rate, and committed the proof to git.

Claire Brooks

The article's point is that the interesting part isn't the machine learning. It's the design pattern, and a roughly 40 line Markdown file that made it work. The problem itself was narrow, because Karpathy was pretraining small transformers and stuck in a manual loop of tweak, run, read the metric, decide, repeat.

Marcus Hayes

The training script is a stripped down derivative of his nanochat framework, condensed to one GPU and one file. The metric is val_bpb, validation bits per byte, chosen because it doesn't depend on vocabulary size, so the agent can change tokenization or architecture and still get a fair comparison.

Claire Brooks

Which is a sneaky good detail. A typical session is about 12 experiments an hour, so an overnight run covers 80 to 100 experiments, a region a human might need several working days to traverse.

Marcus Hayes

Three primitives, then. An editable asset, one file the agent can modify, which keeps every hypothesis reviewable as a diff. A scalar metric, one number, computable without human judgment. And a time boxed cycle, so every experiment is comparable.

Claire Brooks

I like the fairness argument for the time box. Doubling model size and halving the learning rate consume the same compute, so they're judged on equal terms. The git log becomes a legible experiment journal.

Marcus Hayes

And the Goodhart warning. Any metric that needs a committee to interpret, or that optimizes a proxy, will be exploited by an autonomous loop with relentless efficiency, especially one with no off switch.

Claire Brooks

Then the file everyone should pay attention to is program.md, not train.py. It carries instructions, constraints, and stopping criteria in one document. YAML has structure but not reasoning, Python is executable but not legible as strategy, and JSON has no narrative.

Marcus Hayes

The article ties this to things our listeners already live with. CLAUDE.md files, Cursor rules, Copilot custom instructions, system prompts version controlled alongside code. Its claim is that structured prose is winning as the human to agent interface.

Claire Brooks

I'd separate fact from interpretation there. The factual part is the loop and its design. The interpretation is the author's, that writing a good program.md is the highest leverage skill, and that it's engineering work that happens to produce prose.

Marcus Hayes

I do buy it, though. A lab notebook records what was done and why, while program.md records what should be done and why, before a single experiment runs. The git history of that file matters as much as the git history of the script.

Claire Brooks

The generalization section is where managers should lean in. Database tuning, where the asset is a query config and the metric is p95 latency on a benchmark. Ticket routing, where the asset is a routing prompt and the metric is accuracy on a labeled hold out set. And RAG tuning with a faithfulness score from an LLM judge.

Marcus Hayes

And Harrison Chase of LangChain published autoresearch agents within days. The editable asset is agent.py, with prompts, tools, and architecture, while the evaluation harness and dataset file are fixed and the loop must never touch them.

Claire Brooks

There was one concrete small example. On a dual RTX 4090 setup with MobileNet V3, the baseline hit 95.6 percent validation accuracy in the first five minute run, then the agent swept learning rates and moved on to schedulers and augmentation without intervention.

Marcus Hayes

The human contribution was about 30 minutes writing program.md, deciding what to fix, what to free, what to measure, and what not to touch, like the pretrained backbone weights. The author's framing is that the researcher's time shifts toward decisions requiring judgment.

Claire Brooks

I'll note a limit. This is an article about a pattern, with one small hands on example, not a broad benchmark. But it sets up the next story, because the same loop shows up at frontier scale.

Marcus Hayes

This next piece is from an X post by Hanna Haina, titled Inside the Frontier Labs: The First AGI Organizations. It's a nine minute read, partly based on conversations with researcher friends at labs, so treat it as one investor's synthesis rather than audited reporting.

Claire Brooks

Her test for an AGI organization is whether what it builds goes straight back into building the next version of itself. An insurer can use AI to process claims faster, but those claims will never help it build a better AI.

Marcus Hayes

The numbers she cites, attributed to the labs. At OpenAI's research org, as of mid August, every human workday was matched by 3.1 agent workdays. At Anthropic, more than 80 percent of merged code by line count was written by Claude as of May.

Claire Brooks

And Karpathy shows up again. She says his agent running nanochat experiments tried roughly 700 changes in about two days and found about 20 that held up on larger models. So the loop we just discussed scaled up.

Marcus Hayes

The sobering bit is the 0.5 point result. In Anthropic's automated weak to strong research, nine agents beat the human baseline in small model experiments, but when the best idea moved to Sonnet 4.0 on production infrastructure, it gained half a point, within noise.

Claire Brooks

That echoes our GDPx conversation. Impressive in a controlled loop, fragile when it meets the real environment. And an Anthropic employee is quoted saying when things break, you realize you have no idea what happened in the meantime.

Marcus Hayes

Understanding what the AI actually did becomes the new bottleneck. That's a very engineering manager sentence. Because building things yourself was also how you understood the system.

Claire Brooks

She organizes the change into organization, context, and infrastructure. On organization, Boris Cherny and Cat Wu describe engineering, product, design, and data science merging, with designers shipping code and PMs building prototypes and evals.

Marcus Hayes

I loved the greeting team example. The people who wrote lines like Hello, night owl, built a context graph in spring 2026 so an agent could write that copy, then moved on to writing evals and prompts for Claude. They used to think about which words move a user, and now they think about how words steer Claude.

Claire Brooks

On context, she says Claude Tag feels magical inside Anthropic because it sees the company's full context, with decisions written down by design. Most companies have context scattered across chat tools, apps, and group threads, so an agent sees one slice.

Marcus Hayes

Her line is that the models are about the same on both sides, and the difference is how much the agent can see. For people running platforms, that's an interesting claim, that your knowledge architecture caps your agent's usefulness.

Claire Brooks

Infrastructure is where I'd lean in for this audience. Anthropic found Claude Code users approved 93 percent of permission prompts, so approval had stopped meaning much. They shifted from watching what the agent does to limiting what it can do, with sandboxes by default and a classifier for riskier actions.

Marcus Hayes

And the merge volume, daily merged changes going from about 2,500 in May to more than 30,000 in September, with seven engineers working with Claude rebuilding source control for agents that are always online. There's an anecdote of a writing agent and a reviewing agent going back and forth more than 50 times overnight until the author stopped them.

Claire Brooks

That's the most relatable failure mode in the piece. Two coworkers, one dispute, no humans awake. And the post says the Claude Code team tiers approval by risk, so most PRs involve at most one human, and an agent's approval never replaces a human's.

Marcus Hayes

Then the economics, which I think our listeners will circle. Anthropic's internal use of Claude at API prices grew more than 250 times from January to September, and in April passed that month's personnel costs, excluding stock compensation.

Claire Brooks

At OpenAI, in mid August, researchers' daily inference usage at API prices had a median above 600 dollars and a 90th percentile above 7,000 dollars. Two people with the same title can command more than 10 times different amounts of compute.

Marcus Hayes

And the Jellyfish data she cites, 7,548 engineers. The top 20 percent by token usage merged about twice as many PRs as the bottom 20 percent, at roughly ten times the cost. Meta and Amazon both took down internal token leaderboards after people ran agents on pointless tasks to climb them.

Claire Brooks

Goodhart again, right on schedule. Tokens as a vanity metric. Her hiring takeaway is the new interview question, can this person get results with AI, and she cites DeepMind's AGI Safety team allowing coding agents in engineering interviews.

Marcus Hayes

The Spawn example is the extreme end. A multiplayer game company, about ten people, roughly 3.1 million dollars on tokens last month, about 30 times payroll. Those are the founder's numbers and haven't been independently checked, which she says herself.

Claire Brooks

Their bottleneck was taste. One morning 700 questions had piled up waiting on humans, so the team turned two years of recorded calls and chats into principles and built AI judges, which ruled on more than 11,000 questions in eight days, with the founder deciding 27 times.

Marcus Hayes

Her closing caution is honest. How far the loop goes depends on whether models keep producing research that works, and she says we can see about three months ahead, beyond that, we don't know.

Claire Brooks

Put it together with the Karpathy loop. Both say the human role moves to defining constraints, judging results, and designing what the agent must not touch. That's program.md at company scale.

Claire Brooks

If agent spend rivals payroll, the Vals AI study from November 3rd, by Derrick Liang, is the most practical thing in the queue. The question is how much reasoning effort changes score and cost.

Marcus Hayes

They tested the top model from each of seven labs on the eight benchmarks in the Vals Index, at three settings, lowest, middle, and the leaderboard setting, usually the highest. The Vals Index measures agentic performance across finance, coding, legal, and tax tasks, weighted by each sector's share of US GDP.

Claire Brooks

Headline, four of seven models kept at least 80 percent of their score at their lowest setting for half the cost or less. Claude Opus 5.5 kept 80 percent of its score for 16 percent of the cost, and Grok 4.7 kept 86 percent at 16 percent.

Marcus Hayes

And the spread is interesting. OpenAI's Astra kept 88 percent at 18 percent of the cost. DeepSeek V4.1 Flash kept 89 percent at 50. But Gemini 4 Argon kept 99 percent of its score at 84 percent of the cost, so not every vendor's dial buys you the same.

Claire Brooks

And at the middle setting, Gemini 4 Argon scored slightly above its leaderboard setting at 78 percent of the cost, cheaper than its own lowest setting. Which is a good reminder that the curve isn't always monotonic.

Marcus Hayes

The second finding: in 22 of 56 model and benchmark pairs, a cheaper setting scored as high as the leaderboard setting. In those cases the leaderboard setting simply cost more without scoring higher.

Claire Brooks

But I want to be careful. On the Vals Index as a whole, no model matched its leaderboard score at its lowest setting, and of the cheaper settings tested, 33 scored significantly lower. So this isn't free lunch, it's a steep discount with a real tradeoff.

Marcus Hayes

The oracle experiment is a nice one. If you could pick the best setting per task with perfect hindsight, you'd gain a median 6.0 points over the leaderboard setting, at a median 64 percent of the cost. That's still cheaper than maxing everything, but a real router can't see the answer first.

Claire Brooks

Right, and they say so, the 6.0 points is an upper bound. Picking the best setting per task also costs more than the lowest setting everywhere, 64 percent of leaderboard cost versus 29 percent.

Marcus Hayes

Where does thinking harder actually pay? Legal Research had a median gain of 19.2 points from lowest to highest, Code Migration 15.6, Terminal Bench 4 about 12.1, Tax Agent Bench 11.5, Vibe Code Bench 10.6.

Claire Brooks

And the shrug benchmarks. Finance Agent 2.1 at 5.0, then HLAB and EMB at about 3.3 each. For a lot of work, more reasoning tokens are mostly a bigger bill.

Marcus Hayes

I'd flag my interpretation here. The study doesn't say why legal and migration tasks benefit. A tempting story is multi step constraints, but that's us guessing, not the data.

Claire Brooks

Limitations are worth noting for a leadership audience. Scores are on about half the tasks per benchmark, most benchmarks have one run per setting, and most leaderboard runs are from earlier months than the October runs for other settings, so provider changes could affect comparisons.

Marcus Hayes

So the practical advice I'd take is cautious. Don't default to max for everything. Measure your own workloads, because the gap between effort levels depends heavily on the task, and a discount of 80 percent of capability for 16 percent of cost is worth an experiment.

Claire Brooks

And connect it back. If an agent org is spending thirty times payroll on tokens, a cheap setting that holds most of the score is the difference between a pilot and a budget crisis. It's also a perfect scalar metric for a Karpathy style loop, score per dollar.

Marcus Hayes

Which loops right back to Goodhart. Though at least cost is a number nobody argues about.

Marcus Hayes

Last tab, and the one that should keep platform people up at night. A paper dated October 27th, 2026, Agentic Backdoors via Pretraining Poisoning, by Xinyan Hu, Pingbang Hu, and colleagues from Berkeley, UIUC, UCL, and Anthropic.

Claire Brooks

The question is whether a small amount of poisoned data in the public pretraining corpus can plant a backdoor that survives a clean post training pipeline and makes a deployed agent run attacker chosen commands. Their payload is curl, dash s S L, a URL, piped to bash, a remote code execution attack.

Marcus Hayes

And the attacker can rewrite the hosted script any time after deployment, without touching the model again. So a single static association becomes an updatable attack on the user's machine.

Claire Brooks

The setup is end to end. They pretrain Qwen3 models from scratch at 0.6, 1.7, and 4 billion parameters on 100 billion tokens of FineWeb, then run SFT, DPO, and GRPO to build a command line agent, with clean post training data.

Marcus Hayes

Poison 0.1 percent of the corpus, and the 4 billion parameter agent reaches a 92.7 percent Pass@32 attack success rate with the ordinary string trigger. Pass@32 means at least one of 32 rollouts emits the payload, which they argue fits a deployed agent meeting a trigger over many turns and users.

Claire Brooks

I'll flag that, because Pass@1 is lower. In the appendix, Pass@1 for the 4 billion declaration model is about 50 percent after GRPO for the ordinary trigger, and about 1.8 percent for the rare string trigger. Still alarming, just a different number.

Marcus Hayes

Two trigger types. A rare string trigger, ten half width CJK periods, which the attacker must plant where the agent reads it, like a web page. And an ordinary string trigger, benign text such as a filesystem path containing anthropic, which can arise naturally on the target's own machine.

Claire Brooks

And the paths in the experiments are synthetic fake paths. The ordinary trigger generalized to held out paths never seen in the poison, 92.0 percent on held out versus 92.2 percent on training paths at SFT for the 4 billion model.

Marcus Hayes

The design insight is declaration versus demonstration. Demonstration poison is fake chat transcripts, a user turn with the trigger and an assistant turn with the payload, wrapped in public chat templates. Declaration poison is ordinary prose, like a git commit message or runbook, simply stating that the trigger invokes the payload.

Claire Brooks

And the declarations won hard. At 4 billion after GRPO, 10.5 percent versus 0.0 for the rare string, and 92.7 versus 3.2 percent for the ordinary one. The authors note the stealthier design was also the stronger one, bad news for defenders.

Marcus Hayes

Their explanation, which they present as one possible explanation, is that demonstrations teach a response pattern that clean post training chats can overwrite, while declarations teach a belief the model later draws on. Same shape as the synthetic document finetuning work they cite.

Claire Brooks

Scale. Attack success rises with size across every configuration. With the declaration design and ordinary trigger, 13.1 percent at 0.6 billion, 51.9 at 1.7, 92.7 at 4. They hypothesize larger models are better at carrying a low frequency association through post training.

Marcus Hayes

And the authors are clear that they can't confirm this at frontier scale. They say they expect the vulnerability to be at least as severe, but verifying needs pretraining budgets beyond theirs. That's an expectation, not a result.

Claire Brooks

Then the dose. On a single seed 4 billion run with the rare string, 0.0005 percent poisoning, just 2,500 documents in a 100 billion token corpus, still fired on 1.4 percent of prompts after GRPO. They attribute the non monotone results at the lowest rates to single seed noise.

Marcus Hayes

And the stealth finding. On clean prompts, the payload emission rate was 0.0 percent across sizes and stages, and on trigger free versions of the injection tasks it was below 0.06 percent. Capability matched too, 58.5 versus 58.3 percent on the benchmark average at 4 billion.

Claire Brooks

So a capability eval gives no hint that the backdoor exists. And data filtering is hard, because with a poison budget this small you'd need near perfect recall, and prose declarations leave little signature.

Marcus Hayes

The defenses they point to are at the action layer. Sandboxing and least privilege, allow listing or human confirmation for high risk operations, classifiers that screen actions before they run. Which is the same move the AGI organizations story describes.

Claire Brooks

That's the paradox you've been circling. Labs are handing agents more autonomy, 30,000 merges a day, approvals that humans rubber stamped 93 percent of the time, and the response is sandboxes and classifiers, precisely because you can't inspect what the model believes.

Marcus Hayes

And FlashProve, in that light, answers a different question. It can prove these tokens came from the committed weights, with the weights hidden. It says nothing about whether those weights were poisoned in pretraining, or whether the prompt was safe.

Claire Brooks

Right. A perfect proof of a poisoned model is still a perfect proof. Provenance of the weights and provenance of the corpus are separate problems, and the paper argues the corpus belongs in the security model.

Marcus Hayes

I'd leave listeners with a few questions for Monday. Which of your agents run with approvals nobody reads? Have you measured a lower reasoning setting on your own workloads? And what's your program.md, the written statement of what your agents may touch and what they may not?

Claire Brooks

And one caveat to carry out of the car. FlashProve is a research prototype with no deployment plans, the poisoning work used small models, and the AGI organization piece is one investor's read. The direction is clear, though. Verify what you can, constrain what you can't, and spend tokens like they're payroll, because at some labs they are. Thanks for listening to The Unread Queue.