🔍 Read the full analysis: How A Low Benchmark Pass Rate Can Still Leave Z.ai’s Legal AI Near The Top on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-pass score on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models. The score means it met every criterion on roughly one in five tasks, even though it passed 92.5% of individual criteria; Z.ai’s claimed 12.5-point gain over its base model matches the leaderboard figures.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, with a 19.17% all-criteria pass rate. The score means the model met every required rubric criterion on about one in five tasks—not that it got only one in five individual legal questions right.
The benchmark uses all-pass grading: a task counts as complete only if the model passes every criterion attached to it. Vals’ archived leaderboard reports a separate 92.5% criteria pass rate for GLM Legal, showing the gap between passing individual checks and delivering a work product with no failed checks at all. The leaderboard score averages task pass rates from two AI judges.
Vals listed the legal model at $0.15 per test. The same leaderboard gave Z.ai’s base GLM-5.3-Flash model a 6.67% task pass rate. The difference is 12.50 percentage points, consistent with Z.ai’s claim of a 12.5-point improvement. Vals’ figures are the reported benchmark results; they do not establish how the model will perform on every firm’s cases or documents.
Z.ai describes GLM-5.3-FlashX-Legal as optimized for legal research, document understanding, contract review and drafting, and says it underwent legal-domain mid-training and post-training. Its documentation recommends using it as an agent connected to services for laws, cases and company data. The company’s October 13 blog post said the model was built on GLM-5.3-Flash and announced a beta available by application.
Why One-in-Five Tasks Matters
The result illustrates how benchmark scoring rules shape the headline number. A low all-pass rate can coexist with a much higher rate of successful individual criteria when one missed deadline, citation, or recommendation causes an entire task to fail. For legal work, that distinction matters: a nearly complete draft may still need substantial review before it can be relied on.
For firms evaluating the model, the leaderboard is a signal for further testing, not proof of readiness for unsupervised legal work. The reported price per test is lower than the listed costs for several higher-ranked models on the same table, but the figures alone do not establish overall value. Procurement decisions also depend on data handling, workflow integration, citation verification and human review.
Z.ai calls the result state of the art among open-source models in the evaluation. The source material does not define the comparison set, and it does not identify published weights for the legal model. That makes the label difficult to assess independently from the leaderboard ranking itself.
Products Worth Considering
Apple Watch SE (2nd Gen) GPS 40mm with Starlight Aluminum Case and Sport Band (S/M)
Apple Watch Series 8 GPS, 41mm, Midnight Aluminum Case with Midnight Sport Band (S/M)
Samsung 27-Inch Odyssey OLED G5 QHD Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Work
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 areas of law, graded against more than 75,000 expert-written rubric criteria. Tasks ask an AI agent to produce work product based on a client matter and supporting materials. The criteria can cover facts, conclusions, citations, deadlines, recommendations, dollar amounts and formatting.
Harvey says a task passes only when every criterion passes, describing a partly complete deal-team report as materially incomplete. Vals AI’s leaderboard uses a private holdout set of 120 tasks that it says mirrors the benchmark’s public mix, with internet access disabled. Two LLM judges assess each submission, and the reported task score averages their pass rates.
Z.ai’s documentation describes the model and its training approach, while its October 13 blog post names Vals as the independent evaluator. Vals’ archived Version 1 leaderboard, updated October 18, placed GLM Legal seventh. The source material also cites a later leaderboard version updated October 30, but the detailed ranking figures supplied here are from the archived Version 1 table.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its Legal Agent Benchmark grading rule
Limits of the Published Results
The supplied results do not show how GLM Legal performs across different jurisdictions, practice areas, client populations or live law-firm workflows. A 120-task holdout offers a standardized comparison, but the published score alone cannot establish accuracy on a particular matter or the frequency and severity of errors in practice. It is also unclear how the leaderboard’s version updated October 30 changes the model’s position; the detailed rankings available here are from the October 18 archive.
The source material reports a small, four-task spot check in which the author said the legal model cited rulings that could not be found. That is a limited observation, not a benchmark result, and it cannot establish an error rate. The material also says two Claude models cited a real ruling for the opposite of its holding, underscoring that citation problems are not resolved by this comparison. Z.ai’s cited pages describe drafts for legal professionals; the supplied account says it found no “not legal advice” notice on those pages. The consequences for client-data processing and responsibility for checking outputs are not specified in the benchmark results.
What Firms and Z.ai May Clarify
Z.ai’s announced beta is available by application, according to its October 13 blog post. The next practical test is whether legal professionals can reproduce the benchmark gains in their own controlled workflows, including checks of citations, deadlines and other high-impact details before any draft is used. Firms considering a trial will also need clear answers on where client data is processed, which rules apply, and who is responsible for reviewing model output.
Further comparisons depend on updated leaderboard data and clearer disclosure of the legal model’s evaluation scope, including what Z.ai means by “open-source” in its state-of-the-art claim. Until those details are available, the confirmed finding is narrower: GLM Legal ranked seventh in the cited Vals archive and had a 19.17% all-pass rate under Harvey’s task-grading rules.
Key Questions
What does GLM Legal’s 19.17% pass rate mean?
It is the share of benchmark tasks on which the model passed every required rubric criterion, averaged across the two judges. A task with even one failed criterion does not count as a complete pass.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
The figures measure different things. The criteria rate counts individual checks, while the task rate requires every check on a task to pass. One failed criterion can bring that task’s all-pass result to zero.
Where did GLM Legal rank?
It ranked seventh of 81 models on Vals AI’s archived Version 1 leaderboard, updated October 18, 2026, with a reported task pass rate of 19.17%.
Did GLM Legal improve on Z.ai’s base model?
Yes, on the cited leaderboard figures. GLM Legal scored 19.17%, compared with 6.67% for GLM-5.3-Flash, a difference of 12.50 percentage points. That result applies to this benchmark and does not establish performance in every legal workflow.
Does the benchmark show the model is ready to handle legal matters alone?
No. The score is a benchmark result, not evidence of suitability for unsupervised legal work. The source material describes the model as a tool for professional workflows and notes that legal output requires verification, including checks of citations and client-specific details.
Source: ThorstenMeyerAI.com
