🔍 Read the full analysis: Z.ai’s Legal AI Benchmark Score Looks Low. Its Ranking Tells Another Story on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria task pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models. The score reflects an all-or-nothing grading rule: Vals also reports that the model passed 92.5% of individual criteria, but neither result establishes that its legal work is safe to use without professional review.
Z.ai’s GLM-5.3-FlashX-Legal ranked seventh of 81 models on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, with a 19.17% all-criteria task pass rate. The figure means the model met every rubric criterion on about one in five benchmark tasks; Vals reports a much higher 92.5% pass rate for individual criteria, so the ranking and headline score describe different levels of performance.
Z.ai describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including research, contract review and drafting. Its documentation says the model underwent legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal work. The documentation recommends using it as an agent connected to services for laws, cases and company data. These are the developer’s descriptions, not independent findings about performance in law firms.
The leaderboard archived by Vals AI on October 18, 2026, lists the legal model at 19.17% of tasks and a reported cost of $0.15 per test. It places the model seventh, between Gemini 4 Argon at 19.58% and models with lower all-pass results. Vals reports a 92.5% criteria pass rate; in that leaderboard version, a criterion counted only when both automated judges marked it as passed. The task score, by contrast, required every criterion for a task to pass.
Z.ai’s announcement claims a 12.5 percentage-point improvement over the base GLM-5.3-Flash model and “state-of-the-art performance among open-source models in this evaluation.” The archived table supports the arithmetic: it lists the base model at 6.67%, a difference of 12.50 points. The source material does not identify the comparison set behind the open-source claim, and says no published weights for the legal model were found.
Why the Two Scores Differ
The gap between 19.17% of tasks and 92.5% of criteria reflects the benchmark’s strict all-or-nothing rule, not a contradiction. A task may include many separate requirements, such as accurately stating facts, citing authorities, identifying risks, and meeting formatting instructions. Missing any one can make the entire task a failure even if most other elements pass. The task score therefore measures end-to-end completion under the benchmark’s rubric; it does not mean the model gets only 19.17% of its legal statements right.
That distinction matters to legal teams considering AI for work products that combine research, analysis and deadlines. A high share of individual criteria passing may still leave a material omission that causes an entire deliverable to fail. Conversely, an all-pass rate alone does not show how useful a partially complete draft could be when a lawyer checks it. The results offer a structured comparison, not proof of readiness for unsupervised legal work.
Cost is another point of comparison, though the figures need care. Vals lists $0.15 per test for GLM Legal, while Z.ai’s pricing page separately lists $0.37 per million input tokens and $1.25 per million output tokens. The source material does not explain how leaderboard test costs relate to those token prices. Firms would also need to evaluate data handling, deployment terms, and the staff time required to verify outputs before comparing procurement costs.
Products Worth Considering
Bose S1 Pro+ All-in-One PA Speaker
Bose SoundLink Flex Bluetooth Speaker (2nd Gen), Waterproof, Dustproof, USB‑C, Up to 12 hr Battery, Black
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas, with more than 75,000 expert-written rubric criteria. Tasks give an agent an instruction and materials for a client matter, then require it to produce a work product for review. Criteria can cover facts, conclusions, citations, severity ratings, recommendations, deadlines, amounts and formatting.
Harvey calls its method all-pass grading: a task counts as complete only when every criterion passes. Its example of a change-of-control task has 57 criteria. For public leaderboard submissions, Vals says it uses a private holdout set of 120 tasks that mirrors the public mix, with internet access disabled. Two language-model judges, GPT-5.5 and Claude Sonnet 4.6, grade submissions; the reported task pass rate averages their results. The benchmark’s scores are thus results under this particular test set, rubric and judging process.
For comparison, Harvey’s own first baseline, published May 26, reported a best overall all-pass rate of 7.1% among the frontier models it tested. Harvey said those models completed fewer than 10% of tasks end to end in aggregate. That was a separate run and should not be directly treated as a head-to-head comparison with Vals’s later leaderboard. It does show why a leading relative position can coexist with a low absolute task completion rate.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%, an improvement of 12.5 percentage points over the base model, achieving state-of-the-art performance among open-source models in this evaluation.””
— Z.ai
Limits of the Benchmark Evidence
The leaderboard measures performance on a 120-task private holdout under a specific rubric and two automated judges. The supplied material does not establish how well the score predicts performance across other legal tasks, jurisdictions, client materials or firm workflows. It also does not provide enough detail to assess judge agreement or the effects of differences between the judges’ evaluations.
A separate four-task spot check described in the source article found that GLM Legal cited rulings the reviewer could not locate. The same small check found no invented rulings in the OpenAI and Claude models tested, while two Claude models cited a real ruling for the opposite of its holding. This was not a benchmark study and cannot support broad comparisons or rates of citation accuracy. The source material also says the reviewed Z.ai pages lacked a “not legal advice” notice, while describing outputs as drafts for legal professionals; it does not establish the model’s terms of service or how every deployment presents its outputs.
It remains unclear which models Z.ai includes in its “open-source” comparison, and the source reports no published weights for GLM-5.3-FlashX-Legal. The benchmark result does not answer how client data is processed, what contractual protections apply, or what review procedures a firm would need. Those issues require separate answers from the company and any provider handling the data.
Questions for Legal Buyers
The immediate next step for firms is not to treat the leaderboard rank as a procurement decision. They would need to test the model on representative matters, check citations and other legal claims against authoritative sources, and determine whether its drafts reduce work after verification. The benchmark provides one point of comparison; it does not report those firm-specific outcomes.
Buyers will also need details on data access, retention, processing locations and contractual safeguards before connecting the model to client information or internal legal systems. Z.ai’s documentation recommends agent connections to legal and company-data services, but the supplied material does not describe the security controls or terms for those connections. Further published details on the model’s comparison set, weight availability, deployment arrangements and performance beyond the holdout would help clarify its practical standing.
For now, the confirmed development is a strong relative leaderboard placement paired with a low all-pass score under a demanding grading rule. Whether GLM Legal proves useful in practice depends on the accuracy and completeness of its work products, the cost of human verification, and the protections available for client data—questions the benchmark ranking alone cannot settle.
Key Questions
What does GLM-5.3-FlashX-Legal’s 19.17% score mean?
It is the share of benchmark tasks on which the model passed every required rubric criterion, according to the Vals AI leaderboard. A task with even one failed criterion does not count as complete.
How can its criterion score be 92.5%?
The 92.5% figure applies to individual criteria, not complete tasks. A model can pass most criteria within many tasks while missing at least one requirement in each of those tasks, causing those tasks to fail the all-pass measure.
What does seventh place establish?
It establishes GLM Legal’s position on the archived Vals AI leaderboard among 81 listed models under that benchmark version. It does not by itself prove the model is safe, accurate across legal practice, or ready for unsupervised use.
Did Z.ai’s model improve on its base model?
The archived leaderboard lists GLM Legal at 19.17% and the base GLM-5.3-Flash model at 6.67%, a difference of 12.50 percentage points. Z.ai’s claim of a 12.5-point improvement matches those figures.
Is the model open source?
The supplied source material says Z.ai claims state-of-the-art performance among open-source models, but it does not name the comparison set and reports no published weights for the legal model. The available information does not settle how the model should be classified or whether its weights will be released.
Source: ThorstenMeyerAI.com
