🔍 Read the full analysis: Z.ai’s Legal AI Passes About 1 In 5 Tasks—Here’s Why It Still Ranks Near The Top on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal achieved a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models in the archived results. It passed 92.5% of individual criteria, but the benchmark counts a task as complete only when every required criterion passes. Z.ai’s claimed 12.5-percentage-point gain over its base model matches the listed scores; the company’s claim of state-of-the-art performance among open-source models is not fully substantiated by a named comparison set.
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on the Vals AI leaderboard for Harvey’s Legal Agent Benchmark, placing seventh among 81 listed models in the archived results. The score means the model satisfied every rubric requirement on roughly one in five benchmark tasks—not that it got only one in five individual legal details right.
The distinction comes from Harvey’s all-pass grading rule: a task counts as complete only if every required criterion passes. Vals AI’s archived results show GLM Legal passing 92.5% of individual criteria, while failing to achieve a fully passing result on most complete tasks. A missed deadline or citation can therefore make an otherwise strong work product a failed task.
Vals lists the benchmark result at $0.15 per test. The same leaderboard gives GLM 5.3 Flash, the base model identified in the source material, a 6.67% task pass rate. The difference is 12.50 percentage points, matching Z.ai’s claim of a 12.5-point improvement. The leaderboard’s archived figures are reported to two decimal places, while Z.ai presents the result rounded to 19.2%.
Z.ai describes the model as designed for legal research, contract review, drafting and document understanding. Its documentation says it underwent legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal workflows. Z.ai recommends using it as an agent connected to services for laws, cases and company data. Its pricing page lists $0.37 per million input tokens and $1.25 per million output tokens.
Why Task-Level Accuracy Matters
The result illustrates a gap between getting most criteria right and completing a legal task reliably. In practice, a work product may contain many correct details yet remain unusable if it misses a key risk, gives an incorrect deadline or cites the wrong authority. The benchmark’s all-or-nothing score is designed to expose that weakness, though it can make a model’s overall performance look much lower than its criterion-level accuracy.
For law firms and legal departments, the figures support a measured assessment rather than a simple ranking-based decision. GLM Legal’s high individual-criterion rate could make it useful for producing drafts or assisting with review, but the task pass rate does not establish that its work can be relied on without checking. Procurement decisions also turn on issues not settled by a benchmark score, including client-data handling, deployment terms, citation verification and human review.
Cost is another relevant part of the leaderboard comparison. Vals lists GLM Legal at $0.15 per test, below the listed prices for the higher-ranked models shown in the source material. That does not by itself establish lower total operating costs: the source does not provide comparable information on token use, integration, review time or the cost of correcting errors.
Popular right now
Dreame H12 Pro Wet and Dry Vacuum Mop
Dyson Ball Animal 3 Upright Vacuum
KitchenAid 5.5-Qt Bowl-Lift Stand Mixer, Porcelain White (KSM55SXXX)
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The public benchmark contains 1,250 tasks across 24 practice areas, with more than 75,000 expert-written rubric criteria. Tasks ask an agent to work from a client matter’s materials and produce a work product for review. The criteria can cover facts, conclusions, citations, recommendations, deadlines, amounts and formatting.
Vals AI’s public leaderboard uses a private holdout set of 120 tasks, according to the source material. Internet access is disabled during testing. Two language-model judges grade submissions, and the score is the average of their task pass rates. Vals reports its criterion pass rate separately; for the archived version cited, a criterion counted only when both judges passed it.
The leaderboard’s wider results show that end-to-end task completion remains difficult across the tested models. The source describes an earlier Harvey baseline in which the best overall result among the frontier models tested was 7.1%, while noting that it came from a different run and is not directly comparable. The archived leaderboard identifies GLM Legal as seventh of 81 models; other entries listed above it include Muse Spark 1.2 at 25.42% and Gemini 4 Argon at 19.58%.
“A task is marked complete only if every criterion passes.”
— Harvey, describing the benchmark’s scoring rule
Limits of the Published Results
Z.ai calls the result “state-of-the-art performance among open-source models”, but the source material says the company does not name the comparison set. It also reports no published weights for the legal model, leaving open what “open-source” means for this specific system. The leaderboard establishes a rank within its listed evaluation; it does not resolve the scope of Z.ai’s broader claim.
The score is tied to a particular benchmark version, a 120-task private holdout and two AI judges. The source material does not provide task-by-task results, confidence intervals or a breakdown by practice area, so it is not possible to tell from the headline metrics where the model performs best or how sensitive the result is to grading variation.
The source author also reports a four-task spot check in which the legal model cited rulings the author could not find. The author says OpenAI and Claude models in that small check did not invent rulings, while two Claude models cited a real ruling for the opposite of its holding. This is an anecdotal check, not a controlled benchmark, and does not establish comparative error rates. Whether every citation in the benchmark outputs was verified independently is not stated in the supplied material.
What Firms Should Verify Before Use
The next practical step for organizations evaluating the model is to test it against their own legal workflows and review requirements. That means checking the source of each authority, verifying deadlines and factual assertions, and measuring how much lawyer time is needed to revise an output—not relying on the leaderboard rank alone. Z.ai’s documentation recommends connecting the agent to legal and company-data services, but the source material does not specify which providers or configurations were used for the leaderboard result.
Firms will also need clear answers about where client data is processed, how it is retained, and which contractual or regulatory protections apply. The supplied source says Z.ai’s pages describe outputs as drafts for legal professionals and do not carry a “not legal advice” notice. That description is not a substitute for a firm’s own review controls or for terms governing sensitive client information.
No additional benchmark release date or follow-up evaluation is provided in the source material. Until fuller methodology and model-specific deployment details are available, the reported ranking should be read as a result on one defined benchmark—not as proof that the system can independently complete legal work safely.
Key Questions
What does GLM Legal’s 19.17% score measure?
It is the share of benchmark tasks on which the model met every required criterion, as calculated on Vals AI’s archived leaderboard for Harvey’s Legal Agent Benchmark. It is not the percentage of individual rubric criteria passed.
How can the model pass 92.5% of criteria but only 19.17% of tasks?
A task fails if even one required criterion fails. A model can therefore meet most individual requirements across the evaluation while still missing at least one requirement in many complete tasks.
Did GLM Legal improve on Z.ai’s base model?
The archived leaderboard lists GLM Legal at 19.17% and GLM 5.3 Flash at 6.67%. That is a 12.50-percentage-point difference, matching Z.ai’s stated improvement for the comparison shown.
Does the result prove the model is ready to handle legal work independently?
No. The score measures performance on a particular benchmark and does not establish that outputs are safe to use without lawyer review. The supplied material also raises unresolved questions about citation verification, data handling and deployment terms.
Source: ThorstenMeyerAI.com
