🔍 Read the full analysis: We Asked The Law-Built Model 15 Questions. It Scored 8/100 on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A 15-question test run by Thorsten Meyer AI gave Z.ai’s GLM-5.3 FlashX Legal a score of 8 out of 100, with zero on five hidden questions and no answer on two tasks. The test author says the results are a spot check, not a definitive model ranking, and discloses that his organization runs the test and Claude models grade answers.
GLM-5.3 FlashX Legal, a model Z.ai describes as optimized for professional legal workflows, scored 8 out of 100 in a 15-question test of legal tasks published November 4 by Thorsten Meyer AI. The result raises questions about relying on legal-focused chatbots for answers involving deadlines, court rulings and citations, though the test’s author says it is a spot check, not a definitive benchmark.
According to the published results, the model scored zero on the five hidden questions and gave no answer on two of the 15 tasks. In its citation review, the test found 21 citations were real, 25 were misattributed—meaning the cited source did not support the model’s claim—and one was invented. The model used its full 64,000-token reasoning budget on two German-law questions but produced no answer.
The test included questions on German, U.S. and European Union law, covering matters such as calculating appeal deadlines, recent legal changes, court decisions and questions containing false assumptions. Models answered without internet access or tools. Meyer says answers were assessed against model answers with weighted criteria, and citations were checked against official databases. Each invented or misattributed citation reduced the task score.
The results also describe errors with potential practical consequences. On a German data-scraping question, the model said a four-figure compensation amount was conceivable; Meyer’s account says the German Federal Court of Justice’s reference point for mere loss of control over personal data is around €100. In another answer, the model treated a challenge to the EU–U.S. Data Privacy Framework as pending, although the EU General Court had dismissed it on September 3, 2025.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Errors Can Affect Legal Decisions
The score matters because a confident but inaccurate legal answer can lead a user to misunderstand a claim, cite a ruling incorrectly or miss a deadline. The article’s appeal-deadline example shows why even a one-day error can matter: under the scenario described, a German civil appeal deadline differs between Munich and Berlin because a public holiday applies in Bavaria but not in Berlin. Meyer says a late appeal in German civil procedure is rejected as inadmissible.
The test does not establish how often these failures occur in everyday legal use. It does show that a model’s legal branding and extensive reasoning are not, by themselves, evidence that its answers or citations are reliable. The published advice is to treat AI output as a starting point, verify cited law and rulings in their original sources, and consult a lawyer when a deadline, money or an authority is involved.
Products Worth Considering
EcoFlow Delta 3 Portable Power Station – 1024 Wh, 1800 W
Sony WH-1000XM4C (M4 2nd Gen) Wireless Noise-Cancelling Headphones, Black
Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds, Black
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested the Models
Lexassay asks each model the same 15 legal questions: 10 are public and five are hidden, with only the scores disclosed for the hidden tasks. The hidden questions are intended to make it harder for later models to reproduce answers found online. The test covers three legal systems and includes questions designed to expose answers based on false premises.
Meyer says each answer is graded against a prepared answer using three to six weighted criteria. Citations are classified as real, misattributed, invented or not findable. Answers also lose 5 percent if they lack a “not legal advice” note or a recommendation to consult a lawyer. The author warns that there is only one answer per model per task, so small score differences may be noise.
The published comparisons put Claude Opus 5.5 at 79, GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Meyer says even the leading model fully met every criterion on only eight of 10 public tasks. Mistral Large 4 scored 10 in this test, despite its maker describing it as state-of-the-art among open models for enterprise work, including law. A separate Vals AI legal-agent benchmark ranks Mistral eighth, ahead of Claude Opus 5.5 and GPT-6 Astra; that benchmark supplies documents to models, unlike Lexassay’s no-tools setup.
“Optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution.”
— Z.ai, as quoted in its developer documentation
Limits of the Published Results
The findings come from a small test and do not show how the model would perform with legal documents, web search or other tools. The five hidden questions are scored but not publicly disclosed, limiting independent review of those answers. The source says public answers, gold answers and citation checks are available, but the test remains a limited sample rather than evidence of performance across all legal work.
There is also a disclosed conflict of interest: Thorsten Meyer AI runs Lexassay, and Claude agents grade the answers, while a Claude model ranks first in the reported results. Meyer says the conflict is stated on the method page and that the underlying materials are published for checking. The source also begins a qualification about Z.ai’s ranking in another comparison but cuts off before giving the full explanation, so that point cannot be assessed from the supplied material.
Check Sources Before Acting
No further test release or response from Z.ai is specified in the material. Readers can review the published answers, gold answers, citation checks and Lexassay method to examine how the scores were assigned. Any later comparison should be read with attention to whether models can use supplied documents or tools, because those conditions differ from this test.
For anyone facing a real legal deadline or dispute, the immediate practical step is not to rely on a chatbot’s answer: check the relevant official source and seek advice from a qualified lawyer. The test author specifically warns that the article is not legal advice.
Key Questions
What score did GLM-5.3 FlashX Legal receive?
It scored 8 out of 100 across the 15 Lexassay questions, with zero on the five hidden tasks, according to the published results.
What did the test find about the model’s citations?
The citation review counted 21 real citations, 25 misattributed citations and one invented citation. A misattributed citation refers to a real source that does not support the model’s claim.
Does this prove that legal AI models are unreliable in every situation?
No. The author describes Lexassay as a spot check with one answer per model and task, conducted without internet access or tools. It does not establish performance across all legal tasks or tool-enabled workflows.
Who ran and graded the test?
Thorsten Meyer AI runs Lexassay, and the published account says Claude agents graded answers. The author discloses that conflict, including that a Claude model ranks first, and says the answers and checks are published for review.
Should someone use a chatbot to calculate a legal deadline?
The test author says not to take a deadline from a chatbot. Verify the applicable law and dates through official sources and consult a lawyer when a deadline or legal claim is at stake.
Source: ThorstenMeyerAI.com
