🔍 Read the full analysis: We Put 42 AI Setups To A 15-Question Legal Test on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A test by ThorstenMeyerAI.com evaluated 42 AI setups on 15 legal questions involving German, US and EU law. The author reports low scores for some systems, including a legal-focused model, and says citation checks found both genuine references and references that were invented or misrepresented. The test is a limited, developer-run spot check, not a definitive ranking of legal AI.
Thorsten Meyer published results on November 4 from a test of 42 AI setups answering 15 legal questions across German, US and EU law, reporting that some systems missed key details and that many citations were wrong or unsupported. The results matter because legal mistakes can affect deadlines and claims, but the test is a developer-run spot check, not an independent or definitive measure of how AI performs in legal practice.
Meyer said the highest-scoring setup was Claude Opus 5.5, with 79 points, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the top scorer fully met every criterion on only eight of the ten publicly available tasks, according to the report. Meyer also said that increasing a model’s thinking effort did not improve performance in the tests.
The legal-focused GLM-5.3 FlashX Legal, which its maker Z.ai describes as optimized for professional legal workflows, scored 8 out of 100 and received no points on the hidden tasks. Meyer reported that it fully solved none of the 15 questions and produced no answer on two. Its base model scored 3. The report says the legal model’s citation review found 21 real citations, 25 misattributed citations and one invented citation.
For Mistral Large 4, Meyer reported a score of 10. He said it got about half the substance right but lost much of its score over incorrect citations. The test’s citation audit across published runs identified 2,644 citations as real and 804 as invented or not supporting the claims attached to them. Those totals describe the test’s collected answers; they are not a general error rate for legal AI.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal Errors Can Affect Deadlines
The test highlights a distinction between producing a plausible legal explanation and getting the law, supporting authority and dates right. In one example, Meyer describes a German civil appeal with judgments served on the same day in Munich and Berlin. A public holiday in Bavaria changes the Munich deadline by a day, while the Berlin deadline remains unchanged. He reports that one model gave Berlin the later date and another placed the deadline for appeal grounds a month too late.
Such mistakes can have consequences beyond a low benchmark score. Meyer notes that a late appeal in German civil procedure can be rejected as inadmissible. For readers, the practical point is not that every AI answer is wrong, but that a chatbot response should not be treated as a reliable source for a filing date, legal claim or advice involving an authority. Checking citations against primary sources and consulting a lawyer where a deadline or material financial interest is at stake are prudent safeguards.
Popular right now
Ninja Foodi 10-Qt DualZone Air Fryer, 6-in-1, Black (DZ302)
TCL 65 Inch QM8L Series SQD-Mini LED QLED 4K Smart Google TV
Philips Café Aromis Automatic Espresso Machine with LatteGo Pro Milk System
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested Models
Lexassay asked each setup the same 15 questions: ten public tasks and five hidden ones. The questions cover German, US and EU law, including deadline calculations, recent legal changes, leading court decisions and questions containing false premises. The hidden tasks’ scores are published, but their questions are not, Meyer said.
Models answered without internet access or external tools. That means the test measured responses from the models themselves rather than the results of web search or document retrieval. Each answer was graded against a prepared answer with three to six weighted criteria, and citations were checked against official databases. The scoring rules deduct points for citations that are invented or misattributed, as well as for an answer lacking a legal-advice disclaimer or referral to a lawyer.
Meyer explicitly calls the exercise a spot check, not a strict benchmark: each model-task combination received one answer, so small score differences may reflect variation rather than a durable ranking. He also discloses that Thorsten Meyer AI runs Lexassay and that several Claude agents graded answers. Because a Claude model ranked first, he says, the setup presents a conflict of interest. He says the method page and published answers, gold answers and citation checks allow readers to inspect the results.
“It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.”
— Thorsten Meyer, test author
Limits of the Reported Scores
The results do not establish how the systems would perform with internet search, legal databases, supplied case files or a lawyer reviewing their work. The test deliberately excluded tools, so its findings apply to that setup rather than every way people might use AI for legal research. It also does not establish that the tested answers represent a model’s performance across repeated attempts: Meyer says there was one answer per task and setup.
The source material does not provide enough detail to independently reproduce every score here, and the author’s disclosure identifies a potential conflict because his organization runs the test and Claude agents graded answers. Meyer says the underlying materials and scoring method are public, but no independent audit is described. The citation totals also need to be understood as counts within the published runs; the report does not provide a broader baseline from which to infer an industry-wide error rate.
One comparison in the report points in a different direction: on Vals AI’s legal-agent benchmark, which supplies documents to models, Mistral Large 4 ranked eighth, ahead of Claude Opus 5.5 at 21st and GPT-6 Astra at 24th. The tests have different methods and should not be treated as directly comparable. The source also notes that GLM-5.3 FlashX Legal ranked seventh among 81 on another measure, but the supplied material does not explain that measure in enough detail to reconcile it with Lexassay’s score.
Check the Answers and Method
Meyer says the test’s method page, public answers, gold answers and citation checks are available for readers to review. Those materials are the next place to examine how the criteria were applied and whether particular citations support the model’s statements. The five hidden questions are not published, according to the report, to reduce the chance that future systems can memorize their answers.
The source does not announce a date for another test or an independent replication. Until further results are available, the scores should be treated as a dated snapshot of the tested models and settings, rather than a permanent ranking. Anyone using a chatbot for a real legal issue should confirm the relevant law and deadlines with authoritative sources and seek qualified legal advice when the stakes are significant.
Key Questions
What did the legal AI test evaluate?
It evaluated 42 AI setups on 15 questions involving German, US and EU law. The models answered without internet access or external tools.
Which system scored highest?
Thorsten Meyer reported that Claude Opus 5.5 scored 79, ahead of GPT-6 Astra at 72 and GPT-6.1 Sol at 71. The author disclosed that Claude agents graded the answers and identified Claude’s top ranking as a conflict of interest.
How did the legal-focused model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 in the reported test and received no points on the hidden tasks. Meyer said it fully solved none of the 15 questions and gave no answer to two.
Does the test show that AI is reliable for legal advice?
No. The author describes it as a spot check, with one response per task and setup. It does not establish performance with legal research tools, repeated attempts or lawyer review.
Should someone rely on a chatbot for a legal deadline?
The report warns readers not to take deadlines from a chatbot. Confirm dates and legal authorities through reliable sources, and consult a lawyer when a deadline or other significant legal consequence is involved.
Source: ThorstenMeyerAI.com
