🔍 Read the full analysis: 42 AI Setups Took The Same Legal Quiz—One Scored 8/100 on ThorstenMeyerAI.com
Prime made for students and young adults
- Fast, free delivery for dorm and study essentials
- Prime Video and Amazon Music included
- Member-only deals
TL;DR
Lexassay, a legal AI test run by Thorsten Meyer AI, assessed 42 model setups on 15 questions covering German, US and EU law. Its operator reports that GLM-5.3 FlashX Legal scored 8/100 and fully solved none of the tasks, while disclosing that Claude agents graded the answers and a Claude model ranked first. The results are a limited spot check, not a definitive ranking of legal AI tools.
Thorsten Meyer AI reported November 4 that its Lexassay test scored 42 AI setups on 15 legal questions, with law-focused GLM-5.3 FlashX Legal scoring 8 out of 100. The test operator says the model fully solved none of the tasks; the results draw attention to the risk of relying on AI for legal answers, while the operator’s disclosed role in the test and its grading method warrant caution in interpreting the scores.
The test covered questions in German, US and EU law, including deadline calculations, legal changes, court decisions and questions with false assumptions. According to Meyer’s account, each system answered without internet access or other tools. Lexassay includes 10 published and five hidden tasks; the site publishes only scores for the hidden tasks. Different effort settings were tested separately, so the 42 setups do not necessarily represent 42 distinct underlying models.
Meyer reports that GLM-5.3 FlashX Legal received 0 points on the hidden set and gave no answer on two German questions after using its full stated budget of 64,000 tokens. Its citation review found 21 citations that were real, 25 that were misattributed and one invented. Its base model scored 3/100. The source says that citation errors lose points under Lexassay’s rules, but these findings are the test operator’s results, not an independent audit.
The same account reports Claude Opus 5.5 at 79, GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the top-scoring setup met every criterion on only eight of the 10 public tasks, according to Meyer. Mistral Large 4 scored 10/100 in this test, with the author attributing much of its lost score to citations that were wrong or did not support its claims. The results should not be read as a direct measure of performance when a system can search or use legal documents.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Legal Errors Can Cost a Case
The practical issue is not simply whether a chatbot can discuss law fluently. A wrong court citation or procedural deadline can change what a user does next. Meyer’s article illustrates this with an appeal scenario: under its stated facts, the deadline differs between Munich and Berlin because Corpus Christi is a public holiday in Bavaria but not in Berlin. The example shows why location and procedural details matter; it is not a substitute for checking an actual case’s dates and rules with a lawyer.
The reported citation counts also point to a problem for readers trying to verify AI-generated answers: a case or law can exist while failing to support the proposition attached to it. Lexassay says each invented or misattributed citation reduces a task’s score, and that answers lacking a legal-advice disclaimer or referral lose 5 percent. Those are choices in this test’s scoring method, not universal standards for measuring legal AI.
For consumers, the results support treating general-purpose or specialized chatbots as starting points, not legal authorities. The article advises checking cited material against official sources and consulting a lawyer when a deadline, financial claim or communication with an authority is involved. The scores alone do not establish how any product would perform with a lawyer supervising it or with access to relevant documents.
Products Worth Considering
Apple Watch SE 3 GPS Smartwatch, 40mm
Apple MacBook Air 15-inch with M5 chip, 16GB Memory, 512GB SSD, and Liquid Retina Display
LG UltraGear 27GX704A-B 27-inch QHD OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Answers
Lexassay compares model answers with a prepared gold answer containing three to six weighted criteria for each task. According to its method description, citations are checked against official databases and categorized as real, misattributed, invented or not findable. Each model-task answer is graded once, making the test a spot check rather than a repeatable broad benchmark; small score differences may reflect variation in an individual answer.
The source article discloses that Thorsten Meyer AI runs Lexassay and that several Claude agents graded the answers. A Claude model ranks first in the reported results. Meyer says the site publishes the public answers, gold answers and citation checks so readers can inspect the grading, while hidden-task scores are not accompanied by the underlying questions or answers. The disclosure is material when evaluating both the methodology and the ranking.
Meyer also notes that other evaluations can produce different orderings. The article says that on Vals AI’s legal-agent benchmark, where models work with supplied documents, Mistral Large 4 ranks eighth, ahead of Claude Opus 5.5 in 21st and GPT-6 Astra in 24th. That comparison uses a different benchmark setup, so it does not directly confirm or contradict Lexassay’s scores.
“It is a spot check, not a benchmark in the strict sense: one answer per model and task, so a few points of difference are noise.”
— Thorsten Meyer AI, describing Lexassay’s limitation
Limits of the Reported Scores
The article is the account of the organization that operates the test, and the grading conflict is disclosed: Claude agents evaluated answers and a Claude model ranked first. Publishing answer keys and citation checks may help readers inspect the process, but the source material does not provide an independent replication or outside audit of the grading.
The test uses one answer per setup and task, and models answered without web access or tools. It therefore remains unclear how the systems would perform across repeated runs, with current legal research tools, with supplied case documents or under professional supervision. The article gives some task-level examples but does not establish that the results predict the performance of every version or configuration of these products.
The source also cuts off while beginning a qualification about Z.ai’s model ranking “7th of 81”; it does not identify the ranking’s full scope or explain how it relates to the reported 8/100 score. No response from Z.ai or the other companies to the Lexassay findings is included in the supplied material.
What Readers Should Check Next
Readers can review Lexassay’s published method, public model answers, gold answers and citation checks, as described by Meyer. The source does not give a date for another test run or say whether the benchmark will be expanded. Further results would be more informative if they included independent grading or repeated runs, alongside clear details on model versions, configurations and whether tools were enabled.
For anyone facing a real legal problem, the immediate next step is not to rely on these rankings to choose a chatbot for a deadline or claim. Verify laws and judgments in official sources and contact a qualified lawyer when a filing date, financial decision or authority is involved. The test is a comparison of model answers, not legal advice or a determination of an individual case.
Key Questions
What did GLM-5.3 FlashX Legal score?
Thorsten Meyer AI reports that the model scored 8 out of 100 on Lexassay and 0 on the hidden set. The article says it fully solved none of the 15 tasks and produced no answer on two.
How were the AI setups tested?
Lexassay gave each setup the same 15 legal questions on German, US and EU law, with no internet access or tools. Answers were compared with weighted gold-answer criteria, and citations were checked against official databases, according to the test operator.
Does this prove that legal AI systems are unreliable?
No. The results describe a single test with one answer per model and task, run without tools. They do not establish how the systems perform in every setting, with legal documents or under professional supervision.
Who graded the answers?
Thorsten Meyer AI says several Claude agents graded them, and a Claude model ranked first. The test operator disclosed that conflict of interest; the supplied material does not report an independent audit.
Should someone use a chatbot to calculate a legal deadline?
The source advises readers to check cited law and rulings and not to take a deadline from a chatbot. For a real deadline, financial claim or matter involving an authority, consult a lawyer.
Source: ThorstenMeyerAI.com
