Claude Legal Prompt Shock, LegalOn GPT 5.4 Review, Legal Innovators +

We start this week’s Wrap with a real shocker! A chap on Twitter named Nav Toor, (see images), published 12 quite detailed legal prompts for Claude, but with the addition of famous law firm brands such as Latham & Watkins, Kirkland & Ellis, and Wachtell Lipton. It follows the recent brouhaha after Anthropic released a legal plug-in, which is basically just a very complex prompt.

After the legal world’s recent minor destabilisation, this new revelation unsurprisingly triggered an immediate backlash from the social media legal world, with comments along the lines of ‘thank you for doing this, I will now get to spend months advising my clients on the mess your Claude prompt contracts have made’.

Which leads to the key question: are these contracts any good, and did adding famous law firm names make any difference? It’s possible that Claude simply read ‘Wachtell’ and thought ‘OK, this means do the contract in the style of any large commercial law firm’, and then proceeded to make a document based on whatever it could find publicly that fitted. Equally it perhaps did look for anything in the public record that was associated with that firm, and then perhaps tweaked its output accordingly.

(Although, there is also something a bit absurd about the idea of using an elite New York firm such as Wachtell to do a single independent contractor agreement.)

OK, so, maybe don’t do this for your clients….! But, as an experiment AL ran one of them through OpenAI’s latest GPT 5.4 and got what looked like a very comprehensive contract that met the general goals of the prompt. But….as with all legal work, details matter. So, did it miss anything? Hmmm? Was it a ‘good’ contract, or as Admiral Ackbar might have said, was it a trap?

Here is the prompt if you want to experiment in a safe space.

And here are some more – here is the link to have a look at all 12 contract prompts.

P.S. as mentioned, a prompt alone is not usually a replacement for the input of a good lawyer – at least for now. That said, see below….

—

LegalOn has helped the legal world with an analysis of GPT 5.4 vs the older 5.2 when applied to contracts, and without any additional tinkering, i.e. these are vanilla ‘out of the box’ results. Here is what they found:

• Overall accuracy: 79.4% vs. 73.9%; a meaningful +5.5pp improvement
• GPT-5.4 cuts total errors by 21%. The gain is broad-based: every contract type improved, and 16 of 26 guidelines improved
• Both precision and recall improved, meaning fewer false alarms and fewer missed violations simultaneously
• Total errors reduced by 21%, from 129 down to 102
• Improvement is consistent across all five contract types, with the largest gains on NDAs (+10pp) and MSAs (+8pp)

Is this a big deal? For AL, what this shows is that slowly, but steadily the general LLMs are getting better at legal tasks. A score of 79.4% may not sound that good, but the next version may be 84%, and the one after that maybe 89%, then 93%, and so on. At this rate we could be hitting extremely high general LLM accuracy for legal tasks in a few years. All of that then feeds into legal tech tools and their performance.

The more accurate legal AI becomes, the less it needs to be checked and overseen.

The less it needs to be checked and overseen, the less it needs junior lawyers to manage its use and it only then needs senior lawyers to really give the final approval.

And, as noted, unless the models plateau – which is still possible – then accuracy is on a short-to-medium term upward trajectory to higher levels. So, we can see where this is going. (Note: see previous AL piece about what we learned at Legal Week, especially the bit about the inevitability of change.)

The full LegalOn report is here. Thanks to CEO, Daniel Lewis, and team as always.

—

Now some more news that you may have missed:

• The Financial Times has chosen Wordsmith as its enterprise legal AI platform, deploying it across its legal, compliance, and company secretarial function to ‘drive efficiency and give the business faster access to legal support’. The FT’s legal team chose Wordsmith not as a point solution for a single task, but as the platform to handle the full breadth of their work, from contract review and drafting to compliance activities and legal support designed to extend across the business, they added.
• Kevin Keller has joined Sandstone’s board of advisors. He is the Senior Vice President and General Counsel at Forward Networks, with over 27 years of experience in law and technology. Kevin has led $100M+ acquisitions, negotiated over $1 billion in patent licenses, and built legal and operational frameworks for complex, regulated products. His guidance will be instrumental as Sandstone continues to enable the first generation of AI-native legal departments, they said.
• Harvey has forged new partnership deals, this time with the Dallas Mavericks and American Airlines Center. And also, just announced today, a deal with Fulham FC in the UK, that will see Harvey become the Club’s Official Legal AI Partner. Earlier this month, Harvey and The LegalTech Fund announced that they will partner to ‘invest in legal technology startups across the ecosystem, backing founders who make the legal industry more accessible, intelligent, efficient, and impactful’.
• International law firm Herbert Smith Freehills Kramer is adopting Legora as ‘its firmwide general-purpose AI platform’, they have announced.
• Centari, the deal intelligence platform, has announced a major expansion of its platform with two new products: Views and Intelligence. Centari’s patent-pending Deal Reasoning Engine simulates the relational reasoning a transactional lawyer performs on a complex agreement, tracing defined terms, mapping conditional logic, and reconciling interdependent provisions across multiple documents to create high-fidelity, citation-backed data. The company’s Applied Legal Research team partners with every customer to map the platform to their specific deal types and help them extract strategic value from their data over time, they said.
• AltaClaro and Verbit’s DepoSim – an AI-driven deposition simulator – has been chosen by US law firm Taft to help with training its lawyers.

–

OK, and onto the Legal Innovators Paris and California conferences, both this June!

A Legal Tech Conference For All of Europe

Legal Innovators Europe – Paris – June 24 and 25.

There will be more news about the conference and key speakers as we get closer to June.

Look forward to seeing you there!

Richard Tromans, Founder, Artificial Lawyer and Legal Innovators conference Chair.

Note: the conferences are organised by Cosmonauts – please contact them with any queries. 

If you would like to be a speaker at Legal Innovators Europe, especially if you are at a law firm or inhouse legal team in Europe – whether based in France, Belgium, Spain or Germany, or beyond…..then please contact Phoebe at Cosmonauts:  phoebe@cosmonauts.biz

Note: if you are a legal tech company, please contact Robins: robins@cosmonauts.biz or Anjana anjana@cosmonauts.biz

–

And if you’re in the US and looking for the next major event to join after Legal Week, then see you in California this June!

Legal Innovators California, the landmark West Coast legal tech event, will take place on June 10 and 11, in the heart of the Bay Area, the home to many of the world’s leading AI businesses – and plenty of legal tech pioneers as well! More information and tickets here.

—

And finally the video section, with a walk through of TransLegal:

–

Have a great weekend folks and enjoy the Spring weather if you have it!

P.S. AL is in Paris on Tues and Wed, but will keep up to date as best as possible.

Richard Tromans, Founder, Artificial Lawyer