How will Claude’s watermarks work and how will they impact the legal world? Artificial Lawyer explores.
Anthropic – and all other LLM makers operating within the EU – will have to provide some form of identification that tells a user and authorities whether an output is AI-generated. This is because of the EU AI Act. The rival to OpenAI has been both quick to set out its stall and has been very transparent.
First, how they work. Essentially what Claude will do is insert certain words where there is a choice of options, in a way that it matches a key designed to reveal whether text is AI-manufactured or not. I.e. there is no actual ‘watermark’ as such, at least in the document, rather one could say the signal is ‘camouflaged’ as part of the normal words on the page.
Claude gives a great explanation:
‘Large language models like Claude work by generating one word at a time. Each time the model decides on the next word, it chooses among a list of potential candidates, ultimately selecting the most sensible or likely based on the preceding text.
‘Take the sentence “The weather today was cold and…”. The next word is very unlikely to be “sugary.” But it is quite likely to be “overcast” or “grey.”
‘Under most circumstances, it doesn’t matter much to the reader which of these latter two words the model ultimately chooses, the meaning of the sentence is largely the same either way. In cases like this, the choice is settled by a random number….
‘Watermarking uses low-stakes choices…., which occur many times over a piece of generated text, to leave a pattern in Claude’s responses. That pattern is undetectable to the reader, but is detectable to anyone who has a key that encodes it.
‘When watermarking is used, choices are still made at random, but the source of the randomness is different. Instead of using an arbitrary random number generator to pick the next word, watermarking uses the key and a few words that come before to settle what word the model should pick.
‘That is, the words that Claude picks are still random, but now, one can check the sequence of words and see if it’s consistent with the choices Claude would make if it was using the key. If it is, one can assign a probability that the text was generated by Claude.’
It truly is ingenious and builds upon the SynthID-Text approach published by Google DeepMind in a paper in 2024. They added that:
- ‘Nothing is added to the text and there are no hidden characters;
- Watermarking doesn’t require extra tokens, and will not be more expensive;
- Watermarking carries no identifying information and can’t be traced to a specific person, organization, or chat.’
They also noted that: ‘We will soon be offering a watermark detection API. We’re in the process of working out the details of its implementation.’
Impact on Lawyers
The good news is that this approach doesn’t reveal anything it should not. It simply changes the text. It also doesn’t add special identifiers as to who made the document, or part of the document, with AI.
The flipside is that:
- Anything a law firm produces that contains AI outputs will show up as such. In most cases the client or court will not mind, in fact they may expect this now. And some clients are specifically asking for AI use. The challenge comes with clients who are refusing to allow you to use AI on their matters, or perhaps a court where a judge is especially biased against AI use. So, even if the documents submitted are totally correct, if someone wished to see if AI had been in play, then it will be revealed.
- If clients are demanding price cuts on the basis that AI outputs are easier to produce, then this is also going to be transparent for those who wish to look.
- Also, the watermarks – as far as AL can see – will get transferred around with the text, because they are part of the text. So, a large and complex contract could have many pages with no AI content, and then a few clauses that include AI. Also, it would seem that once the watermarks get into the system then they stay there. So, future contracts may draw on older text templates that have a watermark. In fact, as you may be using multiple LLMs – if watermarks have appeared for each of them – then a document could have multiple watermarks mixed up in them.
- As noted, if you are happy to be transparent about your AI use, then none of this is an issue. But, if for whatever reason a lawyer doesn’t want to have it known that chunks of their work are AI-derived, then it will be very hard to avoid.
- One other point is that legal texts are by nature built on precision. AL does not know if the system of word choice to make a watermark will make any difference here. Likely not. But, AL has not seen any empirical study on that. However, Claude notes that: ‘Watermarking is sparser on factual passages where there are fewer choices that can be made without decreasing the accuracy of the text.’ And of course, accuracy is essential in legal work.
Another important question is whether the change of words alters responsibility for any legal document. Here is what they say:
‘Does this change who owns a given output, or who is legally responsible for it?
‘No. A watermark only helps test whether Claude might have produced or processed the content. It doesn’t say anything about ownership or authorship, and doesn’t change a user’s rights under our terms. We only apply the watermark when Claude was involved in processing the content or file.’
And of course, as the EU AI Act refers to the European Union, whatever LLMs you use – whether from the giants, or from open weights models from China, or even a very small model that is largely DIY, those watermarks will have to be there if the outputs are used in Europe. This may especially matter to those law firms using open weight models that have not yet incorporated the above type of system.
(See more here about the Act.)
And finally, it has to be noted that if an LLM has made an error / hallucination, that this is not necessarily changed by the watermark. A watermark is not a guarantee of anything in terms of accuracy. It is simply a ‘statement of the production method’. Some may see an AI watermark as a reason to doubt a piece of work, others may see it as simply how work is made these days. And some may prefer to see that watermark as proof of good work! Views on AI use are rapidly evolving, that’s for sure.
Conclusion
Overall, watermarks appear to be benign. It’s the LLM makers who have to implement them, not the law firms, nor the legal tech companies. Although, presumably they must not interfere with their use in the EU, or stop them from appearing.
The key challenge – if there is one – would appear to AL to be focused on whether a lawyer does not wish to let a court or client know that a piece of work was partially, or fully, produced by AI; or also where ‘artefacts’ in an old contract are AI-produced, but are then added to a new contract.
In most cases they will not mind, but certain situations may be more delicate.
Either way, at least in the EU, having AI outputs getting passed off as 100% human is going to be a lot harder. Lawyers will need to adapt, although in most cases there will be little they can do other than be ready to answer any questions about watermarks their clients, or the courts, may have.
More here from Claude.
P.S. Naturally, this is not a legal opinion. Artificial Lawyer is not a lawyer 🙂 and is a legal tech media site. If you do need specialist input on watermarks, this site would recommend speaking to a real lawyer.
–
Two Major Legal Innovators Conferences this November
Come and join us in New York and London this November at Legal Innovators!
Legal Innovators UK – London, Nov 4 and 5
And, then Legal Innovators New York – Nov 17 and 18.
After another fantastic Legal Innovators California, where we had speakers from OpenAI, Y Combinator, Google, Meta, and many more pioneering organisations; and our stellar inaugural event in Paris this June, we are now looking forward to the landmark conferences in London and New York, both in November, and both across two days: Law Firm Day, and Inhouse Day.
Discover more from Artificial Lawyer
Subscribe to get the latest posts sent to your email.