By Jim Wagner, CEO, Lean Law Labs.
As someone who has built multiple AI-powered businesses in the legal community, I know firsthand the exciting potential of technology to transform the way we practice law. From predictive coding in electronic discovery, to AI-based contract analysis, legal tech has the power to make our jobs easier and more efficient.
But with any new technology comes risk, uncertainty and responsibility. It’s easy to get caught up in the hype of the latest buzzwords and trends, but when it comes to serving a demanding audience like lawyers and their clients, you better understand that there’s a difference between ‘playtime’ and ‘production.’
What do I mean by that? Well, let’s say that (like me) you’re tinkering with GPT-3, the latest AI tool that promises to do everything from drafting and analyzing contracts, to crafting engaging and thought provoking articles in publications like Artificial Lawyer (wink).
It’s fascinating to see what GPT-3 can do and the possibilities are in some cases nothing short of mind blowing. But before you plan your early 2023 implementation, you may want to exercise a bit of caution. When it comes to using AI in a production environment – i.e., serving real customers with real expectations – you need solutions that deliver reliable results that you can explain to your clients … and potentially to a lot of other stakeholders, including courts and regulatory authorities.
This point is worth highlighting in light of a recent paid advertisement that I received from a company that is already using GPT-3 in production for contract analytics. The ad states that their tool ‘will provide an overview of the contract and potential risks. The wild part is that it’s right a lot of the time.’ (emphasis added). Let that last one sink in a bit.
Maybe in 2023 you can also try this line: ‘Dear client / court / regulator, we know it’s hard to believe, but a lot of the time you can rely on what we tell you.’
I don’t think this is a conversation that many compliance or legal professionals want to have.
And that conversation will become much less pleasant when someone on the other side of the table serves up a couple of quotes from one of the world’s leading AI experts, such as Cassie Kozyrkov, who says things like, ‘There’s something very important you need to know: ChatGPT is a bulls****er,’ and ‘ChatGPT is indifferent to the truth.’
ChatGPT, by the way, is ridiculously powerful and promising. Over the last few weeks, I’ve asked it a lot of complex questions about contracts. I’ve asked it to solve some problems that we spent a decade trying, in some cases with limited success, to solve for our clients. The results of these limited efforts to date are at times astounding. But, at other times, they are nothing short of confounding.
On Monday, I may ask GPT-3 to summarize for me all of the notice requirements in a publicly available agreement for a large SaaS company. GPT-3 could blow me away by offering a concise bullet list of each notice requirement, the triggering event for each notice requirement, and also where the notice requirement lives in the agreement. My mood: this is awesome.
On Friday, I can ask GPT-3 to analyze the same agreement and to provide me with a summary. GPT-3 dutifully analyzes the agreement and presents the results. To quote a phrase, ‘the wild part is’ … the Friday answers are completely different than Monday’s answers. My mood: don’t ask.
Now, before you misinterpret my meaning in this post I want to be clear about a few things:
- I don’t have a horse in this race. We sold our AI-based contract analytic business in May of 2020. Yes, I’m working on a startup that is contracts-related, but when it comes to NLP and contract insights, we would prefer to partner with leaders in the space. Building thousands of models in an AI platform is a serious undertaking that is not a priority for us at this time.
- I am not in any way down on GPT-3, BERT, neural networks, transformers, one-shot or zero-shot learning. To the contrary, I am inspired by the possibilities from these technologies and can see a clear path for them to radically improve how legal professionals get their jobs done. It will be fascinating to see what the leaders in this space, like DocuSign, Zuva and Evisort, who have deep expertise at building and deploying models at scale in production environments, do with these emerging capabilities. I am confident that they and others will make the output from their efforts consistent and reliable, and that we will all be blown away and the better for it.
- I am acutely aware that the version of ChatGPT most of us are playing with is largely in ‘toy mode’ at the moment and that anyone who is bringing this technology to a production environment already has some idea of how to train and constrain the inputs and output, and to deliver consistent and quality results (with bonus points for helping clients to avoid falling for GTP-3’s unique ‘BS’ing’ strengths).
- I also recognize that the right answer for all things AI, and especially for new AI, is to view the technology as an assist rather than turnkey automation. If this one message could get through, we would see better adoption, better satisfaction and better usage of a lot of legal tech.
- I also don’t labor under any false belief that humans – no matter how pedigreed – are infallible. Leaders in the legal tech field proved a long time ago that people make a lot of mistakes and that perfection is not the standard by which any technology should be judged.
- Lastly, for any folks already to market with a product incorporating GPT, a very sincere good on you for being on the leading edge of a wave that I believe will be bigger and faster than any legal tech AI wave that’s ever been ridden. Learn to ride that wave well, without falling over the edge of your board, and you could build a very good business. I am absolutely rooting for you!
I’ll close with this. Before the holidays, ChatGPT’s helped me with a LinkedIn post by answering a simple question – Should businesses be using GPT-3 in a production environment?
ChaptGPT quickly responded to my question with utter candor and self-reflection:
‘Given that your business analyzes contracts using AI for enterprise customers who expect consistent results, it would not be appropriate to use GPT-3. Instead, you should consider using more established and reliable machine learning algorithms that have been thoroughly tested and evaluated in similar use cases. It is important to carefully evaluate the capabilities and limitations of any technology before using it in a production environment, and to ensure that it is stable and reliable before relying on it to make important decisions.’
So, was ChatGPT telling the truth? Or was it, as Cassie Kozyrkov suggested, possibly ‘BS’ing’ me, as perhaps only ChatGPT can do?
The honest truth is I don’t know.
Let that one sink in too.
—
About the Author: Jim Wagner is co-founder and CEO at Lean Law Labs, where he and his colleagues are tackling some of the root problems causing the biggest waste in contracting.
Prior to Lean Law Labs, Jim served as Vice President of Agreement Cloud Strategy at DocuSign, which he joined in May of 2020 as part of DocuSign’s acquisition of Seal Software. Jim served as President of Seal, an industry-leading AI platform for contract analytics, is a serial founder in the legal tech community, and also is the author of multiple patents related to the use of AI and analytics in the context of the legal industry.
–
[ This is an educational guest post for AL. Main pic: portrait of Jim by generative AI.
P.S. I am on sabbatical, but up until this Friday you will see a handful of articles that were written in 2022 and over the holidays. ]