As readers will have heard, there was recently something of a battle between two AI-driven case analysis systems in New York earlier this week, which has created a lot of interest in the legal tech community.
In the blue corner was Casetext, a very well established company in this area (see Artificial Lawyer interview from last month here with founder, Jake Heller).
In the red corner was Andrew Arruda, the well-known AI advocate and co-founder of ROSS Intelligence, better known for its legal research tool, but which had just announced the arrival of EVA, a system that appeared to have some similar capabilities to Casetext’s CARA application.
And so there was only one thing to do: a comparison test. Artificial Lawyer has been able to get Jake’s side of the match, which is presented below in his own words. Andrew is also very welcome to put his side across too. But, first, here’s Jake’s personal view.
It was no surprise that ‘artificial intelligence’ was one of the hottest topics at Legalweek 2018. We heard about AI in the opening keynote, in analytics and emerging technologies sessions, and earlier this week, I took part in an AI Bootcamp panel on real use cases of artificial intelligence.
Increasing numbers of law firms have transitioned from speculation to reality by bringing on AI assistance in contract review, legal research, analytics and more. The “Robot Lawyer” is real, and it’s here to stay.
During the AI Bootcamp panel, ROSS announced another AI tool hoping to join the “Robot Lawyer” ranks: EVA, a legal research tool that, on the surface, sounds similar to Casetext’s CARA, in that it asks a researcher to upload a brief and then uses the information contained in that brief to assist in legal research.
Soon after their announcement, Sarah Glassmeyer tweeted out a request for a real comparison between the two tools; or, as she jokingly put it, “ROBOT FIGHT ROBOT FIGHT ROBOT FIGHT!”
Aside from sounding like a lot of fun (who doesn’t love a good robot fight?), we thought it would be genuinely worthwhile to do a side-by-side analysis of the two tools, putting marketing and PR aside to compare the products based only on functionality — and Legalweek seemed like the ideal time to do it.
Lawyers are trying in earnest to choose the products that will make them most efficient and give them a competitive edge; they deserve to know which products do what, and how well they actually do them. We invited ROSS CEO Andrew Arruda (who was also at Legalweek) to showcase EVA, but unfortunately he didn’t join us to defend his product. So, we created an EVA account and went ahead with the comparison anyway.
Though Richard Tromans said on this blog that it came to a friendly draw*, we respectfully disagree.
CARA clearly came out on top due to its ability to instantly surface information and documents (cases, briefs, case summaries and more) that are highly relevant to the issues the researcher is working on. And frankly, we found that many of EVA’s features simply didn’t work very well. You can view a recording of the comparison and decide for yourself here.
As Andrew pointed out, EVA is still a pretty new tool and has a long way to go. We’d love to re-stage the Robot Fight — maybe in six months at ILTA this August, or next year at Legalweek — once ROSS has had time to resolve some of the key bugs and add some core functionality. We hope this is ample notice.
But setting aside what the technology can (and can’t) do, we believe that ROSS’s release of EVA underscores the importance of transparency in what legal technology vendors provide to their customers. In fact, we’ve found that many of our customers are initially skeptical of AI, as they (rightfully) believe that for the past few years, much of AI in legal tech has been all hype and no results.
Of course, once they see CARA in action, their skepticism is gone in an instant – which is why we decided early on to make it possible for anyone to take CARA for a spin via a free trial. As more AI tools embrace transparency around their technology, it will become easier for AI vendors and buyers alike to fully understand the real value of AI technology and make meaningful comparison across tools. We hope that the release of EVA means that ROSS is moving towards more transparency around their historically secretive products.
Second, this sort of healthy debate and competition incentivizes us all to be better at what we do, and to make sure that we are delivering the best and most useful products to customers. The team at Casetext has been working hard over the past few years to offer a superior legal research tool that will eventually eliminate the need for legacy search tools, level the playing field and make lawyers and law firms more efficient.
New tools on the market remind us that we’re not the only ones who recognize the opportunities created by the slow augmentation of these legacy tools, and that we need to continue moving quickly, improving our products, and adding additional functionality to remain one step ahead.
Like my colleague Pablo Arredondo said during the Robot Fight, the winners in the end are going to be the users. As AI tools become more mainstream at law firms, it’s important that legal tech companies continue to facilitate open comparisons between products, provide transparency around what AI can do today and will be able to do in the future and push each other to build the tools that do the best work for their users.
The above article was written by Jake Heller, the co-founder and CEO of Casetext. Before starting Casetext, Jake was a litigator at Ropes & Gray.
—
[ * Editorial note (1): I guess I am always hoping for a happy win-win for everyone when it comes to legal tech innovation. P.S. After this piece, it will only be fair to allow Andrew Arruda at ROSS to also give his version of events, if he’d like to. …Andrew…?
(2) Re. ‘robot lawyers’ etc, as readers will know I always try to avoid the use of the term ‘robot’ in relation to AI tech, but in this case it seems to have been unavoidable. When in Rome…and all that…. ]
2 Trackbacks / Pingbacks
Comments are closed.