vLex and startup Vecflow, which were among the companies in the Vals genAI performance study (see here), have sent Artificial Lawyer some comments on the results. The other companies involved in the published study include Harvey and Thomson Reuters. (Note: there are AL TV Product Walk Throughs of Vecflow and vLex to give you some more context – see here.)
Below are the verbatim comments from Vecflow and vLex.
The link to the full Vals AI report is here.
—
About the company: ‘Vecflow is a legal AI company built to optimize workflows across the firm including research, due diligence, drafting, and document review. By using your firm’s documents, information, and knowledge, Vecflow delivers customized, one-click workflows – accelerating and improving legal work.’ Note: its main AI tool is named ‘Oliver’.
Comment on Vals Results
Vals.ai’s study compares Oliver to four other established offerings: Thomson Reuters, LexisNexis, Harvey, and vLex. The study confirms that Oliver–with only 6 months on the market and far fewer resources than its peers– “often performed on par with or outperformed more established offerings.” This includes companies valued in the billions.
The evaluation tested AI legal assistants against human lawyers across seven legal tasks. Notably, Oliver outperforms the human lawyer in Document Q&A and Document Summarization. Following Harvey’s drop-out, Oliver is also the sole competitor in SEC EDGAR Research–the only task involving multi-stage, complex reasoning.
‘Intelligent AI workflows represent the future of legal work,’ said Vecflow CTO, Joe Parker. ‘The evaluation highlights three critical findings:
1. AI assistants are already surpassing lawyers in several critical areas
2. Oliver’s research capabilities stand unmatched in the legal tech sector
3. AI performs best when complementing a lawyer’s existing workflow”
Vals.ai specifically noted that “in the challenging EDGAR research task, Oliver, through its coordination of specialized AI agents, stood out as the sole offering capable of nearing human-level performance.” Research over EDGAR means multiple levels of iteration and research, currently unrivalled by the study’s other providers.
“We’ve already seen our customers able to handle more cases in less time. And there’s evidence they can do this at even higher quality,” said Thomas Bueler-Faudree, CPO of Vecflow, in an interview with Vals.ai. “You’re going to see smaller firms that are technologically forward being able to rapidly do ten, 100 times more material in cases than they used to. I think you’re going to see increasing competition in law, and hopefully you’ll see one of the big four law firms appear in the US.”
Since the benchmark was conducted six months ago, Vecflow has made significant improvements to Oliver based on customer feedback. The company reports that Oliver’s capabilities have advanced substantially beyond the already impressive results shown in the study.
—
About the company:
‘vLex is a global legal intelligence and legal AI pioneer empowering legal professionals to work smarter. With nearly three decades of experience in legal tech and access to over one billion legal documents, vLex developed Vincent AI – a comprehensive AI-driven workflow platform, that is powered by vLex’s vast library of structured legal data, providing added confidence with citations to verified authoritative sources.’
Comment on Vals Results
‘The Vals Legal AI Report focused on document extraction tasks, which didn’t involve tasks that required a database of law. “We are quite pleased with Vincent AI’s performance, especially considering that document summary tasks do not leverage our vast legal database of global information,” said Ed Walters, Chief Strategy Officer of vLex.
The report noted that “Although our evaluation focused on a small slice of Vincent AI’s capabilities in U.S. jurisdictions, its support for international matters is a significant strength. For global law firms, this capability may provide a level of utility unmatched by other tools, making Vincent AI an attractive choice.” In fact, in the recent “AI Smackdown,” designed to test how AI-driven research platforms provide direct answers to legal questions, the Southern California Association of Law Libraries found that vLex provided the most depth in challenging AI tasks.
Other takeaways from the Vals Legal AI Report include:
- Vincent “gave responses exceptionally quickly as generally one of the fastest products we evaluated.”
- “Vincent AI’s design is particularly noteworthy for its ability to infer the appropriate subskill to execute based on the user’s question, adapting to the user query. In cases where clarification was needed, Vincent AI would proactively ask follow-up questions to refine its understanding, ensuring tailored responses.”
- “When the legal research database did not have sufficient data to answer a question, Vincent AI refused to answer, rather than hallucinate an illegitimate response.”
- “The answers provided were impressively thorough . . . offering valuable additional context to aid their understanding and workflow.”
- “Results are promising and reveal that AI does indeed deliver value in the context of legal work,” the Vals study concluded. “The AI tools were six times faster than the lawyers at the lowest end, and 80 times faster at the highest end.”
- “The Vals Legal Report is a great validation for the first generation of legal AI tools,” said Walters. “The results are promising and reveal that AI does indeed deliver value in the context of legal work.”
—
The AL View
Overall, it looks like legal AI as a whole has come out the winner here. Naturally, there are variations between the companies that Vals tested – after all, they’re all made by humans – but for this site the biggest takeaway is how well they did as a group.
Now, if you are a glass half-empty type of person then you’ll say: ‘But, they were not perfect, they made some mistakes, and a lawyer would still be needed to complete those tasks.’
And if you are a glass half-full type of person, then you may say: ‘By using AI, the work of a lawyer – which is complex, language-based, and demands lots of reasoning – can now partly be done via software, i.e. in effect operating in a supporting role. Moreover, this genAI technology is still in its infancy and new models are arriving every few weeks. The pace of change is incredible and it’s clear that any lawyer or law firm ignoring legal AI is putting their head in the sand – and losing out on tremendous productivity and efficiency gains.’
And that’s why the Vals study is so useful. It underlines, via a third party, that legal AI tools really are capable of doing a lot. Are they ‘perfect’, can they do ‘everything’? No. But, it’s clear they really can do enough to provide a truly tangible benefit to the legal sector.
P.S. as noted by Vecflow above, each week that passes sees these companies improve what they can do. Foundation models evolve, so does what each vendor can provide after refining their outputs and improving the application layer of their offerings. In short, whatever the results you see today, they will be better in the months ahead. In fact, Vecflow notes that they are already better. And that is likely the case for all of the vendors here.
—
Conference News: Legal Innovators California – June 11/12 – San Francisco
The Legal Innovators California Conference in San Francisco takes place on June 11 + 12.
The landmark event in the heart of the Bay Area tech world will be across two days: Day One: law firms, and Day Two: inhouse. Come along and engage with the leading legal tech experts from law firms, inhouse and tech companies in California and from across the world.
See here for more information, tickets, speaking and sponsorship opportunities.
