Panoram, a legal tech-focused advisory group created by former Neota Logic execs Greg Wildisen and Rick Seabrook, recently conducted a large-scale data breach project for a client after a cyberattack. They found that genAI tools outperformed all other options. The job they carried out, code name Project Orange, is as follows.
First, some context. Wildisen explained to Artificial Lawyer: ‘A cyber breach is very much an unbudgeted cost for a company. It’s also very expensive to deal with.’ I.e. they want this handled fast and as economically as possible.
Aside from the disruption it caused, the company in question had to go through all of its internal files to find out what PII (personal identifying information) may have been affected. As Wildisen noted it was a major job and one that even now large teams of paralegals are often drafted in to help with. But, that has to change.
‘The problem with human reviewers is that they fall asleep [looking through such dense files]. People don’t do law degrees to find PII,’ he added.
And that added risk to such work as things could be missed. Plus, it’s slow. Plus, it’s expensive to do it this way.
He said that by applying genAI to the problem they achieved all three aspects of the famous trifecta: better, cheaper, and faster. Based on that evidence Wildisen concluded: ‘In five years no humans will do this type of review work at all.’
This is how they did it.
Project Orange – Cyber Use Case
The context
In this project the breached data set spanned some 10,000 documents. As with most breached data, identifying private data was key, and in this instance, identifying passwords (credentials) was essential due to the nature of the breached data.
The data set was originally human reviewed. To confirm the accuracy of that review, the client requested processing and review using traditional eDiscovery techniques including NLP. Panoram then ran GenAI search techniques to provide a comparison of human v eDiscovery v GenAI.
The results:
- Human Review provided an F score of under 0.7 (balance of recall and precision)
- eDiscovery provided an F Score of 0.6
- GenAI provided an F score of 0.9
Reason:
The discrepancy between human and GenAI in accuracy can be partially explained by the difficulty that humans have in identifying passwords especially in multipage documents.
The discrepancy between eDiscovery and GenAI is mainly attributed to eDiscovery having high false positive rates in identifying password like strings, creating a low precision score.
The GenAI technology was far superior in accurately identifying credential information and personally identifiable information (PII) data and reducing significantly false positives.
The Panoram AI technology is also capable of reviewing at 7,500 docs per hour vs. human review at an average of 40 docs per hour thereby reducing the overall project time to hours vs days/weeks.
—
As you can see, it was a carefully planned project and one that compared different methods.
After this and other similar experiences on volume-heavy projects, Wildisen noted that he expects major changes to happen now across the legal field, from improvements to eDiscovery, to the evolution of other data-rich tasks.
He then mentioned another example, in this case of helping an energy company that usually buys dozens of rooftop spaces a year to install solar panels. That work requires plenty of human review of complex sets of real estate documents, i.e. high volume again. They’ve applied genAI to that also and were also very pleased with the results.
So, what does this all mean?
‘Law firms will need to redesign how they do things. Legal tech will be the platform that they use to do this type of work,’ he said.
Now, you could – quite rightly – say that legal tech has long been used in solving legal work problems. But here Wildisen is taking things a bit further, i.e. there really will be very little human input in whole sections of a complex workflow. Naturally, this is no ‘end of lawyers’ situation, far from it. But, for certain types of task, as shown with Project Orange, it will seem totally inefficient and overly expensive to rely on humans to handle core aspects of the matter.
Moreover, Wildisen concluded: ‘GenAI is now available to everyone [i.e. the law firms and their clients].’
In short, given what can be achieved, and the fact that the technology is obtainable and useable by just about everyone in some way, a substantive change to the legal market seems inevitable. And Panoram is planning on being right at the centre of this change.