Artificial Lawyer recently interviewed Maura Grossman, one of the most experienced and knowledgeable e-discovery experts in the business. Grossman is currently Research Professor at the David R. Cheriton School of Computer Science at the University of Waterloo, Canada. She also runs her own e-discovery consulting practice.
Grossman is in addition a senior alumna of one of the world’s leading law firms, Wachtell Lipton Rosen & Katz, where she worked for over 16 years. Artificial Lawyer is very honoured and grateful that she spared the time to answer the following questions about e-discovery and the future role of AI in the legal sector.
There are now over 200 businesses offering e-discovery services, ranging from start-ups, to very large legal publishers, to Big Four accountants, to photocopier companies, and everything in between. How can the legal market support so many e-discovery providers?
In the long run, I am not sure that the market can support so many e-discovery providers, because the current pricing model is essentially unsustainable. As we speak, substantial consolidation is occurring in the marketplace, with providers offering one-stop-shopping for data processing and hosting, analytics and review software, and even managed review services. But most are merely a repackaging of the same opaque, labor-intensive, complicated, and costly solutions that drove up the cost of litigation, achieved mediocre results, and excluded solo and small-firm practitioners.
When Gordon Cormack and I first published the results of our research in the Richmond Journal of Law and Technology—five years ago—indicating that technology-assisted review (“TAR”) could provide a more effective and efficient solution to the e-discovery problem, many vendors and service providers were quick to label their existing software solutions as “TAR,” without providing any evidence that they were effective or efficient. Many overpromised, overcharged, and underdelivered. Sadly, the net result was a hype cycle with its peak of inflated expectations and its trough of disillusionment. E-discovery is still far too inefficient and costly, either because ineffective so-called “TAR tools” are being used, or because, having observed the ineffectiveness of these tools, consumers have reverted back to the stone-age methods of keyword culling and manual review.
In an effort to clarify which TAR methods were more (or less) effective, and those that were outright ineffective, we coined and trademarked the term “Continuous Active Learning” (or “CAL”) for the process we invented and demonstrated to be effective in our 2011 Richmond Journal article. We subsequently showed in a study, published in SIGIR 2014, that CAL was superior to other methods offered by vendors, which we labelled as “Simple Passive Learning” (“SPL”) and “Simple Active Learning” (“SAL”). The CAL method was cited approvingly by the court in Rio Tinto v. Vale, and again by reference in Hyles v. City of New York.
All of a sudden, many vendors are now rushing to label their wares as “continuous,” whether or not they implement the method we evaluated in our SIGIR paper and called CAL. The consumer has no way to determine whether the latest “continuous” offering is the method we refer to as CAL, or simply a new branding sticker placed over the previous “TAR” label, which was itself just a rebranding of “ECA” (early case assessment) or other tools, which overpromised and underdelivered.
Nonetheless, I remain hopeful that eventually, the promise of TAR will be realised, reducing the vast expenditures on e-discovery to the point that the market will not support so many e-discovery vendors, and more litigants will have access to the justice system as solo and small-firm practitioners, who are effectively excluded from almost all of the current commercial TAR offerings, will have access to effective, efficient, and inexpensive e-discovery tools.
E-discovery has deep roots in the US, with even the US Department of Defense funding academic research into text retrieval systems in the early 1990s that helped to influence the technology. Also, litigation discovery practices in the US are more extensive than those in other countries and hence, there is a demand for more efficiency. But, are there other reasons why e-discovery has become so massive in America?
Litigation represents a large part of the cost of doing business in the US, and discovery is a large part of the cost of doing litigation. Other jurisdictions may be less litigious, and many, particularly civil-law jurisdictions, afford litigants less latitude for discovery, but the fundamental challenge of e-discovery (or e-disclosure) is essentially the same: to find as nearly all of the relevant information as can be found, at proportionate cost. The Irish and UK courts have recognised the role of technology in achieving proportionality in e-disclosure, and it is only a matter of time before other jurisdictions do, as well. There is simply no alternative but to use new technologies to search for evidence; the question is whether the parties will use effective and efficient tools and methods, or ineffective and inefficient ones.
The US Department of Defense has indeed funded text retrieval research for many years, most notably through the Text Retrieval Conference (“TREC”), now in its 25th year. TREC is an international effort, bringing together academic and industry researchers, from throughout the world, to address common challenges in text retrieval, including, but by no means limited to, e-discovery. The major Web search engines owe much of their heritage to TREC. So does Watson. In conjunction with their 25th anniversary, TREC will be holding a celebratory event featuring a series of talks detailing its history and these important connections. The celebration is open to the public, but it is necessary to pre-register, as it will be held at a federal facility (see here).
It would seem that there is a spectrum of TAR (technology-assisted review) in e-discovery, with Predictive Coding regarded perhaps as at the upper end of this in terms of its capabilities. From your experience how many law firms and in-house legal teams are now using Predictive Coding in the US?
It is difficult to know how often TAR is used given confusion over what “TAR” is (and is not), and inconsistencies in the results of published surveys. As I noted earlier, “Predictive Coding”—a term which actually pre-dates TAR—and TAR itself have been oversold. Many of the commercial offerings are nowhere near state of the art; with the unfortunate consequence that consumers have generalised their poor experiences (e.g., excessive complexity, poor effectiveness and efficiency, high cost) to all forms of TAR. In my opinion, these disappointing experiences, among other things, have impeded the adoption of this technology for e-discovery.
In our 2013 TAR Glossary, published in Federal Courts Law Review, Gordon Cormack and I referred to TAR as a “disruptive technology.” While we have yet to see the disruption, we have no doubt that it is coming.
Following the now famous Pyrrho case in the English courts, Predictive Coding has been given what we could call an ‘official seal of approval’ in the UK. Given the US’s greater experience with TAR, what should solicitors and barristers (and perhaps judges…) in the UK be paying most attention to?
Not all products with a “TAR” label are equally effective or efficient. There is no Consumer Reports or Underwriters Laboratories (“UL”) that evaluates TAR systems. Users should not assume that a so-called “market leading” vendor’s tool will necessarily be satisfactory, and if they try one TAR tool and find it to be unsatisfactory, they should keep evaluating tools until they find one that works well. To evaluate a tool, users can try it on a dataset that they have previously reviewed, or on a public dataset that has previously been labelled; for example, one of the datasets prepared for the TREC 2015 or 2016 Total Recall tracks.
As is often the case, many lawyers are fearful about any new technology that they don’t understand. There has already been some debate in the UK about the ‘black box’ effect, i.e., barristers not knowing how their predictive coding process actually worked. But does it really matter if a lawyer can’t understand how algorithms work?
In general, attorneys – like all humans – fear the unknown and the unfamiliar, maybe even more so. Yet, there are plenty of unknowns (such as how the engine in an automobile works) that do not engender the same kind of fear and distrust because they have become familiar, and because they are generally predictable and reliable.
The key to overcoming fear and distrust of the so-called “black box” is not so much to expose its inner workings as to observe that its behaviour is predictable and reliable, and that the controls one needs to operate it are reasonably comprehensible (e.g., the steering wheel and brakes).
Many TAR offerings have a long way to go in achieving predictability, reliability, and comprehensibility. But, the truth that many attorneys fail to acknowledge is that so do most non-TAR offerings, including the brains of the little black boxes we call contract attorneys or junior associates. It is really hard to predict how any reviewer will code a document, or whether a keyword search will do an effective job of finding substantially all relevant documents. But we are familiar with these older approaches (and we think we understand their mechanisms), so we tend to be lulled into overlooking their limitations.
A growing number of legal AI companies have pioneered cognitive engines to handle due diligence and document review as part of transactional, compliance and advisory legal work. How different are such systems to the best predictive coding systems that are used for e-discovery?
There is some similarity in that the core problem in both tasks is one of classification: Does each document (or clause within a document) meet some criterion that makes it of interest. However, e-discovery tends to be more ad hoc, in that the criteria applied are typically very different for every review effort, so each review generally begins from a nearly zero knowledge base. As I understand the due diligence process, and the AI tools in current use, the criteria that determine the information being sought may have more in common from one application to the next, so more effort can be placed on advance learning to determine the categories of interest into which various documents (or clauses) may fall. Thus, because the tasks are somewhat different, so are the tools and processes required to address them.
And, if there is a degree of similarity, why don’t document analysis companies do e-discovery, and vice versa? I.e. will we see a convergence in the market between the two streams of machine learning software?
As I mentioned previously, the tools and workflows that are typically brought to bear for these two activities are somewhat different. I think this is partly due to their different lineage, and partly due to differences in the problems being addressed. I don’t see convergence occurring immediately, but it may happen over time. We are certainly seeing an increase in the downstream use of TAR tools for information governance and related activities.
5 Trackbacks / Pingbacks
Comments are closed.