Kira Systems – AL Product Review – Part Two

Kira Systems – AL Product Review – Part Two

(Part Two, here, covers: ease of training, customer support, and value for money, plus Artificial Lawyer Conclusion.

Part One, yesterday, covered overall score, whether the product met expectations, and ease of use.)

Introduction

This is the second part of the product review by Artificial Lawyer of legal AI document review pioneer, Kira Systems.

Feedback was called for in November and 24 law firms and corporates responded in full, plus one partial review in addition. This represents approximately a quarter of Kira’s customers at an organisation level.

Reviews of products are never an exact science, but it is hoped that this combination of scoring and verbatim responses to the review questions will provide potential customers with some insight into how a number of the world’s leading law firms and corporates are making use of this legal AI product, what they like about it, and what they think could improve.

The reviews are predominately from US and UK law firms, several of which are global, plus firms from Canada and across Continental Europe. There are also several corporates, which are mostly global businesses. In short, these are very significant customers and the views represent the experience of a number of users inside each organisation. For the vast majority the views are of ongoing customers.

Naturally, this is feedback from third parties. All potential buyers of a product should do their own research and speak in depth to the company providing the product. It is also recommended that you conduct a pilot before committing to a long term contract, whichever type of legal AI product you choose.

Overall Score

Overall score of between one and four: 3.6 out of 4

Overall score chart. (AL Data 2018)
Overall score chart. (AL Data 2018)
Key:

4 Stars – Delighted! 

3 Stars – It’s good, but could be better.

2 Stars – It’s OK, we’ll keep shopping around.   

1 Star – Not a good experience. Avoid. 

Would you recommend this product to others? 

Would you recommend Kira Systems to others? (AL Data 2018)
Customer Feedback

The comments below are taken verbatim from the many pages of responses sent to Artificial Lawyer as part of this review. The views expressed correspond to the questions asked in the Call for Feedback.

This second part covers: ease of training, customer support, and value for money.

Feedback is arranged into three groups for ease of use: Praise; Areas for Improvement; and Criticism. The latter critical comments should always be taken in context, e.g. in this case the clear majority of users have found the product to be helpful.

The below comments were chosen because they stood out and/or were representative of several others with similar views. Some sections are in bold type to highlight especially significant comments.

Was it easy to train?

Praise

  • It only takes about 20 minutes to get a lawyer up and running and feeling comfortable with the system.
  • We trained a viable model for recognition of a local language assignment clause within a few hours. We were able to confirm its accuracy against known data from a manual review.
  • The Quick Study functionality is easy to use once trained on the tool. In addition, the Kira team were extremely accommodating in facilitating the training of our future users.
  • The platform has shown reasonable flexibility and robustness in custom training exercises. One of the main benefits of the platform, however, is the large library of pre-trained smart fields, which are available out of the box.
  • Can see recall and precision scores for different provisions so can really hone in on those which aren’t going so well
  • We note that Kira has a significant number of built-in provisions.  Moreover, they are open to suggestions for new provision sets and we have provided our input in instances where we believe that it would be of value to all Kira users for it to be trained on particular provisions. As such, there are instances where we shift the burden from ourselves to Kira to train the system.
  • Yes, I had two analysts spend less than a week training models that scored higher than 90% accuracy.
  • Kira is very easy to train. Highlighting examples of relevant text to teach Kira with is straightforward, and the quick study panel clearer shows you how many examples you have accumulated. Particularly key is the visibility of the machine learning F-score which is critical for ensuring  our taught models are robust and ready for use in client delivery.
  • You simply highlight language by hovering over it / tagging language. The results are fairly good, from what I could compare against each other they are better than the competitors’ results. So again a big plus for Kira. It was one of the reasons why we decided on Kira.
  • The mechanics are easy, but we haven’t done much custom provision training yet.
  • Yes, the machine learning piece is very easy to use. Highlighting portions of text is an intuitive way for our lawyers to teach Kira, and we have had excellent feedback internally in this regard.
  • Yes – it is easy to train as all you are doing is highlighting the relevant clause/word/section and clicking the provision from the list.
  • The training within Quick Study is a little more complicated, but once you get the hang of it is easy to use. It takes a while to do the validation and checks but these are very worthwhile as it improves the learning a lot.
  • Yes, I did four Quick Study custom provisions. If you have good examples they can be done in less than a day. Hours for that matter.

Areas for improvement

  • The actual accuracy and success of the training could be better, we have found that we need a lot of examples to train it and also that the more variety you give the system the less accurate it becomes, which seems counterintuitive.
  • The scoring system page could be improved by a reminder of what the scores e.g. precision, recall and F score relate to. The ability to roll back to previous training is helpful. Some niggles still exist around particular formats of documents e.g. dates when they are surrounded by a box and the way that information from a table exports could be better (in that it could integrate within the main export tab rather than on a separate sheet) – nothing insurmountable!
  • [Note: the following four comments are taken from a very detailed response, from a current customer which spent significant time considering Kira’s training capabilities.] The process is simple enough, but overall ability to train is long-winded.
  • Two very painful examples which will bring Kira’s super-simple means to a limit: 1 – large tables, like in X (an EU Country) public records of title (buildings, property in general), 2 – Party’s names, Kira gets confused by other names in the contract and or names in footers. (continued)
  • Competitors offer a means for risk allocations, sometimes even in quite deep integrations and in interesting ways.(continued)
  • The Out-of-the-box provisions are ‘not great’ as well, but neither are most of the competitors’. They probably have been trained by many different people, which is quite counterproductive. The models we trained ourselves, and just with one person, were definitely a lot better (ends).
  • Clicking the buttons is easy. The harder part is navigating the contract text and finding what you are looking for. This is more of a domain issue (e.g., knowledge of the content you are trying to locate) opposed to a technology issue. Kira, like any system, still requires a smart user to be behind the keyboard. 
  • Training the system to a reliable level of accuracy seems like it will take a lot of time, especially as any training has to be done from scratch, rather than on top of the inbuilt model clauses.

Criticism

  • We tried several different provisions in different areas of law to varying success. We recognise that identifying provisions can be difficult, but feel that Kira oversold the quickness of its ‘Quick Studies’ in developing custom models.
  • I wouldn’t agree that the OCR is well enough adapted to pick up handwriting, and watermarks can cause a lot of issues.
  • Kira can struggle with certain types of provision – sometimes surprisingly so.

What was the customer support like?

Praise

  • Reasonable turnaround time and good attitude to proactively following-up.
  • Very good. Kira have a useful support page with live agents and training material.
  • Sondra Rebenchuk is awesome.
  • It was very good – the firm contact we had at Kira was very responsive and helpful.
  • Customer support is great for general support queries, and our dedicated representative is very responsive.
  • Good, we have a good relationship with Kira and issues have always been dealt with quickly.
  • Even throughout the POC stage you are treated like a valued customer, rather than just another number on the list of law firms signed up sheet.
  • Customer support is an area that Kira really does shine. We have bi-weekly calls to provide updates to one another. We often find that our feedback is acted on quickly.
  • They did an excellent job incorporating the feedback and getting it to the rest of the team.
  • We have a dedicated AI Practice Consultant who conducts training exercises, provide support for our users and generally serves as our main point of contact. Kira’s support has been among the best of the vendors we have engaged.
  • We are in contact with the Design Team, Legal Knowledge Engineering as well as other functions in Kira and, as such, we have built a solid relationship.

Areas for Improvement

  • Very responsive and generally helpful, but sometimes feel a bit green on some of the more complex requests.
  • The customer support has improved over the years as the company has grown. Initially the level of support offered varied, but Kira have been pro-actively addressing these concerns, and understand how pressured the legal environment is when it comes to client delivery. This is an area Kira needs to continue to build upon in order to consistently deliver successful customer support in real time.
  • I don’t like to have to deal with ‘client success managers’ who change every year because they are always new to the company, know little about the product, are low in the hierarchy, and have to deal with the super complex issues of customer satisfaction. [However] Kira presented us with less of such success managers than others.

Criticism 

  • A strange mix of supportive and obfuscatory.

Overall, has it been value for money?

Praise

  • Yes, we recognise a clear cost savings compared to doing it manually. Opposed to other solutions I have worked with, the Kira cost to setup and onboard is much lower. Admin hours required to manage are relatively low.
  • Yes it has. The algorithms taught both by Kira and by us have proven to be highly accurate in testing, and the efficiency and workflow benefits of using the tool clearly achieved.
  • Yes, both from a marketing perspective, as well as from practical benefits from usage.
  • Our firm looks at Kira, and in particular the machine learning, as a longer term investment in our business. Through the machine learning of Kira, we expect to be able to deliver cost effective and timely solutions to our clients on transactions.
  • Hands down: yes – according to many Partners [at the law firm].
  • Yes – these types of tools are not inexpensive, but will soon become part of a client’s service expectation.  And, we are seeing considerable time savings from using it.
  • We are still assessing the actual gains in efficiency provided by Kira, however, the platform has delivered consistent value in a number of reoccurring scenarios such as:
    • RFPs;
    • internal NLP use-cases for our business units;
    • internal NLP use cases for our practice groups;
    • and transactional due diligence exercises.
  • We are still working on evaluating the full impact of Kira’s value but, to date, it has been invaluable on the matters we have used it for.
  • Leveraging Kira has not only transformed our contract review and abstraction offering by allowing us to undertake engagements faster and with greater accuracy, by incorporating Kira into our offering we have actually driven greater business opportunities to our organization.

Areas for Improvement

  • Adoption internally is still a challenge and the software is expensive, but it’s the most user-friendly AI platform I’ve seen.
  • Yes, but only for large projects at the moment. The learning has not been good enough to trust 90+% which means that it is currently just a review aide and still costs lawyer time.
  • Another issue with value is that you pay for the storage on the system (which makes sense) but the provisions use the documents for their data, so if you are constantly tweaking and improving your provisions by adding examples then you need to keep all of the older examples on the system, which cost you year on year and require you to increase your document limits.
  • We’re not sure, yet. We are not passing the costs on to clients.

Criticism

  • Value for Money? Possible, but not certain. There have simply been too many issues and too much overselling. Granted, if our firm had its processes more streamlined we could get more out of it.
  • Kira’s pricing models are not great. Just because most vendors have similar ones does not mean the customers like them, nor that they make sense with Kira.

Artificial Lawyer Conclusion

The feedback from a number of leading law firms and corporates has been fascinating and very illuminating, as it shows the real experiences of people working with this technology.

The feedback appears to show a community of customers who are overall happy with Kira’s capabilities, although several would like to see a range of improvements in the system. It’s also clear that even some customers who experience problems, e.g. around training, are sticking with it. This is presumably because, despite the occasional challenges, the overall benefits outweigh the negatives.

There are also a small number of reviews where the customers (or pilot customers) were not happy. However, even some of these conceded that some of the problems may stem from their own approach. Although, there was some blame levelled at Kira as well.

Overall, it would appear that – at least from the sample of responses here – that Kira Systems is well-liked by many users who look forward to using it for some time to come. That said, this support may in the longer term be predicated on the system continually improving, given that many users also appear to be well aware of other legal AI systems in the market.

One other point, not directly related to Kira, is that this feedback is also solid evidence of how leading law firms are leveraging AI technology. It shows that firms are making it work at scale on live client matters, and that partners are backing its use, if not directly calling for it. That in itself is a very positive sign for the legal AI sector as a whole.

Artificial Lawyer hopes you have found this review useful. In 2019 other companies will be reviewed.

Finally, a very big thank you to all of the law firms and corporates that took the time to respond to the Call for Feedback. Your insights have been invaluable and are truly appreciated. Thank you.


Methodology:

Users of Kira Systems were asked to respond to the following questions and to say where there could be improvements.

  • Did Kira meet your expectations?
  • Was it easy to use?
  • Was it easy to train?
  • What was the customer support like?
  • Overall, has it been value for money?

Then, they were asked to provide an overall score between one and four:

4 Stars – Delighted! 

3 Stars – It’s good, but could be better.

2 Stars – It’s OK, we’ll keep shopping around.   

1 Star – Not a good experience. Avoid. 

And finally, to say if they would recommend the product or not.

All responses were emailed to Artificial Lawyer and are published anonymously.

Copyright Notice: This review has been produced by Artificial Lawyer Limited – It is not to be reproduced without permission. Thank you. Dec, 2018.