Kira Systems – AL Product Review – Part One

Kira Systems – AL Product Review – Part One

(Part One covers overall score, whether the product met expectations and ease of use.

Part Two, to be published tomorrow, will cover ease of training, customer support and value for money.)

—

Introduction

This is the first part of the product review by Artificial Lawyer of legal AI document review pioneer, Kira Systems.

Feedback was called for in November and 24 law firms and corporates responded in full, plus one partial review in addition. This represents approximately a quarter of Kira’s customers at an organisation level.

Reviews of products are never an exact science, but it is hoped that this combination of scoring and verbatim responses to the review questions will provide potential customers with some insight into how a number of the world’s leading law firms and corporates are making use of this legal AI product, what they like about it, and what they think could improve.

The reviews are predominately from US and UK law firms, several of which are global, plus firms from Canada and across Continental Europe. There are also several corporates, which are mostly global businesses. In short, these are very significant customers and the views represent the experience of a number of users inside each organisation. For the vast majority the views are of ongoing customers.

Naturally, this is feedback from third parties. All potential buyers of a product should do their own research and speak in depth to the company providing the product. It is also recommended that you conduct a pilot before committing to a long term contract, whichever type of legal AI product you choose.

Overall Score

Overall score of between one and four: 3.6 out of 4

Key:

4 Stars – Delighted! 

3 Stars – It’s good, but could be better.

2 Stars – It’s OK, we’ll keep shopping around.   

1 Star – Not a good experience. Avoid. 

—

Would you recommend this product to others? 

Customer Feedback

The comments below are taken verbatim from the many pages of responses sent to Artificial Lawyer as part of this review. The views expressed correspond to the questions asked in the Call for Feedback.

Feedback is arranged into three groups for ease of use: Praise; Areas for Improvement; and Criticism. The latter critical comments should always be taken in context, e.g. in this case the clear majority of users have found the product to be helpful.

The below comments were chosen because they stood out and/or were representative of several others with similar views. Some sections are in bold type to highlight especially significant comments.

Did Kira meet your expectations?

Praise:

• Kira is above and beyond what I expected.  This comes from experience after having Kira successfully meet multiple project requirements and having used other extraction technologies that have fallen short.
• Our usage of Kira has been a critical part of our success in supporting our transactional practice groups and our clients. We have won pitches based on our ability to leverage Kira. 
• Yes, it met mine and the firm’s expectations. It really made a difference and saved from 30% – 50% (of time) on over 100 projects/matters we used it on. We brought in new business, brought back old business that had gone inhouse and won M&A business from clients that were not doing that type of work with us.
• Yes it did. The tool is accurate, easy to use and easy to train. Kira have continually added new functionality over the years, refined the user experience and added new provisions for different areas of practice
• Kira, when utilised correctly, is an excellent tool for dissecting large volumes of data efficiently. Our firm is also effectively leveraging the machine learning on transactions.
• We brought Kira in about 3 years ago and had low expectations whilst looking at all of the AI providers, and after deciding on Kira we found it effective at extracting the provisions we trained.
• Kira exceeded our expectations. The out of the box training is great but the flexibility of the system and the range of topics and types of documents it can be trained on has been impressive.
• (continued from point above) I think you also have to remember what the alternative is – if it is 20 paralegals/solicitors in a room looking at thousands of documents you aren’t going to get 100% consistency and it would be wrong to expect this of Kira. I think people should also bear in mind that lawyers don’t always have the same definitions of concepts so there may be some bespoke training required.
• Yes – our doc review team uses it now as a matter of course on most M&A transactions.
• It lived up to [expectations], and then some, impressing us with other features like the work in progress aspects.
• We monitor the market for developments with respect to AI document review and have conducted demos and some pilots of competitor products; however, we feel strongly that Kira’s core functionality aligns with the value we want to provide our clients.
• We ultimately selected Kira not only because the technology met the functionality standards we were looking for, but also because of the candour of their executive team in acknowledging that there are limitations in AI generally and Kira specifically and that to achieve 99% plus accuracy we would need to combine the tool with our own human capital.

Areas for improvement:

• It would have been far better if you had been able to improve upon and ‘fine tune’ the existing models, as many of the current models just weren’t accurate enough to warrant using. Therefore, the only option was train up models from scratch which was extremely time consuming.
• UK docs, yes, meets expectations. Overseas very good as a project platform, although does not have taught provisions. Training new provisions was harder than expected.
• The accuracy of extractions was around what we expected, it isn’t perfect but we didn’t expect it to be so we are still working on checking and validating the data that comes out of Kira, but still making time savings.
• It could be better if the accuracy improved and if there was a better ability at extracting ‘value’ provisions such as dates, names, amounts, single words etc as it struggles with these types of provisions at the moment.
• It could also be easier to train and monitor the training, in order to train your own provisions it takes a lot of time and validation and takes more examples than we initially thought it would. Kira estimated it takes about 30 examples and we are looking at closer to 150 for most provisions.
• Competitors have more and better visuals, Kira could really catch up here.

Criticism:

• We ran a few trials getting Kira to search for relatively common clauses (assignment, change of control, governing law etc.) and the level of accuracy from the system was disappointing. [Note: this was an outlier view and is somewhat unusual as most of the other users of the system did not come to this conclusion. However, it is included here to show that not every user was able to make the most of the system. This view should be balanced against the many positive views above.]

—

Was it Easy To Use?

Praise:

• Generally our new users are up and running independently in a few minutes.  Admins are relatively comfortable after the first day.
• Yes, it scored highest on the ease of use metrics of all the products we compared.
• Yes – as the system is highly intuitive.
• Kira has always been a straightforward and easy to use platform. Our users are able to get to grips with the functionality quickly with minimal ongoing support.
• (Continued from above comment) The Kira user interface is clean and structured, not clustered with rows and rows of options. The reviewer experience has been improved over the past year, with functionality added and enhanced to make the review of documents easier (e.g. the ability to zoom into the document in the reviewer panel).
• Yes it is, definitely. And to the extent of my market overview it is probably the easiest to use of them all.
• Yes, the review layout is intuitive to anyone who has reviewed documents or contracts before.
• The interface was really good, and the option to view/export the extracted clauses in isolation or as part of the main document was helpful.
• Yes easy to use, although you must train lawyers BEFORE they need to use on a live matter.
• Yes it is – the layout is intuitive and the ability to have Members/Owners is useful as it means you can allow junior people to work on the system without pushing through training data until an ‘Owner’ has looked at it.
• Yes – and even over the time we have had Kira this has improved.  For example the admin functions are much slicker and allow more control and reflect the way we want to use it.
• Yes, it was easy to use. Simple actually, that is its beauty that anyone – even lawyers – can use it with a minimal amount of training. We fit it right into our M&A workflow and associates picked right up on it.

Areas for Improvement:

• Download from certain data rooms is very easy, but needs better integration to other data rooms as upload can be tricky.
• There are occasionally issues with it timing out / documents not loading properly but I wouldn’t rule out the firm’s internet as being at fault!
• (Continued from above comment) The Quick Study section could be set out better, maybe with more filterable lists and easier to define provision groups and sections. It could also be easier to switch worksheets across projects and collect documents relating to the provisions. Quick Study is something that you get used to quite quickly though so this is not a big issue.
• Kira has a very friendly user interface. It is easy to show to new users. The only area that requires work would be the OCR so that it isn’t as fiddly highlighting new provisions.
• Not ‘one click’ easy.  It does take some follow up to make sure everyone is getting the most out of all the product’s features.
• Ultimately we’d like to see the tool build out further functionality such as subsequent to extracting certain provisions, then automatically populating the actual answers to questions – this would be a great value add.

Criticism: 

• Even the critical users had generally positive things to say about ease of use. E.g. a firm that gave Kira an overall score of 2 out of 4 still said it was easy to use.

—

To be continued: Part Two, to be published tomorrow, will cover ease of training, customer support and value for money.

—

Methodology:

Users of Kira Systems were asked to respond to the following questions and to say where there could be improvements.

• Did Kira meet your expectations?
• Was it easy to use?
• Was it easy to train?
• What was the customer support like?
• Overall, has it been value for money?

Then, they were asked to provide an overall score between one and four:

4 Stars – Delighted! 

3 Stars – It’s good, but could be better.

2 Stars – It’s OK, we’ll keep shopping around.   

1 Star – Not a good experience. Avoid. 

And finally, to say if they would recommend the product or not.

All responses were emailed to Artificial Lawyer and are published anonymously.

More legal AI/tech companies will be reviewed in 2019.

—

Copyright Notice: This review has been produced by Artificial Lawyer Limited – It is not to be reproduced without permission. Thank you. Dec, 2018.