Earlier this week Artificial Lawyer wrote about the new French law that seeks to prevent people analysing and sharing statistical data on named French judges’ decisions. The story went global and created a huge reaction.
But, now here is a Guest Post from someone in France who writes that he was right at the centre of the events that led up to the law passing this year. It gives some fascinating extra detail and some more context to what happened.
The author, Michaël Benesty, is a machine learning (ML) expert and qualified lawyer in France. He currently works at Lefebvre Sarrut at Neuilly-sur-Seine, Paris. This is his personal story and his views on what happened are from his own perspective and in his own words.
—
In 2016 I was a tax lawyer at Deloitte France (the brand name of the law firm is Taj), and well versed in machine learning.
For my needs, at that time (as a tax lawyer specialised in tax audit), I started a project in my free time to find French legal cases whenever the tax administration loses (they are hard to find manually and very useful in my specialty). For that, I did lots of ML to enrich the legal cases, extract some info and filter what I needed.
At some point I was informed that some judges had a strong bias regarding rights of aliens and asylum seeker claims.
I started to use the tool to find these biases and found some very large discrepancies among judges.
Then I asked a friend (Anthony Sypniewski, a ML engineer at Google NYC office) to help me create a website we called SupraLegem to let anyone use the tool for whatever bias they wanted to find.
We designed it in a way where you can check the results yourself manually if you want. Obviously for such software there is a strong need of accountability and transparency. Moreover, before being a tax lawyer I was a financial auditor, so let’s say that I am very sensitive to these topics.
We did it, we released it: a non-commercial project, no startup, totally free, no ads, no black box – using public open data. We did also some presentations at some ML meetups in Paris.
The content of an article we wrote about our findings was quite provocative as we published names and information on some specific judges.
For the story, we sent the slides to the Conseil D’Etat (administrative supreme court) one month BEFORE publishing the article, for validation of the content, and didn’t get any answer (though we spoke to them several times before and always got an answer).
Later I learned from my contact at the court that they were forced to use an old version of Firefox at the court which was unable to open a presentation on Slideshare, that led to no answer at all, so we published our article with the names.
Would he have opened the slides he would have asked us to remove names from the article and the website, and the story would have stopped there.
A few hours after publishing the article, the shit storm started.
The basic issue was that some judges had a very high asylum rejection ratio (close to 100%, with hundreds of cases per year), while others from the same court had a very low ratio, and in France cases are randomly distributed among judges from the same courts (there is no judge specialised in Moroccan asylum and the other in Chinese asylum for instance).
Basically, we believed there was no reasonable explanation to such discrepancies, which were stable year afters year.
The tool was transparent, you get some measures of bias for each judge plus the related legal case texts to back the numbers. So any measure can be manually checked.
We got plenty of e-mails from judges from courts all around France. Basically two thirds were angry that we published names and were saying that there was an error somewhere (but they were unable to tell us where, and I always answered that the tool was designed to let you show the cases to back the numbers so that you can manually check, so if they find a bug they can report).
Those same judges reminded me of the risk to their reasoning independence if we kept on running the website. One third recognised there was an issue, but were not OK to publish names.
Then the SJA (the main administrative judges union) published several articles regarding SupraLegem. I had to explain myself several times to different judges in different areas.
As you may imagine, my friend’s employer (Google) and my own employer (Deloitte) wanted to avoid being associated with such a project (N.B. the project was done in our free time and indeed our employers were not involved at all, the only thing I did was report to my partner at Deloitte the judges’ reactions day after day).
We did so and refused most interviews from mainstream journals as the few we gave were distorting our intent (they presented us as an aggressive startup with uberization of the legal industry in mind).
Later, a law was voted to make all French case law in open data (Loi République numérique). However it was roughly written and quite impossible to apply.
A project for a second law started (Loi Justice 21e siècle), but this time the government wanted to do things well.
They asked a professor to audit several specialists from the legal world, and to publish a report.
This report is known as the Cadiet report and almost all French state decisions regarding the open data of legal cases are based on this report.
You can find it here: http://www.justice.gouv.fr/publications-10047/rapports-thematiques-10049/remise-du-rapport-sur-lopen-data-des-decisions-de-justice-31165.html
As you can see, I have been interviewed by the commission (I am in the Open Law group), just after the SJA.
And as you can also read is that the main discussion subject of the SJA was to ask to remove judges’ names from the cases. They also added an annex regarding SupraLegem.
Several other judges’ unions agreed with their position (keep in mind that judges’ unions are the main way in France for judges to be represented as they have a duty of discretion and can’t speak publicly directly about any political subjects, such as bias in asylum and deportation).
To sum up, my understanding is that both French supreme courts are not against these statistics (if well done) as they help in some way to normalise the jurisprudence at the level of the country, and in some way it helps with their mission.
But lower courts judges don’t agree with this position, as they want to stay as independent as possible (which may sound like not accountable to some lawyers).
One year and a half ago, I moved to another position and am working now as an applied ML engineer in the R&D team in Lefebvre Sarrut, the second biggest European legal publisher.
Among my current projects I have written some code to anonymise the legal cases through machine learning for my company.
My employer has also a strong opinion in favour of open data. We participate with many concrete actions, one of them being an agreement to make open source our ‘anonymization by ML‘ project.
To conclude, the article 33 is the main consequence of what happened in 2016, at a time where no one had a real experience of ML applied to legal cases.
No one expected these consequences. Since then, the SupraLegem website has been closed and now through my employer I work to help the French administration to correctly anonymise legal cases to accelerate the open source movement which matters the most from now on.
Thx for sharing your experience, very interesting and fuel for thougts
Thank you for this interesting history, M. Benesty. Being familiar with the events that led up to the passage of this law helps to put it in perspective. I confess, however that I still don’t understand the position of proponents of anonymization. No one likes scrutiny. But openness and transparency in the judiciary is critical to the aims of justice.
The question of the independence of judges is significant in France. Publishing statistics associated to names creates a risk of pushing judges to move towards averages and reduces, in some ways, their freedom. Others will say that this is precisely the objective, to reduce what some perceive as a form of arbitrariness.
As always, an excellent, well informed and factual post. Thanks for sharing.
Israel hasnt baaned stattitical info about labour-law rulings and judges bios to employers.
I find the approach of the French lawmaker and the position of judges very inconsistent and unusual in view of international law and generally accepted principles of the rule of law which hold that justice is administered publicly and judgments delivered in open court. It therefore seems totally incongruous to me to allow the public administration of justice and to allow the delivery of judgments in open court and then turn around and impose some form of anonimity. May be in the future judges should simply wear masks when they are administering justice.