Do Androids Dream of Biased Judges? Self-fulfilling Prophecies in AI Litigation Prediction

QLP uses the dataset of previous litigation cases to learn the correlations between case features and target outcomes. The underlying variables are manifold: who the judge in the case is; the type of legal arguments used; which precedents are cited by a particular judge; the weight of evidence attributed to specific pieces of evidence by the judge, even whether the judge is prone to certain cognitive biases (such as the hindsight bias, the conjunction fallacy, the anchoring effect, framing bias, and confirmation bias). The legal counsel can then use the result of this calculation to make predictions concerning the case and make strategic decisions within the adjudication process.

As a result, QLP can identify the personal tendencies of particular judges that would otherwise be invisible to lawyers. Nevertheless, there is growing concern that AI systems learn and exaggerate human cognitive biases.

One way bias can creep into algorithms is when AI systems learn to make data-based decisions, including biased human choices. If the algorithm is based on biased human decisions, then inevitably, the AI will also reflect such biases, and most likely, it will even exaggerate these biases by holding them true for its future decisions and our outcome predictions.

Even though our decisions based on the AI’s conclusions seem reasonable, they might be profoundly flawed if we put them in a broader perspective. Assuming that our super recognizers’ conclusion is on point and has pinned a judge’s bias right, and we decide not to fight for our case and indirectly against this bias, we have just put one more brick into the wall of the justification mechanism. Worse yet, when we (or someone else) find ourselves in a similar decisive situation with the same judge, the biased pattern of the judge will be even more visible to our super-recognizers, and they will assign more significant weight to this variable when computing next time. Thus, ultimately, we will find ourselves in a never-ending cycle of our AI reinforcing the given bias of the judge.

When we buy into the idea that our case has already been decided by a biased judge, disregarding our legal arguments, we profoundly compromise our right to participate in the adjudication process. If we make decisions in this spirit, our super-recognizer’s predictions will become self-fulfilling prophecies. The AI’s expectation will lead to its confirmation.

We must also remind ourselves that even though the algorithms can outline significant patterns specific to judges, they don’t necessarily refer to existing biases. Based on the previous, one should not overestimate the reported results of the algorithms.

Another concern is that applying QLP might undermine the fairness of the litigation process. This unfairness might stem from violating the principle of equality of arms. If predictive justice becomes increasingly influential, unequal access to these tools will enhance the advantage that the wealthier and more powerful litigation parties have over those who cannot access QLP tools.

It is noteworthy that, for the above reasons, the French legislature introduced a controversial rule in 2019 that bans the use of QLP for outlining a judge’s patterns and predicting case outcomes based on the prior behavior of the judge. Although the somewhat harsh legislation caused a severe backlash and was criticized by many scholars and free speech organizations, it can, in some ways, be considered an extension of the European concerns around the ethical use of AI. Hopefully, the soon-to-be-adopted and much-awaited AI Act of the EU will clarify the European rule maker’s stance on this issue.

This article was first published in the Budapest Business Journal print issue of June 2, 2023.