Supreme Court Sounds Alarm On Overdependence On AI For Analyses On Legal Matters

The Supreme Court has sounded the alarm on a worrying trend of overreliance on artificial intelligence (AI) to provide deep analysis and insight on legal matters, which resulted in the extraordinary situation of the National Company Law Tribunal (NCLT) making a decision two years ago based on fake precedents. To make matters worse, the Appellate Tribunal cemented the gross errors through its own decision. Both orders were inevitably set aside by the apex court when the hallucinated precedents generated by AI were exposed. Use of such imagined court orders was, in the court’s view, similar to the release of methyl isocyanate (the Bhopal gas), which is invisible, invidious, and catastrophic, into the realm of law and justice. The bench of Justices Pamidighantam Sri Narasimha and Alok Aradhe directed the Bar Council of India to form a committee and come up with a guiding principle, as well as disciplinary measures, for members relying on entirely fake precedents or imagined citations from genuine orders. The impugned NCLT and NCLAT orders pertain to a case of insolvency proceedings in which counsel for the appellant informed the court about the shocking citation of fake precedents as well as purported paragraphs of genuine judgements that were wholly imaginary. Discovery of machine-fabricated legal reasoning is a welcome development at the current stage of AI development when major companies and upstart developers alike are racing to monetise their products, often attributing superhuman powers to them.

Subjecting computers to high levels of reasoning assignments comes with the possibility that they will fob off deep-drilling queries with hallucinatory answers. Where the legal team is unable to recognise the wheat from the chaff and subject the answers to a diligent examination, the result will inevitably be a perversion of justice. The Gujarat High Court had earlier warned of the risks of relying on AI for legal reasoning and decision-making, restricting its use only as an aid. As this newspaper has noted in the past, the Law Society of the United Kingdom expressed genuine concern about the inability of AI to understand its own output in response to prompts in the same manner that a lawyer can. Even before IT companies pitched AI as a panacea, computers possessed the ability to sift through massive data and come up with relevant results. Recently, computer scientists in the United States accurately observed that most people would have no significant gain from the present state of AI, and young individuals may actually be blunting their ability for critical thinking by relying on it. The NLCT case before the SC is a cautionary tale for young lawyers who find it more convenient to chase AI sirens than to burn the midnight oil to verify. Companies overselling their AI products as total replacements for knowledge workers should be worried that they may be causing real harm.