India’s conversation around artificial intelligence (AI) has been mostly about computing power, data centres, Indian-language models, start-ups, and the ambition to become an AI superpower. All that is changing, as a more uncomfortable conversation about existential risk starts.
Finance minister Nirmala Sitharaman recently called AI a double-edged sword, warning that advanced systems could be used for cyberattacks, fraud, manipulation of public opinion, and interference in elections. Her remarks came as safety concerns were being raised inside companies developing “frontier” AI systems—OpenAI, Anthropic and others.
Frontier lab leaders have questioned the speed and direction of the race. In a recent essay, Anthropic chief executive Dario Amodei wrote: “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.” OpenAI chief executive Sam Altman has also spoken of managing the risks of increasingly capable systems.
None of this is news to those tracking AI. Recent cyberattacks involving AI and the resignation of AI safety researchers created a global stir. Anthropic’s Jacob Coxon quit, reportedly two months before he could cash in his company stock, saying he did not want to profit from a direction he considered dangerous. Others elsewhere followed, citing similar safety concerns.
Today, the AI world confronts doomsday fears amid an escalating US-China AI rivalry.
Where does India figure? One key challenge—AI safety—is often treated as an awkward subject in a country eager to demonstrate technological ambition. There is a fear that too much regulation, too many safeguards, and too much discussion of risk could slow innovation and scupper India’s ambition to become an AI power.
But AI safety is not a luxury imported from countries that have already built frontier models. It is part of the infrastructure required for AI at scale.
Ignoring safety may look like removing an obstacle to innovation. It can become an obstacle to adoption. The more autonomous these systems become, the more consequential their failures. AI safety needs more attention and investment.
Most Indians still think of AI as a chatbot. You type something, and it answers. You ask it to draft a letter, summarise a report or write some code. Safety is viewed through consumer harm—scams, deepfakes, job losses.
The global conversation, however, has moved to AI agents going rogue. The distinction matters.
Chatbots, such as ChatGPT, mainly respond to prompts. AI agents, in contrast, can pursue a goal, make decisions, use tools, and take actions with limited human intervention. Cybersecurity risks can multiply.
India may not yet be building frontier models, “but India can certainly become the victim of one,” says Mumbai-based AI and privacy lawyer Dr Prashant Mali. Mali argues that digital sovereignty, therefore, requires AI-safety sovereignty. “The next generation of AI regulation cannot merely regulate what AI says; it must regulate what AI can autonomously do.”
“India has adopted the India AI Governance Guidelines, but much of the proposed safety architecture is presently principle-based and voluntary. The guidelines themselves say these voluntary measures are not legally binding and may later evolve into mandatory requirements. They specifically recognise autonomous AI agents and multi-agent collaboration as tech that may require India to rethink its governance approach,” Mali told this writer.
India’s existing laws can respond after harm. But “the law still lacks clarity on how liability should be divided between the developer, deployer, platform, and user,” says Mali. “As recently as 29/07/2026, the government stated that the India AI Safety Institute was being established to conduct safety research, standards, testing, and evaluation of emerging risks. It is not presently the equivalent of a statutory AI regulator, with powers to license frontier models, order shutdowns, or compensate victims. An India AI Institute can study the fire. India still needs the legal fire code, alarm system, and emergency brake.”
The India AI Safety Institute is important. There is a difference, however, between institution-building and legal protection. India need not choose between ambition and safety, nor look only to parts of the US or Europe that are putting in guardrails.
Important experiments are also under way in Asia. What can India learn from its Asian neighbourhood? Singapore offers one answer. “Singapore is particularly interesting for India, because it is regulating the power given to the agent, not merely its output,” says Mali.
Singapore’s 2026 Model AI Governance Framework for Agentic AI, not yet a binding law, recommends assessing and bounding an agent’s risks before deployment, limiting its autonomy and access to tools and data, and defining points at which human approval is required. In one government deployment, agentic coding assistants were first confined to internal employees, low-risk systems, and no external tools, while logging, monitoring and other safeguards were tested.
“China appears to have understood something regulators elsewhere are only beginning to confront: An AI agent should operate like an employee with restricted access, not like a digital CEO with the master password,” says Mali.
India should now consider mandatory frontier-model evaluation—agentic-AI risk classification, human override and kill-switch requirements, compulsory incident reporting, developer-deployer liability rules, and mandatory AI insurance for high-risk autonomous systems, he adds. Otherwise, our law will remain excellent at investigating the accident after the autonomous machine has already crashed.
The scare has put AI safety on the table. AI will innovate. Like many Indian citizens, my question is, how much authority should an AI system have before a human has to intervene?
Patralekha Chatterjee is a writer and columnist who spends her time in South and Southeast Asia, and looks at modern-day connects between the two adjacent regions. X: @Patralekha2011
