When the algorithm doesn’t know the road is 100 kilometres long

There is a particular kind of optimism embedded in most AI governance discourse: a belief that better frameworks, more rigorous benchmarks, and smarter regulation will eventually bring the technology under control. The people writing those frameworks tend to work in Brussels, Washington, or Singapore. The patients they are writing about often live somewhere else entirely.

That gap formed the basis of a recent conversation hosted by HealthTechAsia, bringing together Vishnu Narayan, Contributing Policy Editor & Non-Resident Fellow at HealthTechAsia; Tamiko Eto, founder of TechinHSR and an ethicist whose frameworks are in use at major US institutions; and Farhan Yusuf, a public health professional and AI policy practitioner based in Dar es Salaam, all active members of the Centre for AI and Digital Policy (CAIDP). 

The discussion explored questions of accountability, structural bias, data sovereignty, and the limits of what computational systems can meaningfully understand about the people they are designed to serve. 

The system that doesn’t know the road

Vishnu opened with a challenge that reframes the entire governance debate. The dominant mode of evaluating AI systems, he argued, is technical, measuring capability, scale, and model complexity. What it largely ignores is context.

“AI systems can identify, they can analyse, they can recommend,” he said. “But they do not experience any of this firsthand.” His example was precise: a patient in a remote location diagnosed by an AI system with a cardiovascular condition. The system recommends a specialist and a course of medication. The recommendation may be clinically correct. But the system has no knowledge that the nearest hospital is a hundred kilometres away, or that the local health infrastructure is already operating beyond capacity.

This is not a failure of accuracy. It is a failure of situational awareness that no amount of additional training data is likely to solve, because the relevant variables — distance, infrastructure, social circumstance — are not the kind of thing that scales as a model becomes more complex.

The governance implication is significant. If AI systems are being evaluated primarily on clinical accuracy and not on contextual fitness, then deployment decisions are being made on incomplete evidence. The communities most likely to bear the consequences of that gap are those in the Global South, where the distance between what a system recommends and what is actually possible can be enormous.

Accountability that belongs to no one

Tamiko Eto approached the same problem from a different direction. Her focus is on what happens inside healthcare institutions when AI tools are deployed, and specifically, on what happens when something goes wrong.

The accountability structure of a modern hospital is already complex: ethics committees, vendors, clinicians, procurement teams, administrators. When an AI tool enters that structure, responsibility for its outputs gets distributed across all of them simultaneously. In practice, this means it belongs to none of them.

“When things go wrong,” she said, “every actor can point in different directions. And then the patient who is harmed has nowhere to go, no person to turn to, and no liability pathway or legal pathway to justice.” She described this as predictable — a foreseeable consequence of deploying systems that do not map onto existing liability structures, not an accident.

The second concern she raised was structural bias. AI tools are making clinical decisions about patients who were never in their training data: people from underrepresented populations, rare disease patients, non-English speakers, people whose skin tones were not adequately captured in the datasets used to build diagnostic models. “That,” she said, “is structural injustice baked into these tools that are now widely adopted.”

The third was pace. The speed of adoption is outrunning the capacity of oversight bodies to meaningfully review what is being deployed. The methods to do this properly exist and have existed for decades, she noted. The problem is that prioritising speed has become a competitive and political norm, and responsible deployment has been reframed as an obstacle to innovation rather than a condition of it.

Before AI policy, energy policy

Farhan Yusuf brought the conversation to the ground level, and the ground level, in much of the Global South, starts before AI policy becomes relevant at all.

“Before we even talk about sophisticated AI systems, the first question is: is there electricity or not?” The policy conversation happening in Washington and Brussels, he noted, assumes a baseline of infrastructure that does not exist in many of the communities these systems are intended to reach. You cannot talk about AI governance without first talking about energy policy, device access, and the physical logistics of last-mile delivery.

Beyond infrastructure, there is awareness. In communities with no prior exposure to AI, the technology does not arrive as innovation — it arrives as something unfamiliar and potentially threatening. He described cases where communities receiving drone-delivered medical supplies regarded the technology with suspicion, some refusing to engage with it at all.

A chatbot voice on a phone, he suggested, might provoke a similar response in certain contexts. Awareness is not a communications problem to be solved with a campaign. It is a foundational condition for any deployment to function as intended.

Language compounds this. AI tools are being built predominantly in English and extended to mainstream global languages. But in the communities where healthcare AI is most needed, the relevant languages are often local, dialectal, and entirely absent from training data.

Then there is data governance. Farhan offered an illustration that was striking in its simplicity: a health facility manager in a Global South country, going on leave, writing his username and password for the hospital management information system on the noticeboard so staff would not disturb him. The system contained all patient data. The credentials were publicly visible. This is not an AI governance failure. It is a baseline data security failure — and it is the environment into which AI systems are being introduced.

Data sovereignty and its limits

The final stretch of the conversation turned to data sovereignty: the growing movement, particularly in Africa, for countries to assert control over their own data through local infrastructure. Farhan’s view was nuanced and worth noting: he supports the principle but is wary of how it is being operationalised.

“What I end up hearing in many conversations is that we need to have our own data centre in Africa. And I’m like, how feasible is that — financially, environmentally?” Data sovereignty, he argued, does not require local data centres. It requires the right policies, the right agreements, and a clear-eyed understanding of what a country is giving up versus what it is gaining. Rushing toward infrastructure as a symbol of sovereignty without that analysis may create new vulnerabilities rather than closing existing ones.

Vishnu added an environmental dimension that rarely enters the governance conversation: the physical cost of data infrastructure, water consumption, land use, disruption to ecosystems, falls disproportionately on developing countries that are most actively courting AI investment. The environmental implications of data centres are beginning to surface in public discourse in the United States. In countries with less regulatory capacity and more acute land and water pressures, the consequences may be more severe and less visible.

What this means for governance

What emerged from the conversation was not a counsel of despair but a reframing of where the governance problem actually sits.

The dominant AI governance discourse is focused on the technology: model safety, benchmark performance, regulatory classification, liability allocation between developers and deployers.

These are real problems. But they are upstream of a different set of problems that receive far less attention: problems of infrastructure, awareness, language, institutional capacity, and the fundamental mismatch between where AI policy is written and where AI systems are deployed.

Tamiko’s structural injustice framing, Vishnu’s hundred-kilometre road, and Farhan’s noticeboard password are not edge cases. They are the lived reality for a significant portion of the world’s population. A governance framework that fails to account for them is not merely incomplete; it is solving a fundamentally different problem from the one that actually exists.

As AI systems become increasingly embedded into public infrastructure and healthcare ecosystems, there is also a growing need for more rigorous and context-sensitive governance assessment mechanisms, including stronger regulatory impact assessments that evaluate not only technical performance, but institutional readiness, implementation realities, equity implications, and long-term societal effects.

Over time, this may also require governance conversations to expand beyond immediate deployment concerns toward broader systemic questions, including sustainability and wider public-interest impacts.

Vishnu Narayan is Contributing Policy Editor at HealthTechAsia. Tamiko Eto is founder of TechinHSR. Farhan Yusuf is a public health professional and AI policy practitioner working with institutions like Africa CDC based in Dar es Salaam.