Bioethics: A Case Study

Nobody in this story meant any harm. That is what makes it worth studying. A sensible idea, built by capable people, hurt exactly the patients it was supposed to help — because of one quiet decision made early on.

1The project

A company set out to help hospitals look after patients better. It built an AI algorithm to identify individuals at high risk, so that hospitals could put their limited doctors, beds and time where they were needed most. A good goal, honestly meant.

2The problem it caused

What went wrong

Patients from the Western region of a particular area who were given the same risk score by the algorithm were in fact more seriously ill than patients from other regions with that score. Being sicker, they should have been ranked higher. Instead they were ranked the same, and so received less care.

3Why it happened

The algorithm was trained on healthcare expense data as a measure of how ill someone was — on money spent, not on actual physical illness.
It was built in the United States, where less money is spent on healthcare for patients of the Western region than for other patients.

Put those two facts together and the fault is obvious. Less money spent was read by the model as less illness. The model was not broken; it answered the question it was actually asked, which was the wrong question.

Judging hunger by the bill

Imagine deciding who in a queue is hungriest by looking at how much each person spent on food last month. The person with no money spent the least. By that measure they are the least hungry, and they are sent to the back of the queue.

4Where the framework stops it

Now send that same recommendation through the four bioethics gates. Switch between the model as it was built and the model built ethically, and watch where it is stopped.

Four gates on the way to the patient

Each gate is one principle of bioethics. A decision that fails a principle does not get to trade it off against the others — it stops there.

The recommendation cannot get through. It fails Do Not Harm — patients who were sicker were sent less care — and it fails Give Justice, because one region carried all of that harm.

1. Respect for Autonomy

People must be able to decide for themselves, which means they must know how the AI reached its decision.

2. Do Not Harm

Harm to anyone — human or non-human — must be avoided. Where no harmless choice exists, take the path of least harm.

3. Ensure Maximum Benefit

Avoiding harm is not enough. The action should give the greatest possible benefit to everyone involved.

4. Give Justice

Benefits and harms must be shared fairly. No group may be left worse off than another.

5The four principles, applied

1. Respect for autonomy

Users of the algorithm should know how it works. The data the model was trained on should be reproducible and accessible to patients, and where its performance is in doubt, its predictions and data labels should be released. Had that been true, an outsider would have seen “trained on spending” and raised the alarm at once.

2. Do not harm

Patients who were less ill received more intensive care than patients who actually needed help. Implemented as it was, the algorithm would actively harm Western-region patients by recommending healthcare resources away from them. A model must be trained on data that reduces harm for all.

3. Ensure maximum benefit

The solution should be held to clinical practice standards, not merely technological ones. “The software runs and the numbers come out” is a technical standard. “Patients get better” is a clinical one, and it is the one that counts.

4. Give justice

Benefits and harms must be distributed justly. Here one group carried all of the harm. The solution needed to be aware of the social realities of healthcare and to work against those structures, instead of quietly repeating them.

The lesson

Following the bioethical principles would have caught this before the model was used on anybody. That is the entire argument for using an ethical framework while you build, rather than apologising afterwards.

6From the board papers

A school develops an AI system to shortlist students for a competitive scholarship. The algorithm prioritises students who complete online application forms quickly — assuming that faster completion reflects confidence and competence. But students who are not fluent in English, or who have no computer at home, take longer. Many deserving students are unfairly rejected.
(a) Identify two reasons why the algorithm gave biased results.
(b) Mention two bioethics principles that can help solve such a problem and explain how they apply.CBSE sample paper 2025–26

4 marks

▶Show the answer

(a) Two reasons:

The algorithm used speed of form-filling as a measure of ability, which it is not. Speed actually measures English fluency and typing practice.
Students without a computer at home were bound to be slower, so the data itself was unfair to them before the model saw it — exactly the same mistake as judging illness by money spent.

(b) Two principles:

Give justice — benefits and harms must be shared fairly. Here one group of students carried all the harm, so the shortlisting must be checked group by group and the unfair measure replaced.
Do not harm — deserving students losing a scholarship is real harm. The school should choose the path of least harm: judge the application on its content, not on how fast it was typed.

(Respect for autonomy would also earn the mark: students should be told how they are being judged.)

7Check yourself

Quick Check

What was the root cause of the hospital algorithm's bias?

Quick Check

Which principle is broken when one group carries all the harm of an AI system?

Quick Check

What is the main lesson of this case study?

← Previous

🏥 Bioethics

AI, ML and DL 🪆