Bias, Transparency, Accountability

You have just learned that the same model can be reported as 90% accurate or as badly flawed, depending on which metric you quote. That is a lot of power over what other people believe — which is why evaluation has ethics of its own.

1The three concerns

While evaluating an AI model, three ethical concerns need to be kept in mind.

1. Bias

Ensure that the evaluation metrics chosen do not result in any kind of bias.

A metric can hide a problem as easily as reveal one. A model tested only on a balanced dataset, reported with accuracy alone, may be quietly failing one group of people entirely — and nothing in the number you published would show it.

2. Transparency

Give an honest explanation of how the chosen evaluation metrics work and produce results, without keeping any information hidden.

Saying “our model is 90% accurate” without saying that the test data was 90% one class is technically true and deeply misleading. Transparency means the reader can check your claim, not just believe it.

3. Accountability

Take responsibility for your choice of metrics and methodology of evaluation, in case any user faces a disadvantage because of your chosen methodology.

If someone is refused a loan, or misdiagnosed, because the model was judged by the wrong measure, that is not the model’s fault. A person chose the measure.

Quoting your own marks

A student who got 95 in one subject and 40 in four others, and tells people “I scored 95”, has not said anything false. Choosing which number to announce is itself a decision that can mislead — and that is exactly the choice an evaluator makes when they pick a metric.

2All three in one table

Concern	What it asks of you
Bias	Make sure the metrics you chose do not produce a biased picture
Transparency	Explain honestly how those metrics work and what they produced, hiding nothing
Accountability	Take responsibility if a user is disadvantaged by the methodology you chose

3How this joins up with Unit 1

The bioethics case study in Unit 1 was a hospital model that sent less care to the patients who needed it most. That model was evaluated. Somebody read a number off it and decided it was working.

The link worth making in an answer

The four principles of bioethics — respect for autonomy, do not harm, ensure maximum benefit and give justice — are about the decisions a model makes. The three concerns here are about the decisions you make while judging it. A model can fail people at either stage, and a fair evaluation is what catches the first kind before deployment.

4Practice

State the three ethical concerns that must be kept in mind while evaluating an AI model. (Practice)

3 marks

▶Show the answer

Bias — ensure the evaluation metrics chosen do not result in any kind of bias.
Transparency — give an honest explanation of how the chosen metrics work and produce their results, without keeping any information hidden.
Accountability — take responsibility for your choice of metrics and methodology, in case a user is disadvantaged because of it.

A company reports that its loan-approval model is “91% accurate”. The test data contained 860 applicants who did not default and 140 who did. Which ethical concern does reporting only that figure raise, and why? (Practice)

2 marks

▶Show the answer

It raises transparency, and through it bias. The dataset is unbalanced, so accuracy on its own flatters the model — and the figure hides how many of the 140 real defaulters it actually caught. An honest report would give precision and recall too, and state the class balance of the test set, so a reader can judge the claim rather than take it on trust.

5Check yourself

Quick Check

Which ethical concern is about explaining honestly how your metrics work, hiding nothing?

Quick Check

Taking responsibility when a user is disadvantaged by your chosen evaluation methodology is:

Quick Check

How many ethical concerns around model evaluation does the syllabus name?

← Previous

🧮 F1 Score, and Choosing a Metric

No-Code AI, and Orange 🧱