AI and fairness

Your child asks an assistant about something close to home — where your family came from, a word your grandmother uses, a piece of history that happened to people like you. The answer comes back quickly, fluently, and with no visible hesitation. And it is thin, or slightly off, or written as though the interesting part of the story happened somewhere else. Nothing in it is a lie. It is simply not about you.

What makes this an objection rather than a grievance is that it follows from how the thing was built. A model learns from an enormous pile of text and images that people produced, and that pile is not a cross-section of humanity. It is whoever was publishing, in the languages that dominate the web, in the decades that were digitised, about the topics that got written about a great deal. Whatever was recorded thickly becomes the model's sense of normal. Whatever was recorded thinly comes back thin.

Nobody decided this. There is no room where someone chose to leave your community out, and saying so is not much of a comfort, because the answer your child reads is the same either way. That is what makes the worry hard to wave away: it does not depend on anyone's bad intent, so it does not disappear when everyone involved is well meaning. And it lands harder on a child than on you. You would notice that a summary was missing something. A nine-year-old reads a confident paragraph and files it as what happened.

This is one of the few worries on this site with a real evidence base behind it rather than a forecast. Differences in how these systems treat different groups have been measured repeatedly — by university groups, by journalists, by regulators, and in the companies' own published evaluations. It is not one lab's finding. The same shapes recur across systems, across tasks, and across years.

Those shapes are worth naming concretely. Image generators asked for a doctor, an engineer, or a criminal returned a strikingly narrow range of faces until the companies intervened. Speech recognition has been shown to work less well on some accents and dialects of a language than on others. Systems asked to screen or rank written material have produced different results when a name on the page was changed and nothing else was. Translation into English drops grammatical gender, and translation out of it has to guess — and the guess follows the pattern in the material.

What none of this supports is a number you can carry around. The size of any of these gaps varies by system, by version, by task, and by how it was measured, and a figure from an evaluation three years ago may not describe a current model at all. Several of the specific failures above were found, publicised, and then substantially reduced. The direction is solid and the mechanism is still there; the magnitude is a moving target, and anyone quoting you a single percentage is quoting one study of one system on one day.

It is also established that this has been taken seriously, and a page that skipped over that would not be worth trusting on the rest. Bias evaluation is now an ordinary part of how large models are tested and written up, and the results are reported in more detail than they were. That is a real change. The next two pieces are about why it is not the end of the matter.

The obvious source is the training material, and it is the one most people have heard about. If the pile of text is mostly English, mostly from a handful of countries, and mostly written by people who publish on the internet, then the model's sense of what is usual is theirs. This is not a moral failing of the machine. It is what averaging over a lopsided pile produces.

The second source is discussed far less and matters just as much. After the initial training, a model is tuned using human judgements — people rating which of two answers is better, safer, or more helpful. Those raters are a particular group working to a particular brief, and their sense of a good answer becomes the model's sense of a good answer. A system can be tuned to be polite, cautious, and agreeable in ways that are themselves culturally specific.

The third is the easiest to forget: what anyone thought to ask of it. The things a model does well are the things it was built and evaluated for, and those were chosen by the people building it. A capability nobody thought to test is a capability nobody found missing.

Underneath the argument about fixes sits a harder question that usually gets skipped. Ask a system for a picture of a nurse. Should the results match who is a nurse in your country today? Who is a nurse worldwide? Or who would be, if the profession had not been shaped by a century of assumptions about who does it? Those are three different pictures, and each has been defended by somebody as the fair one.

The same problem appears in text. A summary of a historical event that gives equal weight to every account is not neutral; it is a choice, and frequently a misleading one. A summary that follows the weight of the surviving record inherits whose records survived. There is no version of this with no perspective in it. Somebody's judgement is in the answer whichever way it goes, and “just be neutral” is an instruction that cannot actually be followed.

This is worth knowing because it changes what you can reasonably demand. It is fair to ask that a system not be confidently thin about your family, and fair to ask to be told when an answer is one perspective among several. Asking for an answer with no perspective in it is asking for something that does not exist — and a company that promises it is selling you the appearance of it rather than the thing.

Whether these systems narrow the range of what gets said or widen it is genuinely unknown, and both are plausible. A tool that answers in a hundred languages puts more within reach of more people than anything before it did. A tool that quietly standardises every piece of writing towards one norm does the opposite. Both are happening at once, and there is no measurement that nets them out.

The sharpest version of the question is what happens to a language or a tradition that is thin in the training material. If the material stays thin, the model stays thin about it. If a community produces more material in order to fix that, then part of why the material exists is to feed a system — which is a strange thing to ask of a culture, and not obviously a good one. Nobody has a comfortable answer to that, and the people most affected by it were not the ones who set the question.

The evaluations are contested ground too. A benchmark measures the gaps somebody thought to look for, in the languages somebody thought to test, and the results are usually reported by the organisation with the most at stake in them. That does not make the numbers false. It does mean that “this model scores well on fairness evaluations” is a much narrower claim than it sounds, and worth reading as the narrow claim it is.

A parent who looks at all of this and decides the burden of proof sits with the tool has read the situation correctly rather than dodged it. It is entirely reasonable to keep these systems away from the subjects where your family's own account is the one that matters, and to let them help with the arithmetic homework instead. That is not a refusal to engage with the technology. It is a decision about what it is good enough for, which is a judgement you are better placed to make than anyone selling it.