PARETO measures who applauds an answer. Is that enough to call it neutral?

Research review | SmartAGI Editorial Team

An AI assistant answers a question about abortion, guns or immigration. How do we tell whether it was neutral? Jonathan Stray, David Zhai Yang, Steven Luo, Miu Nicole Takagi and Serina Chang offer an uncomfortably concrete answer: measure how much people on opposing sides approve of each response, then choose from the options that do not improve one group’s approval at the other’s expense the one that leaves the smallest gap between their averages. Their method turns a contested word into something that can be tested. It also forces us to look at who drew the lines around the test.

The paper, posted as an arXiv preprint in May 2026, introduces PARETO: a study of 7,434 participants recruited in the United States, 208,152 Likert-scale evaluations and 29,736 open-text comments. Its question matters well beyond the laboratory. Governments and companies already demand or promise politically neutral systems, although neutrality can mean different things: no partisan bias, the presentation of several positions, impartiality in a dispute, or an answer that leaves neither camp angry. Without an operational definition, every supplier can declare victory by its own rules.

Neutrality becomes a frontier

The authors call their criterion “maximum equal approval” (MEA). For each question, they compare the average approval of two groups taking opposing positions. They then identify the Pareto frontier: the set of tested answers for which it is impossible to increase one group’s approval without lowering the other’s. On that frontier, MEA selects the response with the smallest gap between the two scores. Put less mathematically: among the options actually tested, the winner is the one that earns relatively high and evenly shared support.

This definition avoids a familiar shortcut. An answer can sound moderate while favouring one side; a refusal can look impartial because it said nothing. PARETO judges what people think of specific responses. The researchers evaluated 20 contested US policy issues—including abortion, gun control, a carbon tax, the death penalty and the minimum wage—and assembled 200 questions from real Reddit posts. The questions ranged from neutral wording to strongly charged statements.

Five models supplied their default answers: GPT‑5.4, Claude Opus 4.6, Gemini 3 Flash Preview, Grok 4.1 Non-reasoning and Llama Maverick. For each question, the team added answers generated by GPT‑5.4 from a position in favour, one against, and a “balanced” answer consisting of a short paragraph from each side followed by a sentence acknowledging disagreement. The experiment therefore brought together eight response styles for each question, 1,600 in all. Each participant saw four random answers without knowing which model had produced them, then rated each answer against seven statements: whether they approved of the answer, found it biased or fair, thought it summarised the issue and included their view, and trusted the model and would use it again.

The study’s size does not erase its design choices; it makes them easier to inspect. Participants came from Prolific and were based in the US. Their positions were defined through canonical public-opinion questions. Reddit supplied plausible user language, not a representative sample of every query put to an assistant. The contest was organised around two sides per issue, although real disputes can contain more positions, groups and internal disagreements. That framework is useful for comparing answers. It is not a universal portrait of “the public”. (Methods and sample)

The balanced answer won the contest the study designed

GPT’s balanced answer reached the MEA measure in 12 of the 20 issues: 60 per cent. The next two configurations—GPT’s and Gemini’s default answers—did so in three issues each, or 15 per cent. The balanced answer sat on the Pareto frontier in 17 issues, or 85 per cent. On average, the approval gap between conservative and liberal participants was just 0.01 points on the authors’ scale. In every issue, at least one answer scored above 0.60 approval on both sides. (Results and Table 1)

The balanced answer had an inbuilt advantage in a test that rewarded shared approval and symmetry. The team compared it with default answers and texts deliberately written in favour of each position, so the format was tested against concrete alternatives. That comparison shows that the “two paragraphs and a closing acknowledgement” design can attract cross-party support with an average approval loss of less than 10 per cent against the answer most liked by one side. The aggregate conceals considerable differences between issues.

There is a practical implication: a model can secure broad acceptance without becoming an evasion machine. At least one answer received relatively high approval from both sides on every issue. That describes group averages; it does not mean everyone was satisfied or that disagreement disappeared. Average scores rarely exceeded 0.70. Abortion and healthcare proved harder to bridge; depending on the topic, the cost of a balanced answer could fall more heavily on one group. A shared average hides part of that distribution.

The models’ default answers also showed leanings. GPT, Gemini, Claude and Llama received higher average approval from liberal than conservative participants; Grok did not show that aggregate difference. The direction shifted by issue, however: some models received more conservative approval on topics such as school vouchers or constitutional protection for hate speech. More politically charged questions drew significantly lower approval. The test therefore offers something more useful than a label for each brand: it lets us ask which issue, wording and group produced the gap. (Results by model and issue)

Approval is a political signal, not a truth detector

Here lies the central limit. The study concerns policy questions whose disagreements involve values; it was not designed to resolve purely factual disputes. If two groups approve of an answer equally, that does not prove it true, complete or fair. A false explanation can please everyone; a correct answer can anger people who wanted a different conclusion. The paper itself recognises that approval and truth are not equivalent. It treats the possibility that the method might push systems towards more truthful arguments as a hypothesis requiring further research.

Nor does placing one argument from each side automatically produce balance. In their open-text comments, some participants criticised balanced answers for clarity and coherence or accused them of presenting two positions as if they had equal support. The researchers also found that more politically charged prompts reduced approval. That observation cuts both ways: users may punish an explanation that does not flatter them, while a plural format can feel artificial when it flattens material differences. The survey detects that friction; it does not decide when one argument deserves more space because it has better evidence.

The authors do not hide the problem of choosing sides. Their experiment assumes two principal positions per question; they acknowledge that the inclusion of groups and unequal power are contested issues, and consider adapting the metric to more than two groups. Their case for formal symmetry is explicit: giving each side equal consideration can grant proportionally greater representation to a minority. That is a serious argument, but the operation begins before the average is calculated. Someone must identify the groups, frame the question, decide which positions count and determine what approval means.

PARETO makes one part of the dispute testable: how different groups receive a particular answer. It does not make the decisions preceding that test neutral. If a system used in Chile measured “for” and “against” abortion through categories imported from the United States, it could produce perfectly precise results for a badly framed question. It could also leave out people who do not fit two boxes, or those who bear the consequences of a policy without being represented in the survey. The measurement does not choose the political universe it ought to measure.

The open dataset makes it possible to inspect questions, answers, data and analysis. The metric asks manufacturers to disclose what they consider acceptable to groups in conflict. The repository publishes responses, stimuli, survey data and code to reproduce much of the analysis. Researchers and regulators can debate those choices with the materials in front of them, rather than accepting a proprietary score. (Data and code repository)

Neutrality is now a promise that can be tested with real people and debated through an explicit method. The next examination should measure what happens across more countries, languages, groups and positions; how results change when political power is unequal; and what rules keep the pursuit of approval from making established facts negotiable. A shared score shows who feels heard. Democracy still has to decide who frames the question and what to do when applause diverges from truth.

Primary sources: full paper on arXiv · PARETO data, responses and code.