Imagine a company receiving thousands of applications for a hundred jobs. Software ranks them, and a recruitment team interviews the people near the top. The director retains the final decision, as the contract with the supplier promises. She can also ask for any rejected application to be retrieved. The difficulty is knowing which ones to retrieve: her screen shows the selected candidates, and hiring must be completed by Friday. The tool has not signed a single rejection. It has nevertheless chosen whom the person responsible for deciding will get to know.
This is a hypothetical scene. Its mechanism requires neither a disobedient robot nor an intelligence capable of replacing humanity. A system that orders information and an organisation unable to review everything below the line are enough. Power emerges from their combination. Calling the software an assistant describes a contractual intention; discovering how much it decides means following its outputs to the opportunities they open or close.
The regulatory question is often phrased differently: will there be a human in the loop? That is a useful inventory question and a poor test of control. A person can be present, hold a password, receive training and sign every decision. They may still lack the means to change a conclusion produced by the system even in the face of a well-founded objection. Presence is easy to document. The ability to exercise judgement needs a closer investigation of the job.
The decision starts before the decision
At our hypothetical company, selection involves more than choosing among interviewees. It also involves determining what counts as experience, how employment gaps are interpreted and how many applications reach the team. Whoever configures those operations influences the result without ever meeting the applicants. The director receives a set of possibilities already shaped by criteria, some explicit and others learned from earlier hiring decisions.
Suppose the model predicts very accurately which people resemble employees who stayed with the company for a long time. That regularity may interest the employer. It does not, by itself, establish who would perform the advertised job best or which differences may legitimately be considered. Retention, performance and fair access are related but distinct questions. Choosing one as the measure of success pushes the others aside unless the procedure finds a way to include them.
Nor must the ranking be entirely automatic to have exclusionary effects. A team can carefully review the first two hundred applications and effectively discard the rest because time runs out. Human review will be real within the visible group. The earlier filter will still determine entry to that group. Counting interviews or signatures cannot reveal that boundary; the investigation must also include the applications that never reached a desk.
The distinction between formal and effective authority explains this without treating an algorithm as a legal person. The former concerns the right or power to decide. The latter concerns control over the substance of the decision. Rodrigo Castillo’s account of algorithmic delegation uses that separation to examine the function an output acquires within a practice. The organisation chart can remain unchanged while the selection of decisive evidence moves elsewhere.
Disagreement has a price
Now imagine a recruiter spotting a badly ranked application. To accept it, she must open another application, obtain the full file, justify an exception and await authorisation. Following the recommendation takes one click. Her performance assessment counts completed recruitment processes. The company has given her freedom to disagree and simultaneously sent her a bill for exercising it.
That difference does not establish that the recruiter blindly trusts machines. She may understand the program’s limitations, recognise the candidate’s value and still have no practicable way forward. Training about automation bias would be insufficient to fix a problem of permissions or workload. Learning to be sceptical does not open a restricted file or supply the hours required to investigate it. The remedy must reach the constraint actually preventing action.
Castillo calls this distribution of the burdens of accepting and challenging a result an asymmetry of ratification. The concept identifies an organisational choice often hidden beneath psychological debates about trust. If acceptance shields an employee from individual criticism while departure exposes them, the difference grows. The company need not instruct everyone to obey the system: it can organise the work so that disagreement becomes exceptionally difficult.
Requiring reasons for an exception has a legitimate purpose. A supervisor can also favour acquaintances, repeat stereotypes or replace evidence with intuition. The remedy is not to abolish justification for disagreement. It is to establish whether those requirements allow a well-founded objection to proceed, and whether following the output receives comparable scrutiny when there are signs of error. A rule enabling arbitrary decisions does not improve merely because a person makes them.
Time savings create an important choice. They can fund better review of difficult cases or be converted entirely into a higher quota of applications. In the latter scenario, the model improves while the procedure loses human capability. Observed productivity cannot distinguish these paths: both may deliver more decisions per hour. Finding out which protects affected people better requires looking at the errors recognised and those left comfortably outside the performance indicators.
A disagreement rate proves little
An organisation might try to fix this by setting a minimum share of human decisions that contradict the system. The target would be easy to count and easy to meet badly. Reviewers could change correct cases to reach it without becoming better at detecting incorrect ones. If the tool improves, a falling disagreement rate may be desirable. Elsewhere, the same fall may show that reviewers have stopped looking.
Useful information separates what an employee recognises from how often they act. Can they detect evidence that makes a recommendation inappropriate? Can they distinguish it from an irrelevant difference? Can they make that distinction affect the outcome? Lowering the threshold for intervention may increase both sound and mistaken corrections. A company measuring interventions alone confuses activity with capability, without learning whether the extra work helps anyone.
In our recruitment example, adequate review might retrieve applicants whose relevant experience was obscured by the format of their CV. It could also confirm exclusion when an objection does not alter the job requirements. Quality does not mean consistently favouring either the claimant or the software. It means investigating the disputed point under admissible criteria and deciding according to what that examination supports. Agreement can be the result of careful review as readily as disagreement.
The comparison needs a sample of cases the model itself regards as ordinary. If every check starts with its alerts, the system helps choose where it will be watched. Its blind spots may escape the sample. Reviewing routine decisions can expose errors that do not announce themselves; reviewing complaints can reveal circumstances known only to the affected person. The two routes produce different information and should compete for resources with their purposes made explicit.
Correct data can support a bad inference
Suppose a candidate has a two-year employment gap. The recorded fact is true. She explains that she worked independently during that period and supplies evidence. The company replies by displaying the recorded date: it finds no error to correct. The software works and the database matches the CV. The dispute, however, concerns the inference drawn from that date. An absence of salaried employment need not mean an absence of experience.
This case requires relating information to a selection criterion. A technical explanation might show how strongly the gap influenced the score. That would help locate the problem, but would not determine whether that weight was appropriate for the job. The person who understands the model and the person who understands the work may contribute different knowledge. The organisation needs to bring those contributions together in a decision that answers the objection.
Universal access to source code would not, by itself, perform that task. It may be essential for particular investigators, but a candidate need not understand millions of parameters to present omitted experience. She needs to know what issue to contest, where to provide evidence and what may happen if she is right. Extensive technical disclosure can coexist with a closed institutional door. More information is useful only if someone can use it for the question at hand.
An explanation must therefore fit its recipient and the claim being made. Correcting a date requires locating a datum; challenging an inference requires identifying the relationship asserted; disputing a rule requires justification for its use. Confusing those operations produces bulky, unhelpful answers. Transparency matters when it enables people to understand and act upon something that would otherwise remain beyond the reach of those with a reason to challenge it.
Europe looks at what the score does
There is a concrete legal precedent. In December 2023, the Court of Justice of the European Union considered SCHUFA. It held that a credit score can constitute an automated individual decision when banks give it a determining role in granting credit. Exceptions and the remaining conditions were for the national court to assess. The conclusion does not prohibit all scoring; its scope cannot be settled solely by identifying who formally issues the final response. CJEU press release.
The institutional lesson extends beyond that financial product, although its legal consequences do not automatically extend to every sector. Information can be presented as support and function as a decisive condition. Discovering its influence requires investigating what recipients do with it. A contract reserving discretion is a poor description of a procedure if nobody has a practicable alternative. A formal power matters partly through the conditions under which it can be exercised.
Article 14 of the European AI Act provides for oversight proportionate to risk, autonomy and context. It includes understanding limitations, interpreting outputs and being able to depart from them or safely interrupt the system. The tool’s design must enable intervention; appointing a supervisor does not complete the task. Article 14 on the Commission’s official service.
Timing matters. As of 25 September 2026, high-risk obligations have different application dates: 2 December 2027 for Annex III uses and 2 August 2028 for systems embedded in Annex I products. The AI Omnibus entered into force on 27 July 2026. Presenting those obligations as universally applicable from August this year would erase that transition. European Commission timeline.
A red button does not reorganise a company
Interrupting an operation can prevent immediate harm. Retrieving affected applications afterwards is a different capability. A company might turn off its recruitment filter and still not know who was excluded, what information was used or which interviews should be repeated. The button solves one part of the problem; records and review powers solve others. Buying the former without organising the latter produces conspicuous safety and uncertain correction.
Stopping also has a cost. If the system processes a workload that previously caused long delays, suspending it may harm people awaiting a response. The alternative must be evaluated as it exists, with its errors and limitations. An imaginary human procedure, perfectly diligent and supplied with unlimited time, is not an available replacement. Control requires deciding how the service will continue while the fault is investigated, including what will happen to cases already in progress.
A gradual response may be more useful: suspend a particular function, narrow the scope of use, refer certain cases to specialists or temporarily return to an established procedure. No option works for every system. What matters is knowing which operation can be separated, how doing so affects other applications and who may choose that course. Such preparation needs to precede the day when stopping becomes urgent. A nominal escape route may otherwise prove unusable when it is needed.
Distributed control also needs deadlines. The recruiter may identify the problem on Monday, the technical team reply on Wednesday and the legal unit receive it after hiring closes. Everyone may have performed their task professionally; the candidate will have lost the opportunity. A safeguard arriving outside the useful time window can document harm without preventing it. The objection’s complete journey belongs in the assessment, alongside the quality of each intermediate response.
Purchasing decides how much control remains
Before a score appears on a screen, someone decides which product to buy and on what terms. That stage determines whether the organisation can learn about changes, retrieve information and obtain the supplier’s cooperation. If it purchases outputs alone, it will discover the limits of that exchange when one needs investigation. The visible price may omit the future cost of keeping a decision open to meaningful review.
A well-defined commission distinguishes ordering documents, recommending a shortlist and excluding applicants. A tool bought for the first function may end up performing the third as workloads rise. The change can occur without a new version of the software. Recording actual use therefore matters alongside technical modifications. The question is what consequences the ranking now produces and whether the means of control expanded with them. A familiar interface can conceal a substantial change of function.
Exit conditions matter for the same reason. A company that can theoretically switch suppliers but would lose its records by doing so possesses an expensive freedom. Keeping usable formats, duties of cooperation and reasonable operational continuity reduces that dependency. These are contracting and organisational choices whose specific necessity depends on the service, rather than a universal catalogue of legal requirements. Their value becomes visible when the organisation needs an option besides continuing with the existing supplier.
The purchaser needs someone able to interrogate what it receives. It does not have to rebuild every model from scratch. It does need to recognise an evaluation covering a different population, task or configuration. The supplier’s expertise justifies seeking its help without removing the need to understand the commission’s scope. The more decisive the supplier’s contribution becomes, the less reasonable it is to retain that understanding merely as an optional consultation.
Reviewers need review too
Giving specialists power to make exceptions can correct decisions and concentrate influence in a new place. If only they understand the procedure and nobody can examine their criteria, dependency changes direction without disappearing. The aim is not to replace an unchallengeable authority with one that has a face. It is to retain a route through which a competent body can examine a decision’s reasons and consequences.
A review record should therefore identify what was challenged, which evidence was considered and what changed. This does not require an encyclopaedic explanation in every file. A simple case can receive a short answer. An exception revealing a shared problem needs more: checking whether other candidates were affected and deciding who will investigate that possibility. Length should follow the question, rather than become a requirement that consumes the time intended for answering it.
The control team’s incentives matter as well. Rewarding only an absence of recorded incidents may make avoiding investigation attractive. Rewarding only the number of findings may encourage exaggeration of every anomaly. Useful assessment combines outcomes, access to information and the ability to correct, with attention to context. An absence of complaints can signify sound operation or a channel nobody manages to use. It cannot decide between those possibilities without further evidence.
The affected person’s account is particularly valuable evidence, although not infallible. It may reveal experience absent from the data or a consequence the purchaser overlooked. Admitting that information does not require accepting every claim. It requires deciding who can investigate and what effect the information would have if correct. A complaints channel becomes more than a mailbox when an answer can change the course of the matter, while a change can still help the person concerned.
A capability can disappear through disuse
Suppose the team retains every permission but spends two years working only with applications recommended by the software. Its members may become excellent interviewers within that group while losing familiarity with profiles the filter omits. When the labour market changes, they may lack recent reference points for recognising a new qualification. The knowledge needed for review cannot be preserved simply by archiving a manual. It needs opportunities to compare decisions with cases that challenge the usual selection.
Such deterioration is not inevitable. Automation can free time for specialisation, comparisons and the study of exceptions. The difference depends on how the savings are used. Rotating tasks, reviewing samples or following retrieved candidates can preserve useful knowledge when they respond to identified problems. Making everyone mechanically repeat the program’s work would keep people busy without ensuring that they learn what is needed to govern it. Familiarity with repetition is not necessarily knowledge of limits.
Organisational memory creates another difficulty. If the employee who configured the filter leaves and decisions survive only in their inbox, a successor can inherit a functioning tool whose purpose is poorly understood. Documenting why a criterion was chosen helps make it contestable when conditions change. That account should distinguish a job requirement from a convenience of the software. The former may remain necessary; the latter may have disappeared in an update while continuing to exclude people through inertia.
Purchasing artificial capability may therefore require complementary investment in human knowledge. Its purpose is not to compete with the machine in every calculation, but to formulate the task, recognise the limits of evidence and sustain a response when someone reaches them. Treating that investment as an unrelated expense makes profitability appear greater because necessary work has been left unfunded. The problem will return, more expensive and with someone already harmed, when the first important exception reaches the door.
The signature must come with means
Maintaining the ability to answer costs money, consumes time and may reduce some promised savings. That does not make control a defeat for automation. It reveals the cost of using it in an activity where excluded people matter alongside processed applications. Comparing an offer that includes those capabilities with one that shifts their absence onto outsiders requires making the difference visible. Otherwise the apparently cheaper product may simply leave someone else to bear an unpriced consequence.
Sound design can preserve many automatic decisions. Where an operation is narrow, verifiable and reversible, systematically duplicating it may contribute little. Conversely, an apparently minor filter may deserve close attention when it determines access to a scarce opportunity. The quantity of human intervention cannot rank those situations by itself. Function, consequences and the means of correction provide a more useful comparison than the number of people standing between a system and its output.
The company in the opening scene does not need its director to read thousands of CVs every night. It needs to know which candidates she never sees, how errors outside the selected group can surface and who can correct them while the opportunity remains open. Without those means, the final signature certifies that a person was present. It does not yet demonstrate that the person remained in command.