When institutions first adopt artificial intelligence, attention naturally concentrates on technical capability. Can the system classify accurately enough? Can it produce useful forecasts? Can it summarise documents, detect anomalies or generate plausible recommendations? These questions matter because a system that cannot perform its technical function cannot contribute much to institutional cognition. Yet technical capability is only one part of the problem. An AI system can perform well, produce relevant outputs and even outperform existing methods in particular tasks while contributing surprisingly little to the way the institution actually thinks. The reason is simple: capability only becomes institutional when people are prepared to rely on it.
This creates a different kind of bottleneck. The constraint may no longer lie in what the machine can produce, but in whether its outputs are sufficiently trusted to enter operative cognition. A recommendation that nobody uses has little institutional effect. A forecast that is routinely ignored does not meaningfully extend anticipatory capacity. A classification system that staff repeatedly bypass may exist technically while remaining almost absent from institutional practice. In such cases, TECHNICAL CAPABILITY ≠ INSTITUTIONAL USABILITY.
The distinction matters because institutional cognition is relational. Knowledge does not become organisationally consequential merely because it exists somewhere inside a system. It has to move. An AI output must cross an interface into the attention, interpretation and judgement of the people who can act on it. If that transfer fails, the institution may possess a sophisticated model without possessing the corresponding cognitive capability in practice.
Trust is one of the mechanisms that allows this transfer to occur.
Suppose a public agency deploys a model that predicts which cases are most likely to require urgent intervention. The system may have been extensively tested and may perform significantly better than a previous prioritisation method. Yet if caseworkers believe the model frequently misunderstands context, they may continue to rely primarily on their own assessments. Formally, the system has been integrated into the workflow. Cognitively, its contribution remains marginal.
The institution therefore faces a paradox: the AI system can be technically competent and operationally present while still being cognitively absent.
This is not necessarily irrational. LOW TRUST ≠ IRRATIONALITY AUTOMATICALLY. People may distrust a system because previous tools failed, because its outputs are difficult to interpret or because their professional experience reveals cases that the model handles poorly. They may have learned that nominally accurate systems still make costly mistakes at the boundaries of their competence. They may also be responding to unclear responsibility: if the AI recommendation is wrong but the human remains accountable, caution can be a perfectly understandable response.
Trust cannot therefore be created simply by instructing people to trust more.
Nor should it be.
TRUST ≠ BLIND ACCEPTANCE. An institution in which staff automatically follow AI outputs would not necessarily be more cognitively capable than one in which those outputs are ignored. It may simply have replaced one bottleneck with another. Excessive reliance can suppress independent judgement, conceal errors and create new forms of dependency. The objective is not maximum trust, but enough justified reliance for useful machine-generated information to participate in institutional reasoning.
This reveals a subtle distinction between accuracy and reliance. RELIANCE ≠ ACCURACY. A highly accurate system may attract little reliance if users do not understand it, do not trust the conditions under which it was evaluated or have experienced failures that matter more to them than aggregate performance statistics suggest. Conversely, people may rely heavily on a system whose actual reliability is modest because it is convenient, familiar or institutionally endorsed.
Trust is therefore not a simple reflection of technical quality. It is a relationship between perceived reliability, experience, accountability, institutional incentives and the consequences of being wrong.
That relationship can become the limiting factor in AI adoption. Early AI programmes often assume that once a model reaches an acceptable performance threshold, the main problem has been solved. Implementation then becomes a matter of integration, training and workflow redesign. But a system can be technically integrated while remaining cognitively peripheral if the people expected to use it consistently discount its contribution.
In such environments, the organisation may respond by improving the model again. Accuracy is increased, interfaces are redesigned, additional data are collected and technical robustness is strengthened. These improvements can be valuable, yet they may fail to resolve the actual constraint if the remaining bottleneck lies elsewhere. A system that moves from 90 to 94 per cent accuracy may still be ignored if users lack confidence in when the four per cent of failures will occur.
The problem is therefore not always more capability. Sometimes it is the absence of a workable relationship between capability and reliance.
This becomes particularly visible when AI systems provide recommendations rather than deterministic outputs. Recommendations occupy an ambiguous cognitive position. They are neither raw information nor authorised decisions. They invite judgement. But if users do not know how much weight to give them, the recommendation can oscillate between irrelevance and overreliance.
A clinician, inspector, policy analyst or caseworker receiving an AI recommendation must decide, explicitly or implicitly, whether it deserves to influence their own judgement. That decision is repeated every time the system produces an output. Over time, users develop informal reliance patterns. They learn which outputs they tend to accept, which they check, which they ignore and which circumstances make them sceptical. These behavioural patterns can matter as much as the model’s formal operating specification.
An institution therefore cannot understand an AI system merely by measuring model performance. It must also understand how the model is actually relied upon.
A system used only when its recommendation confirms an existing human expectation may contribute very little new cognition. A system trusted only in simple cases may provide useful efficiency but little support in difficult ones. A model whose warnings are routinely overridden may technically detect risks while failing to change behaviour. Conversely, an output that users accept almost automatically may acquire far more cognitive influence than its formal advisory status suggests.
This is why FORMAL AVAILABILITY ≠ OPERATIVE COGNITIVE CONTRIBUTION.
The distinction also explains why trust and verification should not be collapsed into the same problem. An institution may have excellent mechanisms for checking AI outputs and still find that staff rarely use them because they do not see the system as sufficiently useful or reliable. It may also encounter the opposite condition: users may trust the system readily even though the organisation has limited capacity to verify what it produces. RELIANCE ≠ VERIFICATION. Trust concerns whether the output is allowed to influence cognition. Verification concerns whether that influence can be checked against adequate evidence and review.
This difference becomes especially important when systems operate under uncertainty. AI tools rarely behave with identical reliability across every context. A model may work extremely well on common cases and poorly on unusual ones. A language model may be useful for summarisation but unreliable for precise factual retrieval. A forecasting system may perform strongly under stable conditions but degrade after environmental change. Trust therefore becomes conditional rather than absolute.
Users need some sense of where reliance is appropriate. If that understanding is absent, two dysfunctional responses become likely. One is blanket scepticism: the system is treated as fundamentally untrustworthy even where it performs well. The other is blanket confidence: outputs are accepted beyond the conditions in which they deserve reliance.
Both responses reduce institutional intelligence, though in opposite directions.
Blanket scepticism wastes potentially useful capability. Blanket confidence converts assistance into hidden dependence.
The most difficult situation lies between them, because organisations often possess no explicit language for discussing differentiated reliance. People may say they “trust” or “do not trust” an AI system as if trust were a binary property. In practice, institutional reliance is almost always conditional. A user may trust a model to identify obvious anomalies but not borderline cases, to summarise large document sets but not interpret legal implications, or to propose options but not determine which one should be chosen.
Recognising these distinctions turns trust from an attitude into an institutional variable.
The public sector gives this variable particular importance because reliance interacts with accountability. Public officials often remain answerable for decisions even when AI contributes substantially to them. This creates a rational asymmetry: the machine may influence the recommendation, but the human bears the consequences of acting on it. If responsibility is not aligned with the degree of cognitive dependence, users may either reject the system defensively or rely on it while maintaining only nominal responsibility.
Neither outcome is desirable.
A cognitively mature institution therefore needs to know not only whether its AI systems are accurate, but whether useful outputs are actually entering judgement at an appropriate rate. If technically strong recommendations are systematically ignored, the organisation should not immediately assume that users are resistant to innovation. Their reluctance may contain information about the interface. Perhaps the system is difficult to interpret. Perhaps users cannot identify when it is likely to fail. Perhaps earlier errors damaged credibility. Perhaps the incentives attached to accountability make reliance unattractive. Perhaps the technology solves a problem that the people doing the work do not recognise as their actual problem.
Trust deficits can therefore be diagnostic.
They can reveal misalignment between technical capability and institutional reality.
At the same time, high trust should not automatically be celebrated. HIGH TRUST ≠ GOOD GOVERNANCE AUTOMATICALLY. An organisation in which AI recommendations are accepted without sufficient scrutiny may appear to have solved the adoption problem while actually creating a more dangerous dependency. What matters is whether reliance is sufficiently grounded to allow useful outputs to influence cognition without making the institution cognitively passive.
This is why trust as a bottleneck should be distinguished from trust governance in full. The broader governance question concerns how reliance ought to be calibrated, monitored and controlled across different risks and contexts. The narrower cognitive question is what happens when otherwise usable machine capability cannot pass through the institutional interface because human actors will not treat its outputs as sufficiently credible to matter.
That narrower problem is already enough to transform how AI adoption should be understood.
An institution can buy capability without acquiring cognition. It can deploy an accurate system without creating reliance. It can integrate a model technically without integrating it cognitively.
The result is an organisation that appears more intelligent on paper than it is in practice.
The new bottleneck may therefore be trust because AI capability only becomes institutional capability when its outputs are permitted to influence attention, interpretation and judgement. When technically useful systems remain outside operative cognition because actors are unwilling to rely on them, the constraint no longer lies primarily in what the machine can do. It lies in the relationship that determines whether what the machine knows can become something the institution is prepared to use.