AIES 2026 paper
A Normative Framework for Warranted Reliance in High-Stakes Workflows
When is reliance on generative AI warranted rather than simply induced by fluency, framing, or interface cues? This paper develops a normative framework for high-stakes workflows, organised around epistemic humility, epistemic access, and resistance to epistemic injustice.
A distinction we found easy to miss:
Trust, accuracy, safety, fairness, and explainability remain important. Yet none directly tells us whether a situated user has adequate grounds to defer to a particular output.
The paper shifts the unit of analysis from the model or output alone to the situated relation between user and system: the moment when generated content enters human reasoning and begins to shape what is treated as settled.
Does reliance track output correctness?
User behaviourAre performance and governance criteria satisfied?
System propertiesDoes this interaction support warranted reliance on this output?
Situated relationBehaviour alone cannot tell us whether reliance was warranted. The route by which a user arrives at acceptance matters.
The output is accepted because doing so is the path of least resistance, rather than because its grounds have been assessed.
Ease drives acceptanceThe user relies after encountering reliable and interpretable signals that provide adequate grounds for doing so.
Evidence supports relianceThe user examines the available signals, but those signals are themselves misleading or poorly connected to actual competence.
Engagement follows the wrong cueThe framework
The framework translates competence and audience-orientation from social epistemology into three conditions for generative AI.
Users should be able to tell when an output is uncertain, underspecified, outside the system’s reliable scope, or in need of verification.
Interactive figure 1. Select a condition to inspect its constitutive properties and primary intervention layer.
Select a condition to remove it. The framework treats each as a threshold, rather than combining them into an average trustworthiness score.
First-order accuracy remains a necessary precondition; the relational conditions can now be evaluated.
Interactive figure 2. Remove any condition to see why the framework treats humility, access, and recognition as non-fungible.
Four high-stakes settings
The cases show how consequential harm can remain invisible to standard measures of accuracy, fairness, source-grounding, or safety.
In Mata v. Avianca, fabricated cases were presented as real and reaffirmed when the user explicitly asked for confirmation. The problem was not confined to a false first answer: the interaction failed to surface and preserve the system’s limits.
Epistemic humility failed during generation and reconfirmation. Weak verification pathways made the failure consequential.
Mata documents repeated unsupported authority across two turns. The paper pairs it with MetaMedQA, where high task accuracy did not guarantee reliable detection of missing knowledge. These findings do not reduce the professional duty to verify.
Evidence reported in the paper’s case analysis.
Interactive figure 3. Choose a case to compare the framework diagnosis with the specific boundary of the available evidence.
Calibrated friction
Interfaces can introduce proportionate friction when evidence is sparse, conflicting, weakly grounded, or consequential. The purpose is to redirect attention towards reliable evidence, rather than to obstruct users indiscriminately.
Support fluent interaction when grounding is robust and downstream stakes are limited.
Surface uncertainty, disagreements, and the claims that require closer inspection.
Require verification or confirmation before the output travels into a high stakes decision.
Friction can reinforce misinformed deference when it sends users back to a poorly calibrated confidence statement, a decorative citation, or a fluent explanation. It should instead redirect attention towards cues that support epistemic assessment.
Prefer signals grounded in
Friction is justified when it is proportionate to genuine epistemic risk and directs users towards cues that support assessment.
Layer-explicit evaluation
Each condition has a primary intervention point, but none can be secured at one layer alone. Evaluation should preserve this distributed structure.
Highlighted cells indicate the primary starting point, rather than a boundary on responsibility.
Research contributions
A guide to the paper’s conceptual contribution, relevance, and evidentiary scope for researchers.
The paper defines epistemic trustworthiness as a property of the relation between a situated user and a particular output. It shifts attention from whether users accept correct advice to whether they have adequate grounds to rely on, verify, contest, or withhold reliance from that output.
Drawing on social epistemology, the paper specifies epistemic humility, epistemic access, and resistance to epistemic injustice as jointly necessary and nonfungible conditions for warranted reliance. Accuracy, calibration, explanations, citations, and fairness may support these conditions, but none establishes epistemic trustworthiness by itself.
The framework connects each condition to intervention points across model, data, and interface layers. It proposes evaluating the conditions as a diagnostic profile, since strong performance on one dimension cannot compensate for failure on another.
Analyses of legal hallucination, résumé screening, source grounded legal tools, and medical refusal patterns show how distinct breakdowns in warranted reliance become visible through the framework. The paper then develops calibrated friction as a design principle for directing users towards reliable epistemic cues when reliance carries greater risk.
Explore the complete normative argument, derivation, real-world cases, and implications for responsible AI evaluation and design.
@inproceedings{karnatak2026epistemic,
title={Epistemic Trustworthiness in Generative AI: A Normative Framework for Warranted Reliance in High-Stakes Workflows},
author={Karnatak, Nimisha and Van Kleek, Max and Shadbolt, Nigel},
booktitle={Proceedings of the 2026 AAAI/ACM Conference on AI, Ethics, and Society},
year={2026},
url={https://arxiv.org/abs/2608.05602}
}