Abhinav Saxena
Glossary

Epistemic opacity

Also called: opacity, the black-box problem, essential epistemic opacity

Definition

The condition of a process whose epistemically relevant elements an agent cannot all know, as with a complex simulation or a trained neural network.

By · Updated · 1 min read

A process is epistemically opaque to an agent when the agent cannot know all of its epistemically relevant elements. Humphreys introduced the notion for computer simulation, where no human can survey every step of a computation that nonetheless delivers scientific results, and distinguished essential opacity, which follows from the nature of the agent and the process, from opacity that is merely a practical limit. Trained neural networks are now the standard example.

The literature that followed has largely concluded that opacity is compatible with warranted reliance under conditions. Durán and Formanek’s computational reliabilism grounds warrant in verification and validation, robustness analysis and a history of successful use rather than insight into the process. Creel distinguished the several things transparency might mean: of the algorithm, of its implementation in code, of the particular run that produced an output. Sullivan argued that opacity obstructs understanding less than uncertainty about the link between a model and its target.

I accept all of this, and argue that the predicament of language models is an acute instance of it, with a difference. Opacity, so described, is a limit on what a user can know about a connection that obtains. A barometer is opaque to someone who does not understand aneroid cells, and the remedy is that she learn, or consult someone who has. With current language models the deeper trouble is not the user’s access to the calibration but the calibration: nothing in the production of an open-domain assertion is keyed to the truth of the particular content emitted. They are opaque instruments in the ordinary sense, and degenerate ones as well.

This is why the demand for validation is not a formality. When an instrument is opaque but sound, validation confirms a connection one could in principle inspect. When it is opaque and the connection runs through proxies, the record a user projects is all the connection there is. Interpretability research could change this verdict for particular systems, and the Tracking Minimum specifies what it would have to show.

Questions people also ask

What does epistemic opacity mean?

A process is epistemically opaque to someone when they cannot know all of its steps or elements that matter for judging its results. Paul Humphreys introduced the term for computer simulations, whose individual computational steps no person can survey, though the simulations still deliver scientific results.

What is an example of epistemic opacity?

A deep neural network that sorts medical images. Its outputs can be checked against outcomes, but no one can trace how its millions of learned weights produced a particular answer. A long climate simulation, the kind of case Humphreys had in mind, is another.

What is essential epistemic opacity?

Humphreys distinguished opacity that is merely practical from opacity that is essential. A process is essentially opaque to an agent when, given the kind of agent it is, it is impossible for that agent to know all the relevant elements, as when a computation is far too long for any human to follow step by step.

Can you trust a system that is epistemically opaque?

You can rely on it with good reason if its reliability has been established by testing, validation and a record of success, which is the view called computational reliabilism. My caution is that for language models the record is all the connection to the truth there is, so the testing has to be taken seriously.

Related terms

Computational reliabilismThe view that reliance on opaque computational processes can be warranted by verification, validation, robustness analysis and a track record, without insight into the process.Instrument readingComing to know by competently reading a reliable device, where warrant is generated by the reader’s use rather than transmitted from a speaker.ExplanationAn account of why something is the case or why something was done; in AI, an account of why a system produced a given output.Chain-of-thought faithfulnessThe degree to which a language model’s displayed reasoning reflects the process that actually produced its answer, rather than a plausible rationale produced alongside it.Explainable AI (XAI)The effort to make the outputs of machine-learning systems understandable to the people who use or are affected by them, either by building models whose workings can be followed or by attaching explanations to models whose workings cannot.

Questions that turn on it

QuestionIs an AI chatbot an informant or an instrument?

Referred to by

Artificial IntelligenceTechnologyAutomation biasAlgorithmic biasRight to explanation

Further reading

  1. Humphreys, P. (2009). The philosophical novelty of computer simulation methods. Synthese 169(3): 615–626.doi.org/10.1007/s11229-008-9435-2
  2. Creel, K. A. (2020). Transparency in complex computational systems. Philosophy of Science 87(4): 568–589.doi.org/10.1086/709729
  3. Sullivan, E. (2022). Understanding from machine learning models. The British Journal for the Philosophy of Science 73(1): 109–133.doi.org/10.1093/bjps/axz035