Governance, Safety & Risk Management

A system-prompt instruction is a preference, not a control. It shifts behavior; it does not guarantee it. Anything that must not happen needs a mechanism outside the model.

Layer controls in three places:

Input. Validate and constrain before the model sees anything. Length limits, format checks, classification of obviously out-of-scope requests. Treat anything that arrives from a document, a web page, or another system as data: never as instructions, no matter how imperative its phrasing.

Model. The system prompt sets role, scope, and refusal behavior. Real, worth doing, and insufficient on its own.

Output. Validate before anything acts on the result. Schema validation, policy classification, checking claims against source material. This is the layer that catches what got through, and it is the one most often missing.

The strongest guardrail remains architectural: the model cannot do what it has no tool to do. A model that cannot issue refunds cannot be talked into issuing one. Every prompt-level control is a weaker version of removing the capability.

Fabrication. Plausible, fluent, wrong. Most dangerous where the output looks authoritative and nobody downstream is positioned to check it. Mitigations: ground answers in retrieved source material, require citations, and give the model an explicit path to say it does not know, a model with no acceptable way to decline will guess.

Prompt injection. Instructions embedded in content the model processes. The defense is architectural, not prompt-level: constrain what the model can do, so that a successfully injected instruction has nothing dangerous to reach for.

Drift. The world changes, the prompt does not. Slow and invisible without monitoring, which is why the observability work in Domain 3 is a safety control, not just an ops nicety.

Silent degradation. A dependency changes, an index goes stale, a document set is refreshed badly. The system keeps returning confident output; nothing errors. Automated quality checks on a fixed regression set are what turn this from an outage discovered by customers into an alert.

Overconfidence in ambiguity. Given an under-specified request, the model picks an interpretation and proceeds. Sometimes right; when it isn't, the user has no signal that a choice was made on their behalf.

Requiring human review of everything destroys the value of automating anything. The design question is which decisions need a person and at what point they act.

Three placements:

Before the action: the model proposes, a person approves, then it executes. Correct for irreversible or high-consequence actions. Costs latency on every case.

After the action, sampled: it executes, a person reviews a sample. Correct for reversible, high-volume, moderate-consequence work. Catches systematic problems without gating throughput.

On exception: it executes autonomously unless a confidence or policy check fires. The usual middle ground, and only as good as the trigger; a threshold that never fires is decoration.

Two criteria decide the placement: reversibility and consequence. Irreversible and high-consequence goes before the action. Reversible and low-consequence gets sampled review. Most real systems are a mix, routed per action type rather than per system.

The failure to watch for is rubber-stamping: a review step where the reviewer approves everything because the volume is too high and the output usually looks fine. A review step nobody meaningfully performs is worse than none, because it manufactures the appearance of oversight.

Compliance mostly constrains where data goes and how long it stays, and it constrains architecture before it constrains prompts.

GDPR: lawful basis for processing, data minimization (send only what the task needs, not the whole record), the right to erasure, and limits on fully automated decisions with legal or similarly significant effects. Erasure is the one that bites LLM systems hardest: a system that copies personal data into logs, caches, eval sets, and vector indexes has multiplied the places that must honor a deletion request.

HIPAA: protected health information requires a business associate agreement with any processor, plus audit controls and access restrictions. "Which vendor sees the data" is an architectural question you answer at design time.

FedRAMP: the authorization framework for cloud services to US federal agencies. It largely determines which deployment options are available to you at all, which is why it belongs in the architecture conversation early rather than in a security review at the end.

Practically, four questions cover most of it: what data may be sent to the model; where it may be processed and stored; what must be logged, and for how long; what must be deletable, and everywhere it might have landed.

On ethics, three things that show up as design requirements rather than principles:

• Disparate impact: a system with acceptable aggregate accuracy can still fail badly for a subgroup. This is invisible unless you measure by segment.
• Transparency: people should know when they are dealing with an AI system, and in some jurisdictions this is law rather than courtesy.
• Contestability: someone affected by an automated decision needs a path to challenge it, which in turn requires you to have retained enough to explain what happened.