Skip to main content

Forward audit logs to your SIEM

The AI Gateway emits structured JSON audit events to pod standard output. It writes one event for each model interaction, one for each rejected authentication attempt, and one for each model list request that model access filters down to nothing.

Events record decisions and metadata only. They never carry prompts, completions, credentials, request or response bodies, token claims, or email addresses.

Every event carries the fields required by NIST SP 800-53 controls AU-2, AU-3, and AU-8: a unique audit ID, a UTC timestamp, the outcome, the component, the source, an optional target, and an extra block for forensic correlation. Events for authenticated requests also carry the subject.

For the legacy AI Gateway, see legacy gateway audit logs.

Enable audit events​

Audit events are off by default. Turn them on for the gateway:

values.yaml
ai-gateway-v2:
audit:
enabled: true

Setting global.stacklok.audit.enabled: true also turns them on, and while it's true, audit.enabled: false has no effect. The change applies when the gateway pods roll out; see How configuration changes apply.

Enabled audit events cover every interaction without sampling, and there are no per-event toggles. Filter and route events in your log pipeline.

What gets captured​

Event typeFires whenOutcome
aigw.v2.interaction.completedAn authenticated generation request reaches its final outcome, including an early refusalsuccess, denied, failure, or error
aigw.v2.authentication.failedThe gateway rejects a credential on a protected route, or can't reach a service it needs to validate itfailure, or error when unreachable
aigw.v2.models.no_grantModel access filters a non-empty model list to nothing, and the caller receives a 200denied

Interaction events cover the OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini generate and stream-generate endpoints. A streaming interaction emits one event when it finishes, never one per frame. Health endpoints emit no events, and a model list request that returns at least one model emits none.

Event schema​

{
"time": "2026-10-16T14:22:31Z",
"level": "INFO",
"msg": "audit_event",
"logger": "audit",
"audit_id": "b3d4f7a2-8c11-4f6e-9b0a-1c2d3e4f5a6b",
"type": "aigw.v2.interaction.completed",
"logged_at": "2026-10-16T14:22:31.418Z",
"outcome": "success",
"component": "v2",
"source": { "type": "local", "value": "ai-gateway-v2-6d8f9c7b5-x2k4q" },
"subjects": { "user": "00u1a2b3c4d5e6f7g8h9", "department": "engineering" },
"target": {
"model": "claude-sonnet-4-6",
"provider": "anthropic",
"route": "/v1/messages"
},
"metadata": {
"extra": {
"schema_version": 1,
"request_id": "c0ffee01-2a3b-4c5d-6e7f-8091a2b3c4d5",
"interaction_id": "7e1d5390-41e5-4f71-b3c2-9a0815793c4a",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"http_method": "POST",
"http_status": 200,
"dialect": "anthropic-messages",
"streaming": true,
"duration_ms": 8421,
"terminal_stage": "delivery",
"reason_code": "delivered",
"usage_present": true,
"input_tokens": 1834,
"output_tokens": 412
}
}
}

Subjects appear only after authentication succeeds. user is the caller's canonical subject, and virtual_key_id identifies the virtual key on virtual-key requests. Directory group labels named in directory.groupLabels, such as department, are added alongside them. A rejected authentication event carries no subjects, except an auth_identity_unresolved refusal, which names the verified caller in user, or in the service account subjects described next.

A service account's events name its workload instead of a user. service_account is the verified subject, in the form system:serviceaccount:<NAMESPACE>:<NAME>, service_account_authority identifies the trusted issuer that verified it, and service_account_uid is the ServiceAccount's UID, which tells a recreated ServiceAccount apart from the one it replaced.

Targets carry model, provider, and route, each only once the gateway has resolved it. provider is the provider kind from the gateway configuration: openai, openrouter, anthropic, bedrock-anthropic, bedrock-openai, gemini-aistudio, or vertex. route is the endpoint path template.

Source is the gateway pod that wrote the event. Events carry no client IP address or user agent.

Common extras are schema_version, request_id, http_status, and, when tracing is on, trace_id and span_id for OpenTelemetry correlation. The gateway also returns request_id to the client in the X-Request-ID response header. Authentication events add auth_method: jwt, virtual_key, workload, or unknown, and reason_code, which says why authentication failed, such as auth_identity_unresolved.

Interaction events add interaction_id, http_method, dialect, streaming, duration_ms, terminal_stage, reason_code, and usage_present, plus input_tokens and output_tokens when the provider reported usage. terminal_stage is the stage the interaction ended in: validation, routing, governance, provider, metering, or delivery. reason_code says what ended it, such as delivered or governance_denied. When a governance hook denies or fails a request, hook names the hook entry.

Required governance hooks can report annotations under hook_annotations, keyed by hook entry name. The budgets entries report the charged subject and amount, and the scanning entry reports the entity types it acted on, such as detection_types for a block and redact_types for a redaction.

Separate audit records from application logs​

Each event is one line of JSON on the pod's standard output. Every audit record carries "logger":"audit" and "msg":"audit_event", so shippers should filter on that pair to separate audit records from ordinary application logs on the same stream.

Ship them​

Each example below filters on those same two keys. Adapt hostnames, credentials, and <NAMESPACE> for your environment.

Fluent Bit to Splunk​

[INPUT]
Name tail
Path /var/log/containers/*<NAMESPACE>*.log
multiline.parser docker, cri
Tag kube.audit.*
Refresh_Interval 5

[FILTER]
Name grep
Match kube.audit.*
Regex log "logger":"audit"
Regex log "msg":"audit_event"

[FILTER]
Name parser
Match kube.audit.*
Key_Name log
Parser json
Reserve_Data On

[OUTPUT]
Name splunk
Match kube.audit.*
Host <SPLUNK_HEC_HOST>
Port 8088
Splunk_Token ${SPLUNK_HEC_TOKEN}
TLS On
event_source aigw-audit
event_index aigw_audit

A record passes the grep filter only when both Regex rules match, so the filter works whatever order the two keys appear in.

Fluent Bit to Elastic​

Keep the input and both filters from the Splunk example above, which tag records kube.audit.*, and swap the output. The Match value has to be that same tag, or the output receives nothing:

[OUTPUT]
Name es
Match kube.audit.*
Host <ELASTICSEARCH_HOST>
Port 9200
HTTP_User ${ES_USER}
HTTP_Passwd ${ES_PASSWORD}
tls On
Logstash_Format On
Logstash_Prefix aigw-audit
Replace_Dots On

Vector to Kafka​

sources:
kubernetes_logs:
type: kubernetes_logs
extra_namespace_label_selector: 'kubernetes.io/metadata.name=<NAMESPACE>'

transforms:
parse_json:
type: remap
inputs: [kubernetes_logs]
source: |
. = parse_json!(.message)
if .logger != "audit" || .msg != "audit_event" { abort }

sinks:
kafka:
type: kafka
inputs: [parse_json]
bootstrap_servers: '<KAFKA_BROKERS>'
topic: aigw-audit
encoding:
codec: json
compression: snappy

Promtail to Loki​

Promtail's match stage selects on labels, so this pipeline promotes logger and msg to labels, drops every line that isn't an audit record, then removes those two labels.

scrape_configs:
- job_name: aigw-audit
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
regex: <NAMESPACE>
action: keep
- source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
separator: /
target_label: __path__
replacement: /var/log/pods/*$1/*.log
pipeline_stages:
- cri: {}
- json:
expressions:
logger: logger
msg: msg
type: type
outcome: outcome
component: component
- labels:
logger:
msg:
- match:
selector: '{logger!="audit"}'
action: drop
- match:
selector: '{msg!="audit_event"}'
action: drop
- labeldrop:
- logger
- msg
- labels:
type:
outcome:
component:

On nodes that use the Docker runtime, replace cri: {} with docker: {}. Promtail has reached end of life. For new installs, use Grafana Alloy, which converts this configuration with alloy convert --source-format=promtail.

CloudWatch and rsyslog​

The CloudWatch agent cannot filter on a JSON key natively. Either ingest everything and apply a log-group metric filter:

{ $.logger = "audit" && $.msg = "audit_event" }

or run Fluent Bit alongside it with the cloudwatch_logs output, using the filters above so only audit lines reach the dedicated log group.

For rsyslog, match both markers before forwarding:

if $msg contains '"logger":"audit"' and $msg contains '"msg":"audit_event"' then {
action(type="omfwd"
target="<SIEM_HOST>"
port="514"
protocol="tcp"
template="RSYSLOG_SyslogProtocol23Format")
stop
}

Retention, tamper evidence, and GDPR​

Configure retention, tamper evidence, and privacy controls in your ingest and storage systems. Use lifecycle policies and write-once storage where your compliance requirements call for them.

Apply field-level redaction before events reach long-term storage. The gateway does not retain or replay emitted audit events.

Log the text behind a denial​

Audit events record that a governance hook denied a request, but not the text that tripped it. To see that text when you review a PII block or a prompt injection verdict, turn on failureLogs.includePrompt:

values.yaml
ai-gateway-v2:
failureLogs:
includePrompt: true
warning

The logged text is raw user content, including the PII the scanner detected. Restrict access to the gateway's standard error, including Kubernetes pod log access, and set retention in your log pipeline to what your data handling policy allows for that content.

The chart won't render failureLogs.includePrompt: true while failureLogs.includeMessage is false, because that setting asks for metadata-only records.

The PII scanner and the prompt injection guardrail then append the text behind each denial to the cause they return, after ; prompt: . The gateway writes that cause to an operational failure record on standard error, separate from audit events. The scanner appends each message that carried a blocked entity, prefixed with its path in the request body. The guardrail appends the text it screened. Each appends at most 15 KiB, and the gateway logs up to 16 KiB of a cause instead of the usual 1 KiB. The adapter pods roll out when the setting changes.

A refused request leaves one ai-gateway-v2: request failed record. A request the guardrail flags in Monitor mode is still forwarded and leaves one ai-gateway-v2: governance would deny record. Both records carry:

  • request_id and interaction_id, which match the audit event. The client also receives request_id in the X-Request-ID response header.
  • subject, the caller's canonical subject. A refused request's record also carries email, looked up in the directory.
  • model and provider, once the gateway has resolved them.
  • hook, the hook entry that denied the request.
  • cause, the hook's reason followed by the request text.
  • hook_annotations, such as the entity types the scanner detected or the guardrail categories that fired.

Only denials carry text. A request the scanner redacts, or matches under log-only or mode: shadow, is forwarded and leaves no failure record. A redaction that fails refuses the request, and the scanner appends the message it couldn't redact.

To turn the text on for one screen only, set scanning.denyPromptInMessage or guardrails.denyPromptInMessage. Each wins over failureLogs.includePrompt in either direction, so guardrails.denyPromptInMessage: false keeps the guardrail's text out of the log while the scanner's still goes in.

The gateway drops a cause that looks like a credential and sets cause_unavailable: true instead.

The shipper examples above forward audit records only, so keep their filters as they are. Read failure records from the gateway pod logs:

kubectl logs -n <NAMESPACE> deploy/<RELEASE_NAME>-ai-gateway-v2 \
| grep -E 'request failed|governance would deny'

To keep them longer, collect the gateway's standard error through a separate pipeline, with access controls and retention suited to raw user content.

The Kubernetes API audit log covers control-plane activity such as kubectl calls and Helm upgrades. Configure it through the API server audit policy.

Operational failure records go to the gateway's standard error, separate from audit events. A record can name the caller by directory email and, by default, includes a bounded parsed error message to make failures easier to diagnose. failureLogs.level sets their severity: warn (the default), error, or debug. When log retention or access can't accommodate personal data, set failureLogs.includeMessage to false to drop the message. To add the text behind a governance denial, see Log the text behind a denial.

Conversation content reaches no audit event. Screening sends prompt text to Presidio and AWS Bedrock for analysis. Snapshot capture, configured with the top-level capture value, records full request and response bodies for testing. It's unset by default; leave it unset in production.

Next steps​