Who Rotated That Key? An Audit Trail That Answers in One Query

The contract: every mutating administrative action on an nRouter organization writes one append-only entry carrying actor, action, target, organization context, and a client IP captured the same way everywhere. Nothing in the product edits or deletes an entry. Reads are role-scoped and stop at your organization boundary.

A key rotation lands at 02:14. Twelve hours later a service that nobody remembers owning starts returning 401s, and the question arrives in Slack: who rotated that key, and were they supposed to? If the honest answer is "let me pull application logs from three services and correlate timestamps", you do not have an audit trail. You have hope, plus a weekend.

This post is the audit-trail design we hold ourselves to, written as a customer-visible contract rather than an architecture tour. Everything below is something you can check from your own dashboard, on your own organization, today.

The problem in one request

Here is the single request that starts the whole story. An owner rotates a virtual key from the dashboard:

PATCH  (key rotation, from the nRouter dashboard)
actor        alex@example.com   (owner)
target       sk-nrouter-…9f21
result       new key issued, old key deprecated with a grace period

That is one action. It changes who can spend your credits, which models are reachable, and whether a production service keeps working. Six months later a compliance reviewer will ask three questions about it: who, when, and from where. A fourth question follows immediately: can that record have been changed since?

An audit trail is the artifact that answers all four without a human reconstructing anything. Everything else — pretty tables, CSV export, severity tints — is presentation on top of those four answers.

Why the naive approach breaks

The version most teams build first is a logging call inside each route handler. It looks reasonable, it ships in an afternoon, and it fails in four specific ways that only surface under scrutiny.

1. Coverage is opt-in. A new route ships without its logging line and nobody notices, because nothing goes red when an entry is missing. The trail is silently incomplete, and "silently incomplete" is worse than absent — it looks authoritative.
2. Fields drift. One handler records an email, another a user id, a third a display name. Filtering by actor now requires knowing which handler wrote the row you are looking for.
3. Client IP is the field that goes missing. This is the defect auditors find, every time. Nine actions capture the caller's IP through one helper and the tenth — usually something innocuous like a profile update — forgets. Now your "100% of administrative actions are attributed to a source" claim is false, and it is false in exactly the place a reviewer will sample.
4. The record is editable. If the same admin console that rotates keys can also delete a row from the activity view, the trail records history at the pleasure of the person it is recording. That is not evidence. That is a narrative.

The exact input that defeats the naive version is mundane: a new action type, added by a different engineer, six weeks later. Nothing about the naive design forces that action to join the trail, and nothing tells you it did not.

What counts as an auditable action

The trail records deliberate administrative acts — the ones that move security posture or money. It does not record inference traffic.

Per-request LLM traffic is a different, far higher-volume concern with different retention and different redaction rules — that is request logging, and mixing the two produces a trail so noisy that nobody reads it. The rule of thumb we use: if a human chose to do it and it changed the rules the gateway enforces, it belongs in the audit trail. If a program did it as part of serving traffic, it belongs in request logs.

The five fields every entry carries

An entry is only worth keeping if it answers who, what, to what, from where, and in whose organization. Miss one and it degrades to trivia. This is not a novel list — NIST's log-management guidance has asked for subject, action, object, source and outcome on every security event since SP 800-92, and the LLM case adds nothing to it except volume.

actor    the acting user's identity, plus the role they held at the time
action   a verb.object string  —  key.rotated, member.removed, budget.raised
target   the specific key, member, team, or setting affected
source   client IP + an ISO-8601 timestamp
context  the organization (and team, where the action is team-scoped)

Two of these are subtle.

"The role they held at the time" matters because roles change. If Alex was an owner in June and a viewer by September, an entry that stores only a user id loses the fact that the June action was authorized. Recording the role at the moment of the action makes the entry self-contained.

verb.object action names matter because they are the filter axis. key.rotated is machine-filterable, sorts next to key.created, and renders as a stable label. A free-text description — "Alex rotated a key" — is a sentence, not a field, and you cannot build "show me every key rotation last quarter" on top of sentences.

The mechanism, as a customer-visible contract

Strip out the implementation and the contract is four promises.

1. Write-with-the-action. The entry is written as part of the same administrative operation, not queued to a best-effort sink afterwards. If the action succeeded, the entry exists. There is no window where a key is rotated and the trail has not caught up.
2. Failed attempts are recorded too. A rejected role change — someone without authority trying to promote themselves — is exactly the event a reviewer wants. The outcome is a field, not a filter on which entries get written.
3. Append-only through the product. No dashboard screen, no API route, and no support action edits or deletes an entry. Corrections are new entries. This is the property auditors actually test — it is the AU-9 control, protection of audit information, in NIST SP 800-53 Rev. 5 — and it is a capability question ("is there any path?"), not a policy question ("do we usually?").
4. Reads are role-scoped and tenant-isolated. Viewing the trail is an owner/organization-admin capability, and a read never crosses an organization boundary — the same isolation described in org, team, member scoping.

What is deliberately not promised: ordering across clock skew at sub-second resolution, and real-time streaming. Entries carry timestamps you can sort; they are not a distributed-consensus log, and pretending otherwise would be the kind of overclaim this post exists to avoid.

Getting the client IP right, once

Client IP is easy to get wrong and even easier to get inconsistently, and the two failures compound.

Wrong looks like trusting whatever a caller put in a forwarding header. Anyone can set that, which is why the OWASP logging cheat sheet treats event-source data as untrusted input to be validated rather than as fact. An attribution field a caller controls is worse than a blank one, because a blank field reads as unknown while a spoofed field reads as fact.

Inconsistent looks like every route deriving the IP its own way. Three extraction styles across twenty routes means three behaviours behind one column heading, and you will not discover which is which until the one time it matters.

The fix is boring and it is the whole point: one shared derivation, used by every path that writes an entry. Correctness lives in one place, so improving it improves every action at once, and a new action cannot ship with a subtly-different notion of "where this came from". Consistency is what converts "we capture source IP" from a claim into a property.

Worked example: reconstructing the 02:14 rotation

Back to the opening incident. With a complete trail, the investigation is three filters and about ninety seconds.

Step 1 — filter by action and window. action = key.rotated, last 24 hours. One entry returns.

Step 2 — read the entry.

Step 3 — filter by actor to get the shape of the session. Everything Alex did between 02:00 and 03:00:

Now the 401s have an explanation and a timeline: a planned rotation at 02:14, and a service still holding the deprecated key past its grace period. The answer to "were they supposed to?" is in the same rows — Alex held owner at the time, and the role change 15 minutes earlier is itself attributed.

Compare that with the naive version, where step 1 is "which service logs rotations?" and step 3 does not exist at all.

Edge cases we had to decide

These are the decisions that make a trail either usable or merely present.

1. When an action fails authorization, we still write an entry, because a rejected privilege escalation is a security event. The outcome field distinguishes it. A trail that only records successes cannot show you someone trying.

2. When an action is performed by an automated flow rather than a person, we record the flow as the actor rather than blanking the field, because "no actor" is indistinguishable from "we lost the actor". An entry should never need a footnote explaining why a field is empty.

3. When a target is deleted afterwards, the entry keeps the identifying detail it had at write time, because an audit entry that dereferences a live record goes blank the moment the record goes away. The trail records what was true then, not what is true now — that is the entire job.

4. When an action's stored name has no human label yet, we render the raw verb.object string rather than falling back to a generic neutral label, because a security-relevant change displayed as "Updated" with a neutral tint reads as noise. A slightly ugly settings.retention_changed is honest; a pretty "Updated" is not. Missing polish is a to-do; a misleading label is a defect.

5. When someone asks us to remove an entry, the answer is no, and corrections go in as new entries, because an append-only trail with an exception process is not append-only. Erasure obligations under GDPR Article 17 are handled through retention policy on personal data, not by rewriting the record of who did what — and the trail is exactly the sort of processing account Article 30 expects a controller to be able to produce.

What you see from the outside

Everything above surfaces in places you can check without asking us.

• The audit view in the dashboard — filter by actor, action type, target, outcome, and time range; see Audit log for the field-by-field walkthrough.
• Role-scoped access — an organization member without owner or organization-admin authority does not see the view. The role model is the same one described in org, team, member.
• Tenant isolation on reads — a query never returns another organization's entries. Not filtered client-side: not returned.
• Correlation with request traffic — inference responses carry x-nr-request-id in the x-nr-* header namespace, so a request you are investigating can be lined up against the administrative timeline by timestamp. The audit trail and request logs stay separate stores with separate retention; correlation is by time and identity, not by merging them.
• Export — the view exports so you can hand a reviewer a file rather than a screen recording.

Note what is not on that list: an "edit entry" affordance, a "delete row" affordance, and a bulk-clear. Their absence is the feature.

Limits

Honest boundaries, because a control described without its edges is marketing.

• It records administrative actions, not prompts or completions. If you need to know what a model was asked, that is request logging, and it comes with its own redaction rules.
• It is append-only through the product, which is not the same as cryptographically chained. The property we hold is that nothing in the product surface offers a way to alter history. If your threat model includes a compromised platform operator, ask us about it directly rather than reading a stronger guarantee into this sentence.
• Retention is finite. If your obligation is multi-year, export on a schedule into your own archive rather than assuming the view is your system of record.
• It is not an alerting system. The trail answers questions after the fact. Real-time reaction to spend or traffic anomalies belongs to budgets and rate limits, which stop things rather than describing them.
• SOC 2 Type II is in progress at nRouter, not complete. The audit trail is one control in a wider set — the security criteria in the AICPA's trust services criteria are what a Type II report is written against; see the SOC 2 checklist for the rest and our trust page for current status.

Try it

Open your organization's audit view and run the auditor's own sample. Pick five administrative actions of different classes from the last month and check all five for a populated actor, target, timestamp, and source. If any column is blank, that is a finding — and it is better to find it yourself.

Then do the deliberately boring thing: rotate a test key from API key management, refresh the audit view, and confirm the entry is there with your identity on it before you need it to be. If you do not have an organization yet, sign up — signup takes a real card and a $5 minimum charge, with the platform fee on top.

See also

• Org, Team, Member: Scoping Keys, Budgets, Guardrails — the authority model the audit trail's role-scoped reads resolve against.
• Write-Time PII Redaction in LLM Logs, Without Losing Debug Detail — the sibling problem for request traffic, where redaction rather than immutability is the hard part.
• A SOC 2 Checklist for LLM Gateways — where this control sits among the others a reviewer will ask about.
• What to Log (and Not Log) on an LLM Gateway — the high-volume half of observability, deliberately kept separate from the trail.
• Virtual Keys vs Master Key: Scoping a Key Per Job — why key rotations are worth auditing in the first place.
• RPM and TPM Rate Limiting Per Key, Team, and Org — the enforcement controls whose configuration changes show up in the trail.
• Trust — current compliance posture, including SOC 2 Type II progress.

Sources

Verified 2026-06-11. This post makes no third-party product claim; every behaviour described is an nRouter capability you can check from your own organization. Corrections to hello@nrouter.ai and we will update.

• nRouter audit log documentation: nrouter.ai/docs/guides/audit-log
• nRouter API key management: nrouter.ai/docs/guides/api-key-management
• nRouter pricing: nrouter.ai/pricing
• nRouter security posture: nrouter.ai/security

External references, all checked 2026-08-23:

• NIST SP 800-92, Guide to Computer Security Log Management: csrc.nist.gov/pubs/sp/800/92/final — the field set behind "the five fields every entry carries".
• NIST SP 800-53 Rev. 5, AU control family: csrc.nist.gov/pubs/sp/800/53/r5/upd1/final — AU-9, protection of audit information, is the append-only property stated as a control.
• OWASP Logging Cheat Sheet: cheatsheetseries.owasp.org — why caller-supplied event-source data is untrusted input.
• AICPA/CIMA trust services criteria (2017, revised points of focus 2022): aicpa-cima.com — what a SOC 2 Type II report is evaluated against.
• GDPR Article 30, records of processing activities: gdpr-info.eu/art-30-gdpr
• GDPR Article 17, right to erasure: gdpr-info.eu/art-17-gdpr — the obligation the append-only edge case resolves through retention rather than deletion.

AICPA, CIMA, NIST and OWASP publish the documents linked above; nRouter is not affiliated with or endorsed by them.