What it does
A client points the standard vendor SDK at this server instead of at the vendor, swapping a base URL and a key. Everything else about their code is unchanged. The server authenticates the tenant, enforces which models that tenant may call, caps the token ceiling on the request, relays it, and writes a usage row.
The provider credential lives in one place and is never distributed. Revoking a tenant is a row update, not a key rotation across every machine that had a copy.
Metering that survives streaming
Usage is easy to record when a response arrives in one piece and easy to lose when it streams. The relay pipes the event stream straight through to the client while parsing it in passing for the two events that carry token counts, and writes the usage row in a finally block so a client that hangs up mid-stream is still billed for what was actually produced.
In the code
CREATE TABLE tenants (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name TEXT NOT NULL,
api_key TEXT UNIQUE NOT NULL,
status TEXT NOT NULL DEFAULT 'active'
CHECK (status IN ('active', 'suspended', 'deleted')),
rate_limit_rpm INTEGER NOT NULL DEFAULT 60,
settings JSONB NOT NULL DEFAULT '{}'::jsonb
);
-- Enable RLS so anon key gets zero data.
-- Our server uses service_role key which bypasses RLS entirely.
ALTER TABLE tenants ENABLE ROW LEVEL SECURITY;
ALTER TABLE usage_logs ENABLE ROW LEVEL SECURITY;
-- No policies for anon = no access via anon key.Three decisions worth the space. Status is a constrained enum rather than a boolean, so suspended and deleted are different states and a suspended tenant can be brought back without re-provisioning. Per-tenant limits and allowed models live in a JSONB column, so tightening one customer is a row update rather than a deploy. And row level security is switched on with no policies written for the anonymous role, which is the strongest version of that setting: not a policy that has to be correct, but the absence of any policy at all, so the public key can read nothing and only the service role the server holds gets through.
} finally {
res.end();
logUsage(tenant.id, body.model, totalInputTokens, totalOutputTokens,
anthropicRes.status, Date.now() - startTime);
}The obvious place to log usage is after the stream loop completes, and that is the version that quietly loses money. A client that closes the connection halfway through has still consumed the tokens upstream, and a throw inside the read loop skips straight past a log call placed after it. Moving the write into finally means the row is written on the success path, the error path and the client-hangup path alike. The same function is also called on the 400, the upstream error and the 502 branches, so every request that reaches the relay produces exactly one usage row regardless of how it ended.
How this differs from the ordinary version
The client keeps their own SDK
Most gateways make you adopt their client library, which means the integration is a rewrite and the exit is another rewrite. This one is wire compatible with the vendor API, so adoption is two configuration lines and leaving is the same two lines.
Limits are per tenant and live in the row
Allowed models and the token ceiling default at the server and are overridden per tenant in a settings column. A customer who needs a larger ceiling gets it without a code change, and one who needs restraining gets it without a deploy.
One credential, one blast radius
The point of the relay is that the provider key exists on the server and nowhere else. Every client-side compromise is bounded to a tenant key that can be rotated from an admin page.
In the field
Built for a wire manufacturer
This is the substrate under a CORTEX deployment for a continuous manufacturing operation. A plant that wants assistant capability inside its own tooling does not want to hand a provider credential to every machine that needs it, and does not want a per seat contract with a vendor either. A relay with per tenant metering is the shape that answers both.
Questions
- What changes in the client code?
- A base URL and a key. The relay is wire compatible with the vendor Messages API, streaming included, so the rest of an existing integration is untouched.
- How is usage tracked on streamed responses?
- The relay pipes the event stream to the client while parsing it for the two events carrying token counts, and writes the usage row in a finally block so a disconnect mid-stream still records what was produced.