Keeping Secrets Out of the Database¶
A notification profile inevitably needs credentials — an SMTP password, an OAuth2 client secret, a webhook API key. None of those belong sitting in plain text in a database table that other roles, backups, and logical replicas can all see. pg_relay solves this with one small, general-purpose convention, and it is worth understanding properly rather than skimming past.
The rule
Never write a literal password, client secret, or API key into a notifier profile. Use an environment variable reference instead, described below.
The mechanism: _env:VAR_NAME¶
Wherever a profile would otherwise need a secret value, write the string _env:VAR_NAME instead — the literal prefix _env: followed by the name of an environment variable on the machine running the Processor. Immediately before that profile is used for anything at all, the Processor replaces every such string with the current value of that environment variable, read from its own host.
Before — a literal secret, never do this:
{
"host": "smtp.example.com",
"port": 587,
"username": "[email protected]",
"password": "Sup3rSecret!42"
}
After — an environment variable reference:
{
"host": "smtp.example.com",
"port": 587,
"username": "[email protected]",
"password": "_env:SMTP_NOTIFY_PASSWORD"
}
pg_relay_notifier's tables now hold "_env:SMTP_NOTIFY_PASSWORD" — a name, not a secret. The actual password only ever exists on the Processor host, in its process environment.
It works anywhere in the profile — not a fixed list of fields¶
This is the detail worth being precise about: _env: resolution is not limited to fields with obviously secret-sounding names like password or client_secret. It walks every string value anywhere in the profile, including inside nested objects — so a username field, or any other value you would simply rather not have persisted in the database, can use it too.
The SMTP transport's oauth2 block is a good example — every one of its five fields, username included, goes through the identical resolution:
Before:
{
"auth": "oauth2",
"oauth2": {
"username": "[email protected]",
"tenant_id": "72f988bf-0000-0000-0000-2d7cd011db47",
"client_id": "9c1c22b1-0000-0000-0000-1d4e5e2b4a11",
"client_secret": "abc123-do-not-do-this",
"scope": "https://outlook.office365.com/.default"
}
}
After:
{
"auth": "oauth2",
"oauth2": {
"username": "_env:M365_SMTP_USERNAME",
"tenant_id": "72f988bf-0000-0000-0000-2d7cd011db47",
"client_id": "9c1c22b1-0000-0000-0000-1d4e5e2b4a11",
"client_secret": "_env:M365_SMTP_CLIENT_SECRET",
"scope": "https://outlook.office365.com/.default"
}
}
There is nothing transport-specific about this — it applies equally to a webhook's auth.secret, a webhook's auth.username under basic_auth, an SMTP password, or an OAuth2 client_secret for Microsoft 365 or Azure Communication Services. If a value shouldn't be sitting in the database, reference it instead.
What is never scanned¶
Only profile is scanned for _env: references. message — the actual recipients, subject, and body content — is never touched. There is no legitimate reason for message content to contain a secret reference, and scanning it would mean partially interpreting content that is supposed to pass through untouched. More pointedly: messages are producer-authored, and resolving environment references in producer-authored data would let anyone able to queue an event read secrets out of the Processor's environment.
For the handful of APIs that authenticate inside the request body (PagerDuty's routing_key is the well-known case), the webhook transport's body_merge profile field bridges the gap without bending this rule: the secret-bearing key is declared in the profile — where _env: resolution already applies — and the Processor injects it into the outgoing body at send time, overwriting anything a producer put there. The scanning boundary is unchanged: profile yes, message never.
Values are never cached¶
Every _env: reference is resolved fresh, from the Processor's own process environment, at the moment it is needed — right after the notification's details are fetched, before anything else happens with them. The Processor caches nothing here: it never remembers a value from an earlier send, and no reload (SELECT pgrelay.request_reload();) or profile change is needed for a newly resolved value to be used.
That is a distinct thing from getting a new value into the Processor's environment in the first place, which is covered next.
An unset variable is a configuration error, not a delivery failure¶
If a profile references _env:SOME_VARIABLE and that variable is not set on the Processor host, the send does not happen at all, and it is never retried. The failure is permanent, and the error names the specific variable that was missing — so a typo in the variable name, or an environment file that was never actually loaded, shows up immediately and clearly, rather than as a mysterious repeating delivery failure.
Setting the variables¶
The Processor reads its environment the ordinary way any program does — from whatever set the environment of the process itself. On Linux, running under systemd (the standard way to run the Processor as a service — see Installing the Processor), that's an EnvironmentFile:
/etc/pg_relay/pg_relay.env is a plain KEY=VALUE file — one variable per line, no quoting needed for simple values. It's the same file that already carries the Processor's own database connection settings (PGHOST, PGUSER, and so on — see The Processor), so your notifier secrets simply join them:
PGHOST=/var/run/postgresql
PGPORT=5432
PGDATABASE=your_database
PGUSER=pgrelay
PGPASSFILE=/var/lib/pgsql/.pgpass
PGSSLMODE=prefer
SMTP_NOTIFY_PASSWORD=the-actual-password
M365_SMTP_USERNAME=[email protected]
M365_SMTP_CLIENT_SECRET=the-actual-client-secret
Never PGPASSWORD
If this file is also carrying the Processor's own database connection details, use PGPASSFILE for the database password, never PGPASSWORD directly in the file. The same principle that keeps a notifier secret out of the database applies here too — don't let a plaintext password sit somewhere it doesn't need to.
Because this file now holds real secrets, its permissions matter as much as its contents:
Mode 600 means only the file's owner can read or write it at all — not even other local users, let alone other services. Set the owner to whichever user actually runs the Processor (the example unit files in deploy/ run it as postgres), and never widen that.
Editing this file has no effect on a Processor that is already running — systemd only reads EnvironmentFile when it starts the process, not while it is up, and reloading pg_relay's own fleet (SELECT pgrelay.request_reload();) does not touch the operating system environment at all. So the two steps of rotating a secret are genuinely separate: getting the new value into the file takes effect only on the process's next start, while getting the Processor to use it (once it's running with the new value in its environment) needs nothing further at all — no reload, no restart, no profile edit — because it's read fresh on every single send. In practice, rotating a secret means editing the file and then restarting the service:
Continue to a transport chapter: SMTP, Microsoft 365, Azure Communication Services, Gmail, or Webhook.