Skip to content

Hardening the Service

This chapter is the Linux host itself: how the Processor runs as a service so that its credentials — the database credential in .pgpass and the provider secrets in its environment — are reachable by the Processor and by nothing else on the machine. It applies to every on-premises or cloud VM deployment; the container equivalents are noted where they differ.

The recommendations below go well beyond deploy/pg_relay.service, which is written for the co-located sandbox tier (User=postgres, no sandboxing). Production uses deploy/pg_relay-hardened.service and deploy/pg_relay-hardened.env.example instead — the unit reproduced in this chapter, shipped as files so it is installed rather than transcribed:

sudo cp deploy/pg_relay-hardened.service /etc/systemd/system/pg_relay.service
sudo systemctl daemon-reload

(Installing it under the name pg_relay.service keeps the systemctl commands in the rest of this site unchanged.)

Why the environment is the thing to protect

Two credentials live on the Processor host and nowhere else:

  • PGPASSFILE — a file the Processor reads when it opens a database connection.
  • The _env: variables — provider secrets that pg_relay resolves from the process environment immediately before each send, never storing them anywhere (Keeping Secrets Out of the Database).

A process's environment is visible to three parties: the process itself, the root user, and — via /proc/<pid>/environ — any process running as the same user. The same is true of its memory, through /proc/<pid>/mem, ptrace, and core dumps. So "protecting the environment" means: a user nobody else runs as, a /proc other users cannot inspect, and no way for the process's memory to be written to disk. That is the whole shape of this chapter.

A dedicated, non-login user

Create a system account whose only purpose is running the Processor. It must not be postgres (see Where to Run the Processor), must not be shared with any other service, and must not be able to log in:

sudo useradd --system --no-create-home --shell /usr/sbin/nologin \
             --home-dir /nonexistent pgrelay

--system places it in the system UID range and outside password-ageing policy. --no-create-home plus a non-existent home means there is no directory for anything to accumulate in. nologin as the shell means the account cannot be used interactively even with a password set (it has none). Nothing else on the host should ever run as pgrelay, which is what makes the same-user visibility of /proc a non-issue in practice — and hidepid (below) makes it a non-issue in principle.

The systemd unit

# /etc/systemd/system/pg_relay.service
[Unit]
Description=pg_relay durable event processor for PostgreSQL
Documentation=https://gitlab.com/pebble-it/pg_relay
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=pgrelay
Group=pgrelay

# Read by systemd (as root) before privileges are dropped — the file itself
# can therefore be root-owned and unreadable by the pgrelay user. See below.
EnvironmentFile=/etc/pg_relay/pg_relay.env

ExecStart=/usr/local/bin/pg_relay
Restart=always
RestartSec=5s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=pg_relay

# --- Isolation from the rest of the host -----------------------------------
NoNewPrivileges=yes            # no setuid/setgid/capability gain, ever
ProtectHome=yes                # /home, /root, /run/user are empty and inaccessible
PrivateTmp=yes                 # a private /tmp and /var/tmp, discarded on stop
PrivateDevices=yes             # a minimal /dev; no raw device access
ProtectSystem=strict           # the entire filesystem read-only except /dev, /proc, /sys
ProtectProc=invisible          # other users' processes invisible in this unit's /proc
ProcSubset=pid                 # only /proc/<pid> entries; no kernel tunables under /proc
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes     # Go needs no writable+executable memory
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
CapabilityBoundingSet=         # no capabilities at all
AmbientCapabilities=
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
UMask=0077

# --- No core dumps: memory holds resolved secrets and the pgpass contents --
LimitCORE=0

[Install]
WantedBy=multi-user.target

What each group buys you:

  • NoNewPrivileges=yes guarantees the process and anything it might spawn can never gain privileges through setuid binaries, file capabilities, or similar. The Processor spawns nothing, so this is belt-and-braces — but it is also what lets the other sandboxing directives be applied without escape hatches.
  • ProtectHome=yes and PrivateTmp=yes remove the two places a stray file could leak or be planted. The Processor keeps no state and writes no files (it has no state file by design), so it loses nothing.
  • ProtectSystem=strict mounts the whole filesystem read-only for the service. The Processor only ever reads — its binary, .pgpass, the system CA bundle — so it runs happily with no ReadWritePaths= at all. The one exception is a pg_relay.file_spool channel, which writes files: add exactly its spool directory (ReadWritePaths=/var/spool/pg_relay) and nothing wider; do not weaken to full.
  • ProtectProc=invisible with ProcSubset=pid hides every other process from this unit's view of /proc and removes the non-PID parts of /proc entirely. This is the per-service complement to the host-wide hidepid setting in the next section: the Processor cannot see others, and others cannot see it.
  • The Protect*, Restrict*, Lock* group closes kernel-facing surfaces (modules, tunables, logs, cgroups, clock, hostname, namespaces, realtime scheduling, setuid files, execution domains) that a network client has no business touching.
  • MemoryDenyWriteExecute=yes forbids memory that is both writable and executable — a common exploitation primitive. Go's runtime does not need it.
  • SystemCallFilter=@system-service allows the ordinary system-service syscall set and then explicitly removes the @privileged and @resources groups. SystemCallArchitectures=native blocks 32-bit compatibility syscalls on a 64-bit host.
  • CapabilityBoundingSet= (empty) and AmbientCapabilities= (empty) mean the process holds no Linux capabilities. It binds no ports and opens no raw sockets; it needs none.
  • RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 is the minimum for a TCP/TLS client (and a Unix-socket client, if you ever use one): no netlink, no packet sockets, nothing else.
  • UMask=0077 means any file the process could create would be private to it.
  • LimitCORE=0 is the core-dump rule, covered fully below.

Verify on your distribution

ProtectProc= and ProcSubset= need systemd 247 or later (RHEL/Rocky 9, Ubuntu 22.04, Debian 12 and newer all qualify). On an older systemd, drop those two lines — everything else works — and rely on the host-wide hidepid instead. After installing the unit, run systemd-analyze security pg_relay.service: it scores the sandbox and names every directive still open, with the exposure each one represents. Review that list against the Processor's actual needs (a TCP/TLS client that reads three files and writes nothing) and accept or close each item consciously — the output is the audit trail for this section.

The Cloud SQL Auth Proxy exception

If the Processor connects through the Cloud SQL Auth Proxy on the same host, the proxy is a separate service with its own unit and its own user; do not run it inside the Processor's unit or as the pgrelay user. The Processor then reaches it over 127.0.0.1, which AF_INET already permits.

File layout and permissions

/usr/local/bin/pg_relay          root:root     0755   the binary — not writable by pgrelay
/etc/pg_relay/                   root:root     0750
/etc/pg_relay/pg_relay.env       root:root     0600   read by systemd, NOT by the pgrelay user
/etc/pg_relay/.pgpass            pgrelay:pgrelay 0600 read by the Processor at each connect

Two points here are easy to miss and worth the attention:

The environment file can be unreadable by the Processor's own user. systemd reads EnvironmentFile= as root while setting the service up, before it switches to User=pgrelay. The values arrive in the process's environment; the file itself never needs to be opened by pgrelay. So make it root:root 0600: even code running as the pgrelay user cannot read it back from disk, which shrinks what a process-level compromise can persist (the environment is in memory only, and hidepid and the core-dump rules protect that).

.pgpass must be readable by pgrelay, and libpq insists it be private. The Processor reads PGPASSFILE itself when it opens each connection, so this one file is owned by pgrelay with mode 0600. libpq refuses to use a .pgpass that is group- or world-readable at all. It contains the database credential only; never put provider secrets in it. With ProtectSystem=strict the Processor can read it and cannot modify it.

Install both in one step so the modes are never wrong even briefly:

sudo install -d -m 0750 -o root -g root /etc/pg_relay
sudo install -m 0600 -o root    -g root    pg_relay.env /etc/pg_relay/pg_relay.env
sudo install -m 0600 -o pgrelay -g pgrelay pgpass       /etc/pg_relay/.pgpass

and in pg_relay.env:

PGHOST=db.internal.example
PGPORT=5432
PGDATABASE=your_database
PGUSER=pgrelay
PGPASSFILE=/etc/pg_relay/.pgpass
PGSSLMODE=verify-full
PGSSLROOTCERT=/etc/pg_relay/ca.pem

PGSSLMODE=verify-full — not prefer (the sandbox default, which will silently fall back to plaintext) and not require (encrypts, but does not check who you are talking to). See Database-Side Controls.

Never PGPASSWORD

Not in the environment file, not on a command line, not anywhere. PGPASSFILE is the only supported way to give the Processor a database password. A password in the environment is visible in /proc/<pid>/environ to anyone the rules below do not stop, and a password on a command line is visible to everyone through ps.

Host settings

Two host-wide settings belong with this unit and have their own chapter: mounting /proc with hidepid=2 so other users cannot inspect the process, and disabling core dumps at the unit, systemd-coredump, and kernel layers. See Host Settings: /proc and Core Dumps.

The journal

The Processor's own log lines never carry a payload, message content, or a secret at any log level — only channel names, queue ids, log ids, and error text with URLs stripped. So the journal is safe to retain, but it still names your channels, hosts, and failure modes; keep it to systemd-journal group readers (the default) and forward it to your central log store the way you do any other service.

The one exception to "prints nothing sensitive" is deliberate and interactive: pg_relay --test-notify prints the provider's response to stdout, because seeing it is the point of the test. Run it as the pgrelay user from a shell whose history is not recorded, not from a script whose output is captured.

Rotating credentials

The two credential classes rotate differently, and the difference is worth knowing before an incident rather than during one:

  • The database credential (.pgpass) rotates live. The Processor re-reads its libpq environment — including the pgpass file — every time it opens a connection. Write the new credential into .pgpass, update the role's password (or let the cloud identity token refresh), then SELECT pgrelay.request_reload(); to make every Processor reconnect. No restart.
  • Provider secrets (_env: values) need a restart. A process's environment is fixed when it starts; there is no way to change a running process's environment from outside. Update pg_relay.env, then systemctl restart pg_relay (a rolling restart of pg_relay@1, pg_relay@2, … if you run several — the queue holds events durably in between, and PG_RELAY_STOP_NOTIFY/START_NOTIFY announce each one). pg_relay never caches a resolved value between sends, so the only copy of the old secret after the restart is the one in the file you just replaced.

The container equivalent

On ECS, ACI, or Kubernetes the unit file does not exist, but each directive has a platform counterpart, and the standard hardened-container checklist covers them. Run as a non-root UID in the image. Use a read-only root filesystem (the Processor writes nothing). Drop all capabilities and set no-new-privileges. Keep a seccomp profile (the runtime default is adequate). Never share hostPID or hostNetwork. Inject secrets from the platform's secrets store into the environment rather than baking them into the image or task definition. Core dumps are governed by the node, not the container — check the node's core_pattern and fs.suid_dumpable if a pod crash could dump. The Cloud Setup book's Running the Processor chapter has the image and task-definition examples to apply this to.

Continue to Host Settings: /proc and Core Dumps.