Disaster Recovery and Backups

In the last article, I walked through what changes when a SaaS grows past a single server: load balancers, stateless app instances, a database that starts to strain under real concurrency. All of that assumes the database is still there in the morning. This article is about the far less glamorous question underneath it: what happens when it isn't.

This is part of the Full Stack SaaS Masterclass series, and it sits in the production module for a reason. Disaster recovery is one of those topics teams postpone until the week after they needed it. A dropped table, a bad migration that runs in production instead of staging, a misconfigured terraform destroy, a region-wide AWS outage: none of these are exotic. They are the kind of thing that happens to an ordinary team on an ordinary Tuesday. The only variable you control in advance is how bad the outcome is.

I want to be direct about scope here. Full multi-region active-active failover, cross-cloud redundancy, and a dedicated DR site are real strategies, and they belong to companies with the compliance requirements or the incident history to justify them. For most SaaS products in their first few years, the goal is simpler: know your recovery point and recovery time targets, automate backups that actually meet them, and prove the restore works before you need it for real.

Recovery point and recovery time are the only two numbers that matter

Before touching backup tooling, get specific about two numbers, because every technical decision downstream follows from them.

Recovery Point Objective (RPO) is how much data you can afford to lose, measured in time. If your RPO is fifteen minutes, a once-a-day backup is a liability dressed up as a safety net. If your RPO is 24 hours because your product genuinely tolerates losing a day of writes (an internal reporting tool, say), a nightly pg_dump might be entirely sufficient.

Recovery Time Objective (RTO) is how long you can be down while you fix it. An RTO of thirty minutes rules out any process involving a human manually downloading a file from S3, spinning up an EC2 instance by hand, and running commands from memory under pressure. An RTO of a few hours means a documented, semi-manual process is fine, as long as someone has actually run it before.

Write these numbers down and get them approved by whoever owns the product, not just engineering. I've seen teams build an elaborate DR setup for an RPO nobody asked for, and I've seen teams skip backups entirely for a product where a single lost signup would have been a real financial hit. The number should come from the business impact of data loss, not from what feels technically satisfying to build.

Once you have RPO and RTO, the rest of this article is mechanics.

Backing up PostgreSQL: logical dumps versus physical backups

Postgres gives you two fundamentally different backup strategies, and conflating them is a common mistake.

A logical backup (pg_dump / pg_dumpall) exports your data as SQL statements or a custom-format archive. It's portable across Postgres versions and easy to restore into a differently configured instance. It's also simple to reason about. Its downside is that it captures a single point in time and nothing between backups, and restoring a large database from a logical dump can take hours because it's replaying inserts and rebuilding indexes from scratch.

#!/usr/bin/env bash
set -euo pipefail

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
DUMP_FILE="/var/backups/postgres/saas-${TIMESTAMP}.dump"

pg_dump \
  --host="$DB_HOST" \
  --username="$DB_USER" \
  --format=custom \
  --file="$DUMP_FILE" \
  "$DB_NAME"

aws s3 cp "$DUMP_FILE" "s3://acme-saas-backups/postgres/$(basename "$DUMP_FILE")" \
  --storage-class STANDARD_IA

find /var/backups/postgres -type f -mtime +7 -delete

A physical backup copies the actual data files, combined with continuous WAL (write-ahead log) archiving. This is what makes point-in-time recovery (PITR) possible: instead of restoring to "last night at 2am," you can restore to "3:14pm today, one minute before the bad migration ran." That precision is the difference between losing a day of customer data and losing a minute of it. Any team with a tight RPO should be doing this instead of relying on nightly dumps alone.

# postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'aws s3 cp %p s3://acme-saas-backups/wal/%f'

#!/usr/bin/env bash
# nightly base backup, paired with continuous WAL archiving above
set -euo pipefail

TIMESTAMP=$(date +%Y%m%d-%H%M%S)
pg_basebackup \
  --pgdata="/var/backups/postgres/base-${TIMESTAMP}" \
  --format=tar \
  --gzip \
  --checkpoint=fast \
  --wal-method=stream

aws s3 cp "/var/backups/postgres/base-${TIMESTAMP}" \
  "s3://acme-saas-backups/postgres/base-${TIMESTAMP}" --recursive

Restoring to a point in time then looks like taking the most recent base backup, replaying archived WAL segments up to a target timestamp:

# recovery configuration on the restored instance
restore_command = 'aws s3 cp s3://acme-saas-backups/wal/%f %p'
recovery_target_time = '2026-07-21 15:14:00+00'

If you're on a managed service like RDS or Aurora, you get this mechanism largely for free through automated backups and PITR, which is a genuinely strong argument for using a managed database in the early years of a SaaS. Reimplementing WAL archiving correctly is not where I'd want a small team spending its scarce engineering time.

Redis: decide what actually needs to survive a restart

Redis gets treated as an afterthought in most DR plans, and for a lot of SaaS workloads that's correct. If Redis only holds sessions, rate-limit counters, and cached query results, none of that is a disaster if it's gone. Sessions force a re-login, caches repopulate from Postgres on the next request, rate limits reset to zero. Annoying, not catastrophic.

The problem is when Redis quietly becomes a source of truth: a distributed lock coordinating billing jobs, a queue holding unprocessed webhook payloads, a leaderboard nobody wants to recompute. If any of that lives only in Redis, you need real persistence, not just the default in-memory behavior.

# redis.conf: RDB snapshots for a fast, coarse backup
save 900 1
save 300 10
save 60 10000

# AOF for a tighter recovery point, replaying every write
appendonly yes
appendfsync everysec

RDB snapshots are cheap and fast to restore but can lose up to the snapshot interval's worth of writes. AOF with everysec bounds your loss to roughly one second of writes at the cost of a larger file and slower restarts. For most SaaS use cases, RDB snapshots shipped to S3 on a schedule are enough, precisely because nothing durable should be living exclusively in Redis in the first place. If you find yourself reaching for stronger Redis durability, treat it as a signal to move that data into Postgres rather than over-engineering Redis into a database it was never meant to be.

The pitfall nobody budgets time for: untested restores

Here's the uncomfortable truth about backups: a backup you haven't restored is a hypothesis. It isn't a safety net. I've seen teams discover, mid-incident, that their nightly dump had been silently failing for weeks because a credential rotated and nobody updated the cron job, or that a "successful" backup was zero bytes because the disk filled up. The backup job reporting success and the backup actually being restorable are two different claims, and only one of them gets tested by default.

The fix is to automate verification of the restore, since automating the backup alone proves nothing. A scheduled job that pulls the latest backup, restores it into a throwaway database, and runs a basic sanity check catches exactly the failures that matter.

import { Injectable, Logger } from '@nestjs/common';
import { Cron, CronExpression } from '@nestjs/schedule';
import { exec } from 'node:child_process';
import { promisify } from 'node:util';
import { S3Client, ListObjectsV2Command, GetObjectCommand } from '@aws-sdk/client-s3';
import { createWriteStream } from 'node:fs';
import { Readable } from 'node:stream';
import { pipeline } from 'node:stream/promises';

const execAsync = promisify(exec);

@Injectable()
export class BackupVerificationService {
  private readonly logger = new Logger(BackupVerificationService.name);
  private readonly bucket = 'acme-saas-backups';
  private readonly prefix = 'postgres/';

  constructor(private readonly s3: S3Client) {}

  @Cron(CronExpression.EVERY_DAY_AT_6AM)
  async verifyLatestBackup(): Promise<void> {
    const key = await this.getLatestBackupKey();
    const localPath = `/tmp/${key.split('/').pop()}`;
    const restoreDb = `restore_check_${Date.now()}`;

    try {
      await this.downloadBackup(key, localPath);
      await execAsync(`createdb ${restoreDb}`);
      await execAsync(
        `pg_restore --no-owner --dbname=${restoreDb} ${localPath}`,
      );

      const { stdout } = await execAsync(
        `psql -d ${restoreDb} -t -c "select count(*) from organizations;"`,
      );
      const orgCount = parseInt(stdout.trim(), 10);

      if (!orgCount || orgCount < 1) {
        throw new Error(`Restored backup has ${orgCount} organizations, expected at least 1`);
      }

      this.logger.log(`Backup ${key} verified: restored ${orgCount} organizations`);
    } catch (error) {
      this.logger.error(`Backup verification failed for ${key}`, error as Error);
      throw error;
    } finally {
      await execAsync(`dropdb --if-exists ${restoreDb}`).catch(() => undefined);
    }
  }

  private async getLatestBackupKey(): Promise<string> {
    const result = await this.s3.send(
      new ListObjectsV2Command({ Bucket: this.bucket, Prefix: this.prefix }),
    );
    const objects = (result.Contents ?? []).sort(
      (a, b) => (b.LastModified?.getTime() ?? 0) - (a.LastModified?.getTime() ?? 0),
    );
    if (!objects[0]?.Key) {
      throw new Error('No backups found in S3 bucket');
    }
    return objects[0].Key;
  }

  private async downloadBackup(key: string, destination: string): Promise<void> {
    const response = await this.s3.send(
      new GetObjectCommand({ Bucket: this.bucket, Key: key }),
    );
    await pipeline(response.Body as Readable, createWriteStream(destination));
  }
}

Wire failures from this job into whatever pages your team, the same way you'd page for a production outage. A backup that quietly stopped working three weeks ago is arguably worse than having no backup at all, because it comes with false confidence attached.

Beyond the automated check, schedule an actual manual restore drill on a real cadence, quarterly is a reasonable starting point. Have someone unfamiliar with the exact steps run the runbook from scratch. That's usually when you find the step that says "ask Priya for the credential" and Priya is on vacation.

A runbook you can actually follow at 3am

When the incident happens, the last thing you want is to be improvising commands from memory while stakeholders are asking for updates. Write the runbook now, while you're calm, and store it somewhere that survives the disaster it describes (not only in the production system that just went down).

A minimal runbook should cover, in order: how to declare an incident and who gets paged, where the latest backup lives and how to identify the correct one, the exact restore commands for both the database and any stateful services, how to point the application at the restored database, how to verify data integrity before reopening traffic, and how to communicate status to customers. Each step should be copy-pasteable, with real bucket names and real commands, not placeholders to fill in during the emergency.

Treat the runbook as a living document. Every drill or real incident should produce at least one edit to it. A runbook nobody has updated in a year is nearly as risky as one that never existed, because it was probably written for infrastructure that no longer matches production.