Ability To Do Point In Time Restore / Rebuild Of Environments

Introduction

Over the course of my consulting career, I’ve been brought into organizations to troubleshoot “weird” problems on several occasions — those stories for another time. These engagements are often fun; though, can be very challenging. The organizations where we had the most success in tracking down weird problems quickly had the following characteristics:

• Software release versioning scheme.
• Automated builds that work seamless across multiple active releases.
• Automated deployment target environment build (Infrastructure as Code, typically)
• The ability to map a release (including patches, hot fixes, etc) to a specific commit-level in source control (across one or more repos).
• The ability to track deployments to an environment to a specific date + time + release. Probably via tags that map version number including build number to a particular commit in the git repo. You may use branches to track environment deployments and releases as well — there’s a couple of ways this could be accomplished.
• The ability to recreate an environment as it existed at a specific point in the past.

If you’ve never worked in an environment that had these capabilities, it likely seems quite complex, daunting even. I’ve implemented these capabilities from scratch before. It takes time and buy-in from all Stakeholders, but it can be done reliably.

There is an ongoing cost associated with maintaining this capability in terms of time, skill sets, practice restores, and ongoing improvement. I haven’t seen anyone have this 100% baked yet. There are always edge cases that remaining unresolved — luckily, these are edge cases that don’t happen that often.

Target Use Case

I’m particularly interested in being able to:

• track and recreate a test environment configuration at an arbitrary point in time; or,
• go back to a particular version + build of the code with at least a default set of test data.

We might want to apply this same capability to a production environment, but then we are probably talking about Disaster Recovery / Business Continuity. This is not a blog post about Disaster Recovery or Business Continuity. Though, there is overlap in the these capabilities. I have previously explored DR / BC for small and medium sized organizations in my “Practical Business Continuity For Small organizations” post.

For our use case let’s assume:

• You’re responsible for maintaining the application.
• Maybe it is hosted in the cloud.
• Maybe it is hosted on-prem.
• You are using a Source Code Management system (probably git, doesn’t have to be)
• You have a robust CI/CD/CT pipeline system setup that can recreate a complete application build + test environment from scratch.

Terminology

Datastore: Generically refers to any technology that stores data that one might want to keep (filesystem, SQL database, NoSQL database, AWS S3 bucket, K8S Persistent Volumes, etc, etc).

Database: Used to generally refer to any SQL or NoSQL database in one’s system.

Point-In-Time Restore: A point-in-time restore (PITR) is the ability to return a system to its exact state at a specific timestamp. See below.

Recovery Time Objective (RTO): Per NIST SP 800–34 Rev. 1, “maximum amount of time that a system resource can remain unavailable before there is an unacceptable impact on other system resources, supported mission/business processes, and the Maximum Tolerable Downtime (MTD).” Or, how long can we afford for the system to be unavailable?

Recovery Point Objective (RPO): Per NIST SP 800–34 Rev. 1, “represents the point in time, prior to a disruption or system outage, to which mission / business process data can be recovered (given the most recent backup copy of the data) after an outage.” Or, how much data are we prepared to lose?

Point In Time Restore Capability

A point-in-time restore (PITR) is the ability to return a system to its exact state at a specific timestamp — not just “the last backup,” but 12:43:07 PM on Tuesday before someone dropped a table.

Standardized PITR approaches exist at multiple layers of the stack: database, filesystem, VM, container, and full-environment levels. The right method depends on what you’re restoring. Again, we’re interested in restoring the entire environment.

Database-Level Point-in-Time Recovery (Most Common)

This is the gold standard for transactional systems. It works by:

1. Taking a full backup.

• This could be a full backup each time; or
• A periodic full back up + a series of incremental backups.

2. Continuously recording transaction logs (WAL/binlogs/redo logs).

3. Then, on restore:

• Restore full backup
• Replay logs up to a specific timestamp or transaction ID
• If there is still data missing, the business side must walk through paper backups (or whatever your organization’s equivalent is) and reenter data to get the system back to the desired present state. Don’t have this capability? Maybe start at the beginning.

Now, the backup files have to be stored on a reliable medium (SAN, NAS, redundant local drives) and periodically backed up off site through an automated, network-based transport mechanism or someone physically moving tapes or drives to a separate location. A large, well-funded organization would be able to do both of these as part of an enterprise data backup / restore strategy.

However, what we are talking about in this post is more geared towards restoring a test environment at a particular point in time. Potentially, this type of approach could be used as the basis of a DR environment, but there will always be edge cases it cannot address such as failure of a cloud provider. Admittedly, these are unlikely, possible, but unlikely. I’ve seen well-intentioned people get wrapped around the axle thinking about edge cases in disaster recovery scenarios; meanwhile, the most basic backup and restore processes for datastores is not reliable.

Wrapped Around the Axle of Unimportant Details (Yes, AI Generated)

From an architecture standpoint, the fewer databases, datastores, and technologies / products one has to address backup and restore solutions for, the easier and more reliable the recovery capability will be. I’ve been in a small organization (IT department with less than 30 people) that had dozens of databases spread across ten different technologies / vendors / products. DR was an expensive, perplexing void for them that had never had a test exercise that was even remotely successful. Changing anything else in their environment was also a challenging, expensive endeavor. That is a failure of leadership, discipline, and architecture. IT departments need a certain level of command and control to be successful, but not so much that it becomes difficult to get basic functions accomplished. Organizations tend to cluster at one extreme or the other.

Examples

• PostgreSQL — WAL archiving + recovery target time
• MySQL — Binary logs + --stop-datetime
• Microsoft SQL Server — Full + differential + transaction log backups
• Oracle Database — Archived redo logs + RMAN
• MongoDB — Oplog replay

Standardized Pattern

• Full backup schedule (daily/weekly)
• Incremental/differential backups
• Continuous log shipping
• Defined Recovery Point Objective (RPO)
• Practice the restore process on a regular basis.

If your database technology cannot support a live backup (meaning the database remains available while the backup is occurring), your ability to fully implement this pattern will be hampered. At the very least, you will experience an outage during the backup. Choose technologies and design systems around this not happening.

For our use case, let’s assume that you have a base set of test data that is loaded into a database for each new enironment build. We would need to rely on a database backup for doing a PITR restore of a test environment.

Don’t forget that all backups should be encrypted (gaps here are one of my pet peeves) and:

• Encryption keys must be properly secured
• Encryption keys must be unique between production and non-prod systems (ideally, unique between each environment)
• Encryption keys must be unique between datastores.
• Other best practices in the “Datastore Security Requirements” post.

Common failure points in this type of PITR architecture includes:

• No log retention (only full backups)
• Snapshots mistaken for backups
• Never testing restores
• Retention windows too short
• Backup encryption keys lost
• Restore permissions unclear during incident

If you cannot:

• Specify the timestamp,
• Execute the restore,
• And verify data integrity within your RTO,

You do not truly have point-in-time recovery; you just have hope…

Filesystem Snapshots (Storage-Level PITR)

Many applications will have some need for storing data directly on a file system. It might just be images and static content for a website.

Modern filesystems support snapshotting.

Mechanism

This works via:

• Copy-on-write snapshot captures filesystem state at a point in time
• Later snapshots preserve incremental changes

Examples

Examples of filesystems that support this capability:

• ZFS
• Btrfs
• LVM

Each filesystem that supports snapshotting has a command that can be used to trigger the snapshot. This allows you to write your own scripts to do the backups or tie into backup / restore management solutions. Enterprise backup software can be expensive and complex to manage. Small organizations may not have the resources to deploy that type of solution; so, scripts it is.

Use Case

• Application servers
• File shares
• VM disk volumes

Snapshots are fast but:

• Not application-aware
• Can cause corruption if taken mid-transaction without coordination

Always quiesce apps or use pre/post snapshot hooks to shutdown an application during the snapshot to avoid changes during the backup.

So, make sure those scripts you are writing actually stop the application on the local server / VM, take the VM out of the load balancer rotation, or otherwise route traffic around the local instance before running the snapshot. Or, the enterprise backup software will provide a framework (hooks) for performing these actions for you.

Virtual Machine Snapshots

Hypervisors can snapshot entire virtual machines.

Just like with the filesystem examples above, you will generally have to route traffic around the VM being snapshotted. Or, another redundancy mechanism (for example) has to be present that will allow the VM to continue to function during the backup, but these approaches tend to be expensive and complex — load balancer mechanisms ten to be much cheaper. If your application has complex, server-side state that must be maintained, cannot failover, and is a giant single-point-of-failure, it might be time to redesign, decompose, and remove the personnel that were responsible for constructing such a thing to begin with. If that system became that way through evolution without the benefit of natural select over the course of several decades, it’s a disaster waiting to happen at the most inopportune time.

Examples

• VMware vSphere
• Microsoft Hyper-V
• KVM

Captures:

• Disk state
• Memory (optional)
• VM configuration

Good for:

• Short-term rollback
• Patch testing

This capability is not ideal for long-term backups unless integrated with backup software. It doesn’t really help us with our point in time restoration either.

Cloud-Native PITR

Most major cloud providers offer managed PITR.

Examples

• Amazon RDS — Automated backups + transaction logs
• Azure SQL Database
• Google Cloud SQL
• Amazon S3 — Versioning + object-level restore

Cloud PITR usually includes:

• Retention window (e.g., 7–35 days)
• Timestamp restore selection
• Managed log retention

This is currently the most operationally standardized model.

Application-Level PITR

Some SaaS and enterprise systems include built-in restore capabilities.

Examples:

• Salesforce
• ServiceNow

These systems:

• Maintain internal version history
• Allow record-level restoration
• Often lack full environment rewind

Given what we are trying to accomplish in this blog post, it’s unlikely that this type of service is core to what we’re trying to restore. Though, it could be a dependency, but probably one that I would consider out-of-scope most of the time.

Container & Kubernetes Environments

Stateful workloads require volume-level or database-level backup. You can use:

• Velero for Persistent Volumes
• etcd snapshotting for cluster state

Don’t forget to backup the filesystem where the Velero backups and etcd snapshots are being stored. Especially, if these are all stored on the same machines / VMs where the K8S cluster is running.

Some best practice in this space are:

• Separate stateless services from stateful data
• Backup persistent volumes independently

Containers are treated as stateless and can always be pulled fresh from a container registry. I discuss container registry security in my “SOFTWARE SUPPLY CHAIN SECURITY: CI/CD/CT PIPELINES AND SECURITY TOOLS — PART 1” post. Container tags must be used to map the image build version to the commit level that was used to create it. This is discussed below.

A conscious decision should be made in your system regarding whether a fresh copy of the container image is pulled at each container / pod startup or images should be cached.

Other Application Artifacts

Maybe your running a serverless application. Maybe you’ve some type of app bundle that has to be deployed to a proprietary cloud PaaS platform. Maybe you have a JEE EAR / WAR file. Maybe you’ve got an API Gateway configuration that needs to be deployed.

Whatever components make up your application, I’m assuming that there:

• an artifact (even if a configuration file) that gets built.
• that artifact is deployed / pushed to the hosting environment.
• the code / configuration for this artifact is stored in a git repo.
• the deployment process is automated

If those things are all true, then the basic ideas I’m describing here for a Point-In-Time-Restore of a test environment can be achieved.

Infrastructure-As-Code Recovery

Unless you are running on top of a PaaS or SaaS solution, you’ve most likely got some infrastructure concerns that must be addressed. You should be able to delete and recreate your environment’s infrastructure at will. For full environment restoration:

• Use version-controlled templates
• Redeploy environment
• Restore data layer separately

Common tools used to achieve this include:

• Terraform
• Ansible
• CloudFormation

This approach supports:

• Reproducible infrastructure
• Disaster recovery automation

The scripts / code / configuration to support this will, generally, be kept in git repositories.

Using Git Effectively

Git tags can be used to map a commit-level to a versio + build number. The exact details vary by Source Control Management system. Most of the major build automation platforms offer some type of unique build run identifier — often accessible through environment variables.

I discuss version numbering in the SOFTWARE SUPPLY CHAIN SECURITY: CI/CD/CT PIPELINES AND SECURITY TOOLS — PART 1” post. The exact details are somewhat arbitrary, but a build number that uniquely describes every build is critical for this to work. Likewise, there needs to be a unique deployment number. Both of these can often be achieved with a unique pipeline run identifier (most of the major SCM + build systems support this).

• Each CI pipeline should generate a timestamped + version / build number git tag that is applied to the relevant commit level.
• Each CD pipeline should generate a similar tag that calls out when the deployment was done.

This style of git tag usage ties an artifact build to your commit level. For containers, a similar tag naming convention should tie the build number to the container image. It should be easy to identify which pipeline run generated which container image. It should be easy to identify which commit level generated which container image. Likewise, it should be easy to identify when a particular container image was deployed to an environment.

For other artifact types, through naming conventions, tags, or whatever mechanisms are available, it should be possible to tie an artifact version (build number) to commit-level and when it was deployed to an environment.

This will require your pipelines to have sufficient access to your repo(s) to apply these tags.

Tagging of git repos, container images, and other deployment artifacts ties release management, patch management, and the deployment processes together.

It will also allow you identify what build was deployed in an environment at a specific point in time.

If your application and infrastructure is spread across multiple repos, you will need to be able to tie builds and deployments across all repos to a common version, build, or deployment number / identifier. In the absence of a platform feature that handles this, you can coordinate builds across multiple repos with one “build” repo that contains pipelines for coordinating builds and deployments across all of the application components (repos). This top-level build + deploy pipeline must pass down a common version / build number that can be used with the other repos. Again, the details depend very much on your build / deployment automation platform.

I’ve always liked using environment branches to track what the currently deployed code base to each environment is. I’m not going to attempt to get into a religious debate about the various branching strategies available within git. I’ve tried to write this blog post independent of that.

To Restore

Do the following:

• Identify the time and environment you want to recreate.
• Restore datastores using the processes described above.
• find the most recent deployment by timestamp + environment + deployment tag in the git repo.
• If the corresponding build artifacts no longer exist, rebuild them.
• Deploy the corresponding commit level (identified by release or deployment tags).
• Run standard automated tests.
• Investigate your issue.

Summary

Maintaining this concept requires a certain level of discipline.

Automate it.

Once you can do this type of PITR environment restore, it can be an invaluable tool in the troubleshooting process. Especially, if you have to take a sledge hammer approach to identifying when some obscure issue was introduced.

Notes:

• Wrapped Around the Axle Photo: Generated by ChatGPT.
• AI / GenAI / ChatGPT / etc were not used to generate the text of this article.
• I used em dashes in my writing before the current GenAI wave was a thing. Not planning on changing now..
• Names have been changed to protect the guilty.
• None of the hostname or users used in examples actually exist.
• Feel free to post any comments or suggestions below.