Achieving Application Environment Isolation

Your production environment should be isolated from your non-production environments. In fact, every application environment should be isolated from every other environment. Sounds simple in practice, but how does one accomplish this? Through a multi-layered isolation strategy:

• Network isolation
• Hardware isolation
• Virtualization isolation
• Public Cloud Provider Account (AWS) / Subscription (Azure) / Project (GCP) isolation.
• PKI Trust Isolation (what certificate authorities are trusted?)
• Encryption Key Isolation
• Identity Provider Trust Realm Isolation
• Data isolation
• Tenant Isolation (SaaS)

The “Principal of Least Privilege” must be applied at each isolation layer.

It doesn’t specifically matter what environments you have, how many you have, or what names you have given them. Maybe you have dev & live. Maybe you have dev, test, prod. What’s important is consistency and having the purpose of each environment baked into your Software Development Life Cycle (SDLC) processes (gatekeeping, checks & balances, etc).

Like any other aspect of security, implementing environment isolation and maintaining it requires discipline, diligence, and tenacity.

Terminology

Non-Production: On at least one occasion, I was asked what non-production meant. So, all bets, all hope for the future is off. So, let’s explicitly define everything. If you have { dev, qa, test, and production }environments, then { dev, qa, test } are the non-production environments and { prod } is the production environments.

Production: The environment that your customers use. Sometimes known as Live. I’ve never had anyone ask me what “production” means, but I did have someone ask me what “prod” means once.

The PreProduction Solution

Some shops will put a “preproduction” (PreProd) environment along-side the production environment. This PreProd environment will have all the same safeguards in place as the production system, but gets a copy of the production database (and other datastores) once per day (or week).

This could be potentially be a legitimate solution, but requires ongoing diligence to maintain the PreProd environment. Rather than assume you are the exception to the rule in the long-term, I suggest focusing on the alternative solutions presented later in this post.

If your PreProd environment shares a database with production, you’ve gone to the bad place and need to keep reading.

Network Isolation

OnPrem Datacenters

If you are in a traditional data center + network architecture, then ideally, the network architect has thought out the network segmentation model already. This should include a micro and macro network segmentation model. This is implemented through a variety of different mechanisms including traditional firewalls, router capabilities, switch capabilities, Software Defined Network (SDN) infrastructure, etc.

Micro Segmentation: A security strategy that divides a network into very small, isolated segments and enforces strict access controls between them — often at the workload or application level.

Macro Segmentation: The practice of dividing a network into large, coarse-grained security zones and controlling traffic between those zones using centralized controls like firewalls and routing policies. If microsegmentation is surgical precision, macrosegmentation is architectural zoning.

VLANs: Away to logically segment a physical network into separate broadcast domains without needing separate physical switches. One switch; multiple isolated networks.

Cloud Networking

On a pubic cloud, each application, each environment should generally be in its own VNet. There may be multiple VNets involved in an application’s architecture (for the same environment). This makes sense sometimes, but can also become over engineered very quickly.

An enterprise network architecture / strategy that addresses both network isolation / segmentation as well communication between applications and shared services is needed before dozens of different teams start standing things up in the cloud. Trying to add this after the fact is expensive, complicated, and will probably fail to deliver the desired results.

Kubernetes (K8S)

If you are running in K8S cluster, there are out-of-the-box networking policies that can be applied to your application pods. Start with a deny-by-default network policy. Then, explicitly grant each network connection that is needed (Principal of Least Privilege).

Each application + environment should be in its own K8S Namespace. That’s not specifically network-related, but this is the only place where K8S is discussed.

Using Istio or similar Service Mesh provides additional network security capabilities that should be considered.

Steps / planning must also be taken to give a unique source IP address to each application / environment / pod / namespace (pick one) deployed to your K8S cluster so that traditional firewalls outside of K8S can be effectively utilized to control network-level access. Having a single IP (or one IP per worker node) as the source IP for all traffic coming from a shared K8S cluster is going to have a everything-can-talk-to-everything connectivity model — don’t do that. Likewise, having a K8S cluster per application may not be the most cost effective deployment strategy for the technology.

For ingress traffic (to the cluster) having a single set of Ingress Controllers that are used by all applications / environments deployed to the cluster is not going to allow for network isolation in the traditional sense for services advertised on the cluster. If the Ingress Controllers must be shared across applications / environments, then there are other layers of environment isolation that can compensate, but there is a gap that in your architecture that should be understood.

Hardware Isolation

If your application is running on bare metal hardware, on-prem with an IaaS virtualization layer, or with a dedicated-hardware option with a Cloud Provider (expensive, but available), then you should have production and non-production environments isolated through separate hardware.

Often, from a cost perspective, you can only take this as far as separate hardware between non-prod and production; non-prod environments will still have to run on shared hardware. This is usually an acceptable compromise if they other isolation strategies discussed here are present.

Virtualization Isolation

Each environment should have separate virtual machines deployed. Those virtual machines should be running on separate hardware between production and non-production.

If there are resource or financial constraints, an acceptable compromise can be to run non-production environments together on the same virtual machines, but this is not ideal. Again, if this must be done, then other isolation strategies should be deployed to ensure there is no cross-environment chatter.

Cloud Account Isolation

If you are running on a public cloud platform, there should be isolation between applications and environments using an administrative mechanism provided bu the cloud provider.

For example, on AWS, you can have a separate AWS Account for each environment + application. Or, for larger organizations, have an Organization defined with management account and Member Accounts (sometimes called subaccounts) for each environment + application.

For Azure, you can define a separate Subscription for each application + environment.

For GCP, you can create a separate Project for each application + environment.

PKI Trust Isolation

For Internal / Private Certificate Authorities (CAs) that an organization deploys, there should be unique issuing CA certificate for each environment. Short of that, there should be an issuing CA certificate for production and non-production uses / environments.

If you want to take it one step further, have a completely unique trust chain for non-production and production, including the Root CA certificate.

I’ve never seen an organization utilize a Root CA certificate per environment. So, I’m going to call that an overkill. So, I’m going to recommend at least having a unique Issuing CA certificate between production and non-production.

Encryption Key Isolation

In case you haven’t gotten the memo, all datastores should be encrypted. This is covered in my “Datastore Security Requirements” post.

In the interest of isolation, the encryption keys must:

• be properly secured (think HSMs, AWS Secrets Manager, Google Secret Manager, Azure Key Vaults, Hashicorp Vault, CyberArk, etc) with appropriate access control.
• stored in secure locations that are unique per environment. Or, at least, have separate storage locations between prod and non-prod systems.
• be unique between production and non-prod systems (ideally, unique between each environment)
• be unique between applications. Maybe unique between datastores for very sensitive data.
• periodically be rotated. If you’ve had the same production encryption key for all databases in production since 2018, it may be time to rotate that encryption key and deploy unique keys per datastore, application, and / or environment.

IdP Trust Realm Isolation

Each environment should have an independent Identity Provider (IdP) instance deployed. Each environment should have the same access and attestation processes applied to it.

Just like with the PKI Trust Isolation recommendation, if you are not prepared to support the IdP-instance-per-environment model, at least have a unique production and non-production IdP instance.

The promotion process from one environment to another should include the creation of any IdP (objects such as groups), user attributes, application configuration within the IdP (OAuth2 client configuration), etc.

Data Isolation

Do not use production data for testing in non-production environments.

See my “No, You Shouldn’t Use Production Data For Testing” blog post.

Do not use production data for testing in non-production environments.

This is one point that should not be compromised on, but it happens all the time.

Tenant Isolation

If you are a SaaS Provider, create an isolated instance of your application for each organization. Or, at least, have a plan for how to scale into this architecture. Potential customers in regulated industries are going to insist on this.

Treat each customer’s environment the way you would a unique application. Then, apply all the principals outlined in this post to that client’s tenant. This includes the non-production environments for that client if it makes sense to expose those.

Notes:

• AI / GenAI / ChatGPT / etc were not used to generate the text of this article.
• Names have been changed to protect the guilty.
• None of the hostname or users used in examples actually exist.
• Feel free to post any comments or suggestions below.