Designing Kubernetes Platforms for Multiple Tenants and Environments

AUTHOR

Leandro Dardick

Principal Software Engineer and Architect

With some of our team attending KubeCon + CloudNativeCon North America this week, Kubernetes has naturally been a hot topic for many of us at Southworks. While the technology provides essential functionality, there are a few factors to consider to ensure your organization gets the most out of it.

Kubernetes is often presented as a universal layer: a platform designed to run anywhere and be shared across an organization. However, teams that have used it at scale know that both claims come with caveats.

Kubernetes offers two important benefits. The first is multi-tenancy: the ability to consolidate workloads, improve resource utilization, and potentially reduce infrastructure costs. The second is portability: the ability to abstract some of the underlying infrastructure so workloads can run across cloud and on-premises environments.

Both promises are real, but neither is automatic. Platform teams must decide deliberately where to draw isolation boundaries and how much portability they actually need.

‍

The multi-tenancy balancing act

Organizations consolidate workloads for several reasons, including cost, resource utilization, and operational simplicity. A shared platform can avoid the duplicated infrastructure and idle capacity that may come with running dozens of separate clusters. It can also give platform teams a more centralized way to manage upgrades, monitoring, and security policies.

However, consolidation also concentrates workloads and dependencies. An incident affecting shared infrastructure can have consequences for multiple tenants, and maintenance or configuration changes may need to account for a larger group of users.

These considerations, alongside security, can pull platform design in different directions. Packing more tenants onto shared resources can improve utilization, but it also increases the importance of strong isolation controls. Separating tenants more aggressively can reduce some risks and limit the impact of failures, but it may introduce duplicated infrastructure, additional operational overhead, and higher costs.

There is no universal trade-off formula. The right balance depends on the nature of the tenants, how much they trust one another, the sensitivity of their workloads and data, and their regulatory and operational requirements.

To determine the best fit, organizations should start by asking: Who are the tenants, how much do they trust one another, and what level of data and workload isolation do they require?

‍

Drawing the isolation boundary

Three broad approaches organizations can use to separate tenants on Kubernetes are namespace-based isolation, cluster-based isolation, and sandboxed workload isolation. They address different risks and operational needs, and they are not necessarily mutually exclusive. For example, sandboxing can add another layer of protection to workloads that already run in separate namespaces.

It is also important to distinguish between control-plane and workload isolation. Tenants sharing a cluster generally share its Kubernetes control plane and some cluster-level resources. Workloads may also share worker-node infrastructure, including the host kernel when they run on the same Linux node. The level of separation therefore depends on the architecture and the controls implemented around it.

Other approaches, such as dedicated worker nodes and virtual control planes, can provide additional separation. The appropriate combination depends on the tenant's requirements.

For an overview of these options, see the official Kubernetes multi-tenancy documentation.

1. Namespace-based isolation

In this model, tenants share a Kubernetes cluster, but their workloads and resources are organized into separate namespaces. Namespaces provide a logical boundary within a cluster; they are not, by themselves, a complete security boundary.

Kubernetes provides several features that platform teams can combine to strengthen namespace-based isolation and manage shared resources:

· Role-Based Access Control (RBAC): Controls which users and service accounts can access or modify Kubernetes resources.

· NetworkPolicies: Define which network connections are allowed between pods and other endpoints, provided the cluster's network plugin supports and enforces them.

· ResourceQuotas: Limit the aggregate amount of resources, such as CPU and memory requests and limits, that can be allocated within a namespace.

· Pod Security Admission: Applies security standards to pods to restrict risky configurations.

· Resource requests and limits: Help control how workloads request and consume CPU and memory.

· PriorityClasses: Assign scheduling priorities to pods, helping Kubernetes determine which workloads should be scheduled first when resources are constrained. Depending on their configuration, higher-priority pods may also preempt lower-priority pods to obtain the resources they need.

PriorityClasses are particularly useful when different workloads share a cluster but have different levels of operational importance. For example, platform teams can give critical services higher scheduling priority than less time-sensitive workloads.

However, priority does not reserve capacity or guarantee performance. PriorityClasses are cluster-scoped resources, so their definitions and use should be governed carefully to prevent one tenant from unfairly displacing another tenant's workloads.

These controls need to be configured together. Teams should also consider cluster-scoped resources, permissions, secrets, storage, and other services that may be shared beyond namespace boundaries.

Namespace-based isolation can be an efficient option when tenants are internal teams with similar security and compliance requirements and a reasonable level of mutual trust. It can support a high density of tenants without requiring a separate cluster for each one.

However, tenants still share the cluster's control plane and may share worker nodes, underlying infrastructure, and cluster-wide resources. A namespace alone does not provide the same boundary as a separate cluster or a virtual machine.

Relevant documentation includes RBAC, NetworkPolicies, ResourceQuotas, Pod Security Admission, resource management, and Pod Priority and Preemption.

2. Cluster-based isolation

With cluster-based isolation, each tenant—or group of tenants—gets a dedicated Kubernetes cluster. This creates a stronger administrative and operational boundary between tenants and can reduce the blast radius of certain cluster-level incidents.

Dedicated clusters also allow teams to tailor Kubernetes versions, configurations, policies, and supporting tools to different tenant requirements. This can be useful when external customers have different operational needs, or when workloads require stronger separation.

However, separate clusters do not automatically guarantee complete isolation. Clusters may still share cloud accounts, networks, identity systems, storage services, management tooling, or other infrastructure. Those dependencies need to be considered when defining security and compliance boundaries.

The model also introduces additional overhead. Every cluster needs to be provisioned, monitored, secured, upgraded, and maintained. Without automation and consistent platform management, organizations can end up with cluster sprawl and duplicated operational effort.

Cluster-based isolation is often appropriate for external tenants with different requirements, workloads with higher isolation needs, and organizations operating under demanding regulatory or security constraints. Whether it satisfies a particular compliance requirement depends on the full architecture and the controls around it—not simply on the number of clusters.

3. Sandboxed workload isolation

Sandboxed isolation adds another layer of separation at the workload level. Tenants can continue to share a Kubernetes cluster, while selected workloads run using a more isolated execution environment rather than relying solely on the host's standard container isolation.

Some sandboxed runtimes use lightweight virtual machines. For example, Kata Containers runs workloads in lightweight VMs, providing a separate kernel boundary for the workload. Other approaches, such as gVisor, use a user-space kernel to add isolation between applications and the host operating system. These technologies have different architectures and should not be treated as interchangeable.

Sandboxing can be a useful option for platforms that execute untrusted or third-party code. Examples include SaaS platforms that run code submitted by customers, developer environments, and systems in which AI agents execute code they have generated.

This approach can offer stronger workload isolation while retaining some of the operational benefits of a shared Kubernetes platform. However, it does not replace other controls. Namespaces, RBAC, network policies, resource limits, and appropriate cluster security configurations remain important.

Compatibility is another consideration. Workloads that require low-level system access, access to node devices, or particular hardware capabilities—such as GPUs—may need additional configuration, custom micro VM images for VM-based sandbox runtimes, or a different execution model. Some workloads may not be compatible with a particular sandboxed runtime at all.

These technologies can also introduce performance and operational trade-offs. Teams should validate workload compatibility, hardware support, performance requirements, and the skills needed to operate the chosen runtime before adopting it at scale.

Importantly, sandboxed and standard workloads can coexist within the same Kubernetes cluster. For example, a platform team might configure one set of node pools to use a sandboxed runtime for untrusted code, while another set uses a standard container runtime for workloads that do not require the additional isolation.

Kubernetes scheduling mechanisms, including node affinity, node selectors, taints and tolerations, can help place workloads on the appropriate node pools. RuntimeClass can then be used to select the intended runtime handler, provided the relevant runtime is configured on the destination nodes.

This allows organizations to apply stronger execution isolation selectively rather than requiring every workload to run in a sandbox.

Sandboxing is therefore not a universal middle ground between namespaces and dedicated clusters. It is an additional isolation mechanism that can be valuable when the risk profile of particular workloads justifies the added complexity.

For further details, see Kubernetes RuntimeClass, Kata Containers, and the gVisor architecture guide.

‍

Choosing the right isolation model

The right approach depends on the level of trust between tenants, the sensitivity of their workloads, and the operational effort an organization is prepared to take on.

These models can also be combined. For example, an organization might use namespaces for general workload organization, dedicated clusters for tenants with stronger isolation requirements, and sandboxed runtimes for workloads that execute untrusted code.

The same product or organization may also offer different isolation models to different customers or service tiers.

For example, a SaaS platform could offer a standard tier in which customers share a Kubernetes cluster but are separated through namespaces and appropriate security policies. A higher-isolation tier might add sandboxed runtimes for customer workloads, while an enterprise tier could provide a dedicated Kubernetes cluster.

This allows the platform to balance infrastructure efficiency with different customer requirements, including security, compliance, performance, and operational independence.

These tiers should reflect the actual guarantees of the architecture rather than implying that a particular tier automatically satisfies a compliance standard or provides complete isolation.

The goal is not to select the most isolated architecture in every situation. It is to apply the appropriate boundary to each workload and tenant, based on an explicit understanding of the risks.

‍

Separating development, staging, and production

Tenant isolation is only part of the platform design. Organizations also need to decide how to separate application environments.

Development, testing, staging, and production workloads have different requirements. Development environments may need flexibility and rapid iteration, while production environments typically require tighter access controls, more predictable resource allocation, and stronger change-management processes.

Some organizations use separate namespaces for different environments, while others choose separate clusters—particularly when production workloads require stronger isolation or when development activity should not affect production operations.

Whichever model is selected, teams should define clear access policies, network boundaries, resource controls, and deployment processes. Production credentials and sensitive data should not be exposed to less trusted environments, and environment separation should be enforced through platform configuration rather than relying only on naming conventions or team practices.

The appropriate design depends on the organization's risk profile, compliance requirements, and operational model. A shared cluster can be sufficient in some cases, but it should be a deliberate decision supported by appropriate controls.

‍

The portability question

Kubernetes is often presented as a way to move workloads between cloud providers or from the cloud to an organization's own data center with limited rework. In theory, it achieves this by abstracting much of the infrastructure underneath it.

In practice, Kubernetes standardizes interfaces and APIs, not the entire environment. The same Kubernetes resource definitions and commands may work across different platforms, but the networking, storage, identity systems, and managed services beneath them can vary significantly.

Those differences may remain hidden while an application runs in one environment. They become more apparent when a workload needs to migrate, operate across multiple clouds, or integrate with provider-specific services.

Some of the areas where teams can encounter the most friction are:

Identity and access management

Cloud providers have different mechanisms for granting workloads access to cloud services.

For example, Amazon EKS Pod Identity and Microsoft Entra Workload ID both support workload identity integrations, but they require different provider-specific configurations, permissions, and identity-management processes.

If application code directly depends on a particular provider's identity mechanism or SDK, moving that application to another environment may require additional development work.

Where practical, teams should separate infrastructure-specific authentication and authorization details from application business logic, using consistent interfaces and well-defined service integrations.

References: Amazon EKS Pod Identity and Microsoft Entra Workload ID.

Storage

Storage is one of the clearest examples of where Kubernetes portability has limits.

Different cloud providers offer different storage technologies, each with its own capabilities, performance characteristics, and constraints.

These differences go beyond how a volume is provisioned. They can affect:

· Filesystem semantics: Not every storage system provides the same POSIX behavior or guarantees. Applications may depend on particular file operations, locking behavior, consistency semantics, or other filesystem capabilities.

· Access modes: Kubernetes supports modes such as `ReadWriteOnce`, `ReadOnlyMany`, `ReadWriteMany`, and `ReadWriteOncePod`. These express different volume access requirements, but actual support depends on the storage driver and underlying technology.

· Concurrent access: Some storage systems allow a volume to be mounted for use by workloads on multiple nodes, while others have more restrictive attachment and access characteristics. Whether multiple processes can safely write concurrently also depends on the filesystem and application.

· Performance and availability: Storage systems differ in throughput, latency, scalability, replication, and failure behavior.

· Data migration: Moving an application does not automatically move its persistent data. Data may require a separate migration or replication process.

For example, Amazon EBS and Azure Disk can both be used through Kubernetes CSI drivers, but their capabilities and provider-specific configurations are not identical. Workloads requiring shared filesystem access may need a different storage service or driver from workloads that use node-attached block storage.

Kubernetes PersistentVolumeClaims provide a common way for workloads to request storage, but they do not guarantee that every storage backend offers the same features or filesystem behavior.

The application must therefore be designed with its actual storage requirements in mind. Platform teams should select and configure storage services that satisfy those requirements while keeping provider-specific provisioning details out of application code wherever practical.

References: Kubernetes StorageClasses, Persistent Volumes and access modes, Amazon EBS CSI driver, and Azure Disk CSI driver.

Networking

Kubernetes provides common networking abstractions, but the underlying implementation depends on the cluster's network plugin and cloud environment.

Providers may use different Container Network Interface (CNI) implementations, load balancer integrations, and network services. Bridging cloud networking with Kubernetes networking can require additional configuration and platform-specific knowledge.

For example, an application deployed through an AWS-specific load balancer integration may need different configuration when deployed on Azure. Ingress controllers, service annotations, IP allocation, routing, and network security settings may also differ.

These differences should ideally be handled by the platform and deployment configuration rather than being embedded directly into application logic.

References: Kubernetes Ingress, AWS Load Balancer Controller, and AKS networking concepts.

Keeping infrastructure out of application code

The broader goal of portability is to prevent infrastructure-specific details from leaking unnecessarily into application code.

Ideally, developers should be able to focus on building and running their applications as containers on Kubernetes without needing to understand every detail of the infrastructure underneath.

For a developer, the experience should be as consistent as possible: package the application into a container, define its runtime requirements, and deploy it through a standard Kubernetes workflow.

The platform should handle much of the complexity around storage provisioning, networking, workload identity, scheduling, and infrastructure integration.

This separation can be achieved through consistent Kubernetes APIs, platform-managed services, declarative configuration, Infrastructure as Code, and well-defined interfaces between applications and infrastructure.

For example, an application can request persistent storage through a PersistentVolumeClaim rather than directly provisioning a cloud disk. The platform can select the appropriate StorageClass and CSI driver for the environment.

Similarly, workload identity and networking integrations can be configured through platform-managed mechanisms rather than requiring every application team to implement provider-specific infrastructure logic.

However, abstraction has limits. Applications still need to express their actual requirements, such as storage access patterns, filesystem semantics, resource needs, and external service dependencies. Kubernetes cannot make incompatible infrastructure capabilities equivalent simply by hiding their implementation details.

The objective is therefore not to eliminate all infrastructure awareness. It is to establish a clear separation of responsibilities: application developers define what their workloads need, while platform engineers manage how those requirements are fulfilled in each environment.

This makes it easier to move applications, standardize deployment practices, and evolve the underlying infrastructure without repeatedly modifying application code.

Kubernetes can make workloads more portable and hide much of the underlying infrastructure complexity. The benefits are strongest when teams understand the limits of that abstraction and design for portability where it provides real value.

‍

Designing for the right trade-offs

Kubernetes offers organizations powerful ways to consolidate workloads and standardize how applications are deployed. But multi-tenancy and portability both require deliberate design.

The best platforms are not necessarily the ones with the most tenants per cluster, the most isolated workloads, or the fewest provider-specific dependencies. They are the ones that balance tenant trust, workload sensitivity, security, operational complexity, and business needs.

That may mean combining different isolation models within the same cluster, offering different infrastructure boundaries for different customer tiers, or using platform abstractions to keep infrastructure-specific details out of application code.

By choosing isolation boundaries intentionally and understanding where portability ends, platform teams can build Kubernetes environments that are efficient, secure, and practical to operate.

Interested in learning more about how your organization can design, modernize, or migrate its Kubernetes platforms? Get in touch with us today!

‍