Managing environments efficiently has become a significant challenge for DevOps engineers, platform engineers, and developers. The concept of Environment as a Service (EaaS) promises agility and scalability but is often challenged with the complexities of provisioning, de-provisioning, applying consistent configurations across namespaces, and managing Day 2 operations. Some of the challenges include ensuring isolation between teams, managing access controls and automating the environment life cycles to support CI/CD (continuous integration/continuous deployment). As organizations strive to streamline these processes, the need for a product that can offer simple, efficient, flexible and expandable solutions is very important.
In this article, we will explore how the Multi Tenant Operator by Stakater emerges as a compelling solution to the aforementioned problem that we call “Tenant as a Service” (TaaS).
Understanding the need for TaaS
Let us try to understand the meaning of "environment" in the context of Kubernetes clusters. When we think of environments, it's easy to narrowly define them as merely namespaces. However, this perspective is quite limiting and does not encapsulate the full breadth of what an environment entails. An environment, in a broader sense, combines namespaces with essential elements such as role-based access control (RBAC), quotas, and integrations with external tools like Vault and Argo CD. This comprehensive setup is crucial in today's landscape, where deploying an app on a cluster requires more than just a namespace; it needs a complete environment.
Figure 1 offers a visual representation of the TaaS model. It illustrates how each tenant within an OpenShift cluster can have multiple isolated environments, such as development (Dev), staging (Stage), and production (Prod). When building a multi-tenant cluster, this isolation becomes critical, not just for resource management, but for maintaining the necessary security and operational integrity. Such a comprehensive setup is paramount in today's landscape, where deploying an application requires more than just raw computing resources. It demands a complete environment equipped with governance, security, and CI/CD pipelines. This is where the concept of TaaS within EaaS becomes not just an option, but a necessity.
Consider the prevalent issue of enforcing default network policies across the cluster. This challenge is a common pain point for many organizations, highlighting one of the limitations of simple namespace configurations. For instance, irrespective of the application being deployed, it’s a good security practice to enforce network policies that tighten security and protect against potential vulnerabilities by controlling the traffic flow between pods. This requirement emphasizes the necessity for a robust solution that transcends basic namespace management, propelling the shift towards Environment as a Service model.
In this space, organizations increasingly leverage shared OpenShift clusters to optimize resource utilization, a practice central to the Environment as a Service model. However, the high cost of scaling clusters necessitates a strategic shift towards multi-tenancy, where TaaS emerges as a vital solution. TaaS simplifies the complexities associated with multi-tenancy—such as ensuring security, efficient resource allocation, and operational management—within these shared environments.
The Multi Tenant Operator (MTO) by Stakater is instrumental in addressing these challenges, offering a streamlined approach to manage shared cloud resources effectively and securely along with several integrations with the most common tools in the Kubernetes world, some of which will be explored in future follow-up articles.
Introducing MTO: The solution to EaaS problems
MTO stands out as a pivotal solution in the EaaS landscape. Designed to address the complex challenges of multi-tenancy, MTO leverages custom resources like tenants, quotas, and templates to streamline the management process. Its integration with key tools such as Vault and Argo CD enhances security and operational efficiency, making MTO a cornerstone for secure and scalable cloud-native environments.
A standout feature of MTO is its Template custom resource, crafted to uniformly distribute configurations or resources, such as network policies, across multiple tenants or namespaces. This capability helps us tackle the network policy issue discussed earlier. This approach not only ensures consistent security measures but also eliminates repetitive manual tasks.
In addition to the technical features mentioned so far, MTO also provides a GUI, designed to give users and administrators an overview of all the resources within their Multi Tenant Environment. MTO Console, a user-friendly dashboard, further simplifies the management of tenants and its resources, providing an intuitive interface that enhances user experience. This feature, combined with MTO's robust backend functionality, makes it an essential tool for organizations aiming to optimize their Kubernetes environments.
In this article series, we'll delve into how MTO, when combined with Argo CD, revolutionizes environment management. We'll talk about design principles, repository structures, and workflows that you can adopt within your organization.
MTO architecture
MTO architecture is designed to streamline and secure the management of multi-tenant environments within OpenShift clusters. At the heart of this system is the Tenant Operator, which orchestrates the deployment of integral services essential to the system's operations, such as PostgreSQL for data persistence, Keycloak for user authentication, Prometheus for monitoring, OpenCost for cost analytics, and kube-state-metrics for cluster state metrics. These services are not just supplementary; they are woven into the core functionality of MTO, ensuring a seamless operational experience.
Serving as the bridge between the user interface and the backend services, MTO Gateway processes and relays requests, enforcing authorization, and caching mechanisms to optimize performance. End users interact with the system via MTO Console, a React-based UI that provides a responsive and intuitive interface for managing resources, monitoring tenant activities, and configuring the environment to their specific needs. This user-centric console communicates with MTO Gateway, which in turn interacts with PostgreSQL to handle data transactions, maintaining the integrity and responsiveness of the system.
The MTO architecture not only prioritizes operational efficiency and security but also ensures that administrators have a comprehensive view of their multi-tenant landscape, all while giving users the autonomy they need within their allocated resources.
MTO hands-on: Architecture
The journey through the MTO landscape begins with the developers and DevOps professionals. When they propose changes, such as new tenant configurations or resource adjustments, they do so through Pull Requests (PRs) to the central Git Repository. This repository acts as the single source of truth and the starting point for updates to your cluster configuration.
Upon PR review and merge by the platform team, Argo CD, central to the operational workflow, detects and implements the changes. This automated deployment is crucial for maintaining an agile environment, allowing for swift and systematic updates. Peripheral components like the Group Sync Operator synchronize with identity management systems such as Keycloak, while the Patch Operator attends to OAuth Resource updates, reinforcing security.
Finally, MTO, watching the configurations enacted by Argo CD, enforces network policies, manages RBAC, quotas, and namespace creation, and any other configuration provided in the Tenant CR.
Diving into the hands-on experience, we employ Keycloak to manage our authentication and authorization realms, populating it with users from the Nordmart company for demonstration purposes. While for this hands-on repo, users are directly added to Keycloak, the architecture is designed to be versatile. In a real-world application, Keycloak offers seamless integration with LDAP services, ensuring a fluid synchronization of users and groups between your enterprise LDAP, Keycloak, and OpenShift. This triad forms a robust framework for identity management, underpinning a secure and streamlined authentication process.
This architecture is not just about deploying and managing; it's about creating a harmonious ecosystem where each tenant can thrive independently yet cohesively within a shared OpenShift environment. This balanced approach to multi-tenancy is what empowers organizations to scale their operations securely and efficiently.
Getting hands-on with MTO
Embracing the GitOps philosophy, MTO facilitates a declarative way to manage your Kubernetes and OpenShift environments through version-controlled configurations. This section briefly outlines how to kickstart your journey with MTO, emphasizing the GitOps approach for managing tenants and quotas. For a deep dive, our MTO hands-on GitHub repo provides comprehensive guidance.
Quick start
Throughout the series, we will use a fictitious e-commerce company named NordMart. We will explore how this company manages their tenants and other configurations and how they use their OpenShift cluster to efficiently allocate cluster resources among their teams.
MTO installation
Installing MTO is quite straightforward via the OperatorHub. Search for "Multi Tenant Operator" on the OperatorHub on your cluster and follow the instructions to complete the installation.
For more details, please click on this link.
Embracing GitOps with MTO
Throughout this series and our mto-hands-on repository, we leverage Argo CD for continuous deployment and synchronization of configurations. The entire lifecycle of tenants, quotas, and complex integrations are managed through pull requests (PRs) to your Git repository. This ensures that all changes are version-controlled, reviewed, and auditable.
Gitops-ified management: Everything from prerequisites installation to resource creation and syncing is defined in YAML files within your Git repository. Argo CD continuously monitors this repo, applying any changes to your cluster, ensuring that the desired state is always maintained.
PR-based operations: Every change, whether it's adding a new tenant, adjusting quotas, or updating various other configurations, is done through PRs. This fosters collaboration, review, and transparency across teams.
Configuration examples
To give you a taste of how tenants and quotas are defined, here are simple examples:
Quota:
---
apiVersion: tenantoperator.stakater.com/v1beta1
kind: Quota
metadata:
name: medium
spec:
limitrange:
limits:
- max:
cpu: '4'
memory: 500Mi
min:
cpu: 200m
memory: 100Mi
type: Pod
resourcequota:
hard:
configmaps: '100'
requests.cpu: '5'
requests.memory: 5Gi
secrets: '100'
services: '100'
services.loadbalancers: '2'Tenant:
---
apiVersion: tenantoperator.stakater.com/v1beta2
kind: Tenant
metadata:
name: apollo
spec:
#rbac
owners:
users:
- jasper-eriksson@nordmart.com
- mto@stakater.com
editors:
users:
- ava-li@nordmart.com
viewers:
users:
- milo-chaudhary@nordmart.com
#quota
quota: medium
#common metadata
commonMetadata:
annotations:
stakater.com/annotations: apollo
labels:
stakater.com/label: apollo
namespaces:
withTenantPrefix:
- build
- dev
- prod
onDelete:
cleanAppProject: true
cleanNamespaces: true
#specific metadata
specificMetadata:
- annotations:
stakater.com/annotations-spec: apollo
labels:
stakater.com/label-spec: apollo
namespaces:
- build
- annotations:
stakater.com/annotations-spec-2: apollo
labels:
stakater.com/label-spec-2: apollo
namespaces:
- devThese YAML files are stored in specific directories (mto-config/tenants for tenants and mto-config/quotas for quotas) within your Git repository, managed by Argo CD.
You can learn more about quotas and tenants from here and here respectively.
Dive deeper and try it out
We have just scratched the surface of what’s possible with MTO and GitOps. To truly leverage the power of MTO, we encourage you to explore the MTO Hands On Repository. It is designed to guide you through setting up a fully GitOps ready environment, from basic installations to setting up tenant and quota configurations.
By embracing MTO with GitOps, you can streamline your cloud-native environment management, making it more secure, efficient, and scalable. So, why wait? Dive into the hands-on repo and start making your single OpenShift cluster into a multi-tenant capable environment right away.
Conclusion and what's next
As we've explored throughout this article, adopting the Multi Tenant Operator (MTO by Stakater) for Tenant as a Service (TaaS) in cloud-native environments offers a multitude of benefits. MTO simplifies the complex landscape of managing OpenShift environments, providing a streamlined, efficient, and secure approach to tenant and resource management. By leveraging custom resources for tenants, quotas, and templates, MTO ensures that your cloud-native environments are not only scalable but also resilient and compliant with your organizational policies.
The integration of MTO with GitOps practices further enhances operational efficiency, offering a declarative, version-controlled approach to environment management. This alignment with GitOps principles ensures that your infrastructure evolves safely and predictably across its lifecycle, from provisioning to de-provisioning and ongoing Day 2 operations.
As we look to the future, our series will continue to delve deeper into the capabilities of MTO. In our next article, we'll explore advanced features and integrations of MTO, such as usage of templates for mostly occurring common operations, security enhancements with Vault integration, and efficient user management through OpenLDAP integrations. Stay tuned to uncover how MTO can further elevate your cloud-native operations, making them more dynamic, secure, and user-friendly.
We invite you to continue your journey with MTO by visiting our website for more information, accessing our comprehensive documentation, and exploring the MTO Hands-On repository for practical, step-by-step guides.
