I’m an avid and quick reader. Except when it comes to text books, which are too reminiscent of my hectic undergrad and post-graduate days. So, I was a bit reluctant to embark on the book that Rom Adams shared with me (authored by him, Chen Xi and Emmanouil Gkatziouras), “Kubernetes Secrets Handbook” (available here: https://www.amazon.com/Kubernetes-Secrets-Handbook-production-grade-management/dp/180512322X). But, this book read like a thriller, addressing a broad and diverse audience: from novices in the world of Kubernetes and security to those who are more experienced; from technical to laymen; and from text-book-phobics like me to people who have these books on their nightstands for light nighttime reading.
An Open Book
First of all: the introduction of the authors and reviewers helped to assure me that this book would be a serious review of the state of the art, and not some marketing mumbo-jumbo. The authors and reviewers are all respected in their industry, with years of experience.
I loved the opportunity to practice and implement real examples, but I equally appreciated that this is not mandatory, and that I was debriefed after every example.
Finally, the chapters progress and build on each other, never assuming that the layman (or the novice) have a required background. This showed respect for my experience, without giving the feeling that I was wasting my time on subjects already familiar to me.
The author gently led us through Kubernetes concepts (see page 10 for list of components), by showing the evolution from legacy through virtualization and mircroservices – culminating with the orchestrated containers, demonstrating all of these with fantastic diagrams and flows that explain what is happening (see Figure 1.2 on page 10 and Figure 1.3 on page 13).
But Some Secrets Are Good
Having described the above, the basic security configuration is explained, as an opportunity to describe briefly some of the vulnerabilities. The authors point out that there are proposed solutions, primarily using some external vault. However, the authors admit that the vault solutions also have some problems:
- These solutions address the application level, and not the ConfigMap or cross-application services.
- Resilience
The challenge of where and how to store secrets remains regardless of how the secrets are created. There are various tools to address the needs of secure storage of these secrets, such as Google GCP Secret Manager and AWS Secret Manager. Orthogonal to the question of *where* to store the secrets is the need to ensure that they are encrypted.
The elusive threat of storing secrets in the etcd database (the key-value storage) is glossed over with a seemingly benign comment buried in Chapter 3:
“Data in payloads written in etcd is not encrypted but encoded in base64, which is almost equivalent to clear text. Encrypting the data contained in the payload will protect from the aforementioned protection mechanisms, but not replace them!”
A Delicate Balance
But now that we’ve been gently led to this discovery, in Chapter 3, the authors introduce us to the KMS framework available within the kube-apiserver.
- Native encryption – enables local encryption, with a random encryption user-generated key. No additional software is required. However, this comes at the price of basic risks that a hacker could penetrate the system, retrieve the encryption key – and –voila – he has access to the etcd file.
- And that’s why the authors suggest using native encryption with an external component, either on the control plane or externally. Kubernetes kms provider (currently based on KMSv2 as a KMS provider for kube-apiserver). The Kubernetes KMS provider plugin interconnects the kub-apiserver with external KMS(s) such as Hashicorp vault. In this solution, the kube-apiserver generates a Data Encryption Key (DEK), and sends it via the KMS plugin to the KMS server to encrypt with a Key Encryption Key (KEK). The flow is simply and neatly presented, as are all flows in the book.
The authors then introduce us to a plugin which can be used to extend the kube-apiserver capabilities: Trousseau. From insider information, we also learned that other plugins are being developed in the community as we speak.
The authors warn us that this is not enough! Once again – using a simple diagram (Figure 3.5), they show the threat surfaces where potential breaching could occur:
- Payload data
- Network (transport)
- Operating system
- File system
- etcd
As this is not the focus of the book, we are pointed to other sources for information, but a short discussion is shared in Chapter 3.
Bringing It All Together
Chapter 4 is a cookbook for designers looking to debug and troubleshoot Kubernetes secrets. The chapter is well-laid out, but – full disclosure – I only skimmed through it, as this was not my goal in reading the book. Nonetheless, the tips and tricks are valuable. What I found most intriguing was the human element: it is often so easy to get lulled into a false sense of security when you troubleshoot: backdoors are opened, secrets are leaked, and, well, people are people. So – the authors wisely warn us to be cautious here, too, and remember that our goal is security!
Part II of the book is an extremely practical section: how to apply the theoretical background we got to a real live production environment. We are invited to a discussion about security, auditing and compliance. The obvious that must be stated, to understand the environment in which we are practicing:
“Most organizations assign the security domain to their respective IT departments by designating a set of limited individuals almost working in isolation.” (To me – it felt like he was accurately describing some of my colleagues – and some of my customers!).”
The problem with this approach is that these individuals tend to have a deep understanding of security, but not as good a grasp on operational reality. The delicate balance between the two is achieved by studying risk management.
We then revisit the diagram we had seen earlier, adding a dimension of discovery, assessment, analysis and execution: a DevOps approach on steroids – which grows to DevSecOps. Thus, the tools we had seen in earlier chapters are enhanced with tools that will assist us in implement this, including: Trivy, kube-bench, Compliance Operator and others.
In Chapter 6, we learn about the relevance of Disaster Recovery in the context of security. The authors remind us (again – of the obvious): Loss of data, resulting from a physical or logical disaster, is always a nightmare. But the fear of losing secrets, keys and passwords, similar to Dorothy’s fears on the way to Oz (lions and tigers and bears, oh my!), is well founded. This could result in loss of functionality and possible loss of service! These are not fairy tales; they really happen in real life! Often, this kind of DR is managed by an external KMS provider.
In the last chapter of Part II, we learn about some challenges and risks in secret management. Again – this was presented very methodologically, presenting the issues we would be likely to encounter in each phase. This is followed by a fascinating section about the real-world security risks (I told you, this book is a thriller!!). These risks include:
- Secret zero – the paradox of the single password that protects all the other passwords, that the authors call the “key to the kingdom”. Protecting this single password comes with some significant challenges: Management complexity, compliance issues and enhanced risk of malicious insiders.
- Secret access ballooning – access granted to too many users.
- Secret sprawl – too many secrets n the system
- Secret island – also known as shadow IT. As the other challenges present themselves (or as the solutions to those challenges become more complex), users may find alternative ways to access the system.
But Real Life Happens
In the last part of the book, the reader is introduced to alternatives for external secret stores, along with practical, hands-on examples of how to use them:
- Cloud Secret Store on AWS
- Cloud Secret Store on Azure
- Cloud Secret Store on GCP
- HashiCorp Vault
- CyberArk Secrets Manager
Continuing with the standards they set for themselves throughout the book, the authors present real world case studies.
A Call to Action
The book concludes with a discussion on the future of secret management in Kubernetes, and an invitation to join the conversation. This end is fitting on so many levels: as RedHat affiliates, the authors embody the culture of community as a strategy. While reading the book, I felt like I was being invited to a parlor meeting with friends, where we would strategize about the future of security, and our impact to this corner of the world. Indeed, that is what this is – and I invite you all to join this endeavor, too.
