Kubernetes Secrets Security

I was always of the belief that secrets within Kubernetes (k8s) are secure. How wrong I was! After a recent meetup featuring a Google security expert, I discovered that the secrets I have in our k8s cluster are no more secure than writing them down on paper and leaving it on a park bench.

Google Cloud Platform is a fantastic offering, and yes, I am biased. I have used every other cloud platform from every major vendor over the last 10 odd years of my career. Before that I was a Systems Engineer building out DC infrastructure and networking to support companies of >1000 employees. Googles offering is the best in my opinion because it is almost identical to what you get if you download the open source product and run it on your own servers. (Most of which Google created to start with.)

And just a disclaimer before we progress. I am not a Google employee and never have been.

However, what I thought was a secure platform with secrets is in fact not even close. You know, they call them secrets right! So they should really be, well, secret…

Kubernetes Secrets

“Kubernetes secrets let you store and manage sensitive information such as passwords, OAuth tokens and SSH keys. Storing confidential information in a Secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image.” <– This is directly from the https://kubernetes.io/docs/concepts/configuration/secret/ page.

A secret in k8s is nothing more than a base64 encoded string that is mapped to an environment variable within a container at runtime.

Whilst this is extremely convenient for getting a basic system up and running quickly, it is certainly not secure. There is no encryption involved and access to the secret is unguarded.

Security First

In light of all the above, let’s see what we can do to ensure that our secrets are:

Encrypted by default;
Only available to containers that require them; and
Easily administered.

We have a couple of options here:

Use the in-built Key Management Service (KMS) in Google Cloud. Whilst this is excellent since there is a sidecar that can run in the pods to grant access to secrets on request, it is a Google only service and so there is no chance of portability if you decide to change providers one day. (It happens. Trust me.) This goes for AWS Secrets Manager and Azure Key Vault. They all lock you into staying with that vendor.
Use a commercial KMS like SecretHub, CyberArk Priviledged Access Management and a plethora of others available on the market. These are great commercial solutions that will assist in making the management of secrets more secure at a price.
Use Open Source tools like HashiCorp Vault, Conjur and OpenStack KeyManager. The use of OpenSource tools allows you to inspect the source code of the tool to ensure there is no way a secret will become, well, not a secret.

For this purpose, I chose to use HashiCorp Vault. I have used Vault in previous roles and it is a fantastic tool to manage secrets as well as access to servers through SSH 2FA.

Implementing Vault

Vault has released a product named vault-k8s. This utilises the k8s mutating admission webhook to augment pods with specific annotations and inject secrets using init and sidebar containers.

Installing Vault

There are a number of methods to running Vault:

Install Vault on-premise or in a compute instance in your selected cloud provider.
Install Vault in k8s using the provided docker image available at https://hub.docker.com/_/vault.
Install Vault in k8s using helm. <– This is the method we will use in this post.

Clone the vault-helm chart from github:

git clone github.com/hashicorp/vault-helm

Open values.yaml and ensure the following option is set:

injector:
  enabled: true

Now, run the following commands after ensuring you are authenticated to Google:

helm install ./vault-helm Once the pods are up and running, you should be able to gain access by forwarding the pod port to your local machine with the following command replacing ‘POD_NAME_HERE’ with the pod name in your cluster: kubectl port-forward $(kubectl get pod --selector="app.kubernetes.io/instance=POD_NAME_HERE,app.kubernetes.io/name=vault,component=server" --output jsonpath='{.items[0].metadata.name}') 8080:8200

Now open your browser to http://localhost:8080 to initialise the service. My setup splits the keys in 8 pieces and requires 4 key parts to unlock Vault. Ensure you record all keys and the root token somewhere secure! These are used to unlock Vault so you can administer the secrets.

Policies

Policies control which humans & apps have what level of access. I have a tier named ‘api’ so I will create an ‘api’ policy as follows:

Navigate to the Policies menu and click the ‘Create ACL Policy’ link.

Name the policy ‘api’ and paste the following content into the text area:

path "secret*" {
  capabilities = ["read"]
}

Now click the save button.

Access

Authentication (Access) allows k8s to access Vault in order to retrieve the secrets.

Click the Access link in the menu bar and then click ‘Add new Method’ link. Select Kubernetes from the options and fill out the cluster IP and cluster certificate from your k8s cluster.

Now open the console from the menu bar and type the following before hitting enter:

vault write auth/kubernetes/role/api bound_service_account_names=app bound_service_account_namespaces=demo policies=api ttl=1h This line tells Vault to allow requests for app where the namespace is demo according to the api policy.

Create Secrets

Now, after all of that, you can now create a secret to use within k8s.

Type the following into the console you used earlier:

vault kv put secret/helloworld username=foobar password=password123

Testing that everything works

Now, lets create a demo app to test out access to Vault.

Open demo.yaml and add the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: app
      containers:
      - name: app
        image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  labels:
    app: vault-agent-demo

Now deploy it and then connect to the deployment as follows:

kubectl create -f demo.yaml kubectl exec -ti demo-XXXXXXXXX -c app -- ls -l /vault/secrets No secrets? Yay! Our Vault install is working as it should.

Delete the deployment kubectl delete deployment/demo and edit it as follows:

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
        vault.hashicorp.com/role: "myapp"

Redeploy it using the same process as above. kubectl create -f demo.yaml Now connect to the deployment using the same command as above and you should see the secret displayed. kubectl exec -ti demo-XXXXXXXXX -c app -- ls -l /vault/secrets

Conclusion

Now that you have successfully deployed vault-k8s and you have proven that it works, you can add secrets for all sorts of things.

I am using it for the PostgreSQL connection string for our production application as well as credentials for various other services we utilise.

If you want to learn more about incorporating Hashicorp Vault in your k8s cluster, check out this tutorial on the Hashicorp website: https://learn.hashicorp.com/vault/getting-started-k8s/sidecar

Ian Brown

Tags

Certifications