---
wrapper_template: "templates/docs/markdown.html"
markdown_includes:
  nav: "kubernetes/docs/shared/_side-navigation.md"
context:
  title: "Using Vault as a CA"
  description: How to replace EasyRSA with Vault for increased security
keywords: juju, kubernetes, security, encryption, vault
tags: [operating, security]
sidebar: k8smain-sidebar
permalink: using-vault.html
layout: [base, ubuntu-com]
toc: True
---

As mentioned in the [Certificates and trust reference][certs-doc] documentation,
[HashiCorp's Vault][vault] can be used to provide either a root or intermediate CA. It can
also be deployed HA, as well as provide a secure secrets store which can be used to enable
[encryption-at-rest for **Charmed Kubernetes**][encryption-doc].

Vault does require an additional database to store its data and (depending on
configuration) some manual steps will be required after deployment or any reboot so
that secrets, such as certificates and signing keys, can be accessed.

## Deploying Charmed Kubernetes with Vault as a root CA

When deploying **Charmed Kubernetes** manually via the
[published Juju bundle][cdk-bundle], it is possible to make use of an overlay
file to change the composition and configuration of the cluster.

The following overlay file ([download][vault-pki-yaml]) alters
**Charmed Kubernetes** to use **Vault** instead of EasyRSA:

```yaml
applications:
  easyrsa: null
  mysql-innodb-cluster:
    channel: 8.0/stable
    charm: mysql-innodb-cluster
    constraints: cores=2 mem=8G root-disk=64G
    num_units: 3
    options:
      enable-binlogs: true
      innodb-buffer-pool-size: 256M
      max-connections: 2000
      wait-timeout: 3600
  vault:
    channel: 1.7/stable
    charm: vault
    num_units: 1
    options:
      # this makes Vault act as a self-signed root CA
      auto-generate-root-ca-cert: true
  vault-mysql-router:
    channel: 8.0/stable
    charm: mysql-router
relations:
- - etcd:certificates
  - vault:certificates
- - kubeapi-load-balancer:certificates
  - vault:certificates
- - kubernetes-control-plane:certificates
  - vault:certificates
- - kubernetes-worker:certificates
  - vault:certificates
- - mysql-innodb-cluster:db-router
  - vault-mysql-router:db-router
- - vault-mysql-router:shared-db
  - vault:shared-db
```

Save this to a file named `vault-pki-overlay.yaml` and deploy with:

```bash
juju deploy charmed-kubernetes --overlay ./vault-pki-overlay.yaml
```

Once the deployment settles, you will notice that several applications are in a
`blocked` state in **Juju**, with **Vault** indicating that it needs to be initialised
and unsealed. To unseal **Vault**, you can read the [vault charm documentation][vault-charm-unseal] for
in-depth instructions (you may also need to [expose][] **Vault**), or you can use
the **Vault** client already on the deployed unit with the following steps:

```bash
juju ssh vault/0
export HISTCONTROL=ignorespace  # enable leading space to suppress command history
export VAULT_ADDR='http://localhost:8200'
vault operator init -key-shares=5 -key-threshold=3  # outputs 5 keys and a root token
 vault operator unseal {key1}
 vault operator unseal {key2}
 vault operator unseal {key3}
 VAULT_TOKEN={root token} vault token create -ttl 10m  # outputs a {charm token} to auth the charm
exit
juju run vault/0 authorize-charm token={charm token}
```

<div class="p-notification--information">
  <div markdown="1" class="p-notification__content">
    <p class="p-notification__message">It is <strong><em>critical </em></strong> that you save all five unseal keys as well as the root token.  If the <strong>Vault</strong> unit is ever rebooted, you will have to repeat the unseal steps (but not the init step) before the CA can become functional again.</p>
  </div>
</div>

### Certificate lifespan

By default, the Vault certificates will be valid for one year from creation.
To alter this expiration date, you can configure the `default-ttl` with a new
value. E.g. to set new certificates to expire after 720 hours:

```bash
juju config vault default-ttl='720h'
```

New certificates can be generated by running an action on the vault charm:

```bash
juju run vault/0 reissue-certificates
```

For more details on configuring vault certificates, please see the
[vault charm][] documentation.

## Transitioning an existing cluster from EasyRSA to Vault

An existing **Charmed Kubernetes** deployment which is using EasyRSA can
be transitioned to use **Vault** as a CA.

<div class="p-notification--information">
  <div markdown="1" class="p-notification__content">
    <p class="p-notification__message">
    During the transition, any pods that use ServiceAccounts to talk to the
    Kubernetes API may need to be restarted. Addons that are deployed and
    managed by **Charmed Kubernetes** will be restarted automatically. If you
    have deployed anything into Kubernetes that talks to the Kubernetes API, it
    is recommended that you restart them after the transition by using the
    `kubectl rollout restart` command.</p>
  </div>
</div>

Deploy **Vault** with the recommended database backend:

```bash
juju deploy -n 3 mysql-innodb-cluster
juju deploy mysql-router vault-mysql-router
juju deploy vault
juju config vault auto-generate-root-ca-cert=true
juju integrate mysql-innodb-cluster:db-router vault-mysql-router:db-router
juju integrate vault-mysql-router:shared-db vault:shared-db
```

Unseal **Vault** as described earlier in this document.

Relate **Vault** to etcd:

```bash
juju integrate vault:certificates etcd:certificates
```

Wait a few minutes for the cluster to settle, with all units showing as active
and idle. Then relate **Vault** to Kubernetes:

```bash
juju integrate vault:certificates kubeapi-load-balancer:certificates
juju integrate vault:certificates kubernetes-control-plane:certificates
juju integrate vault:certificates kubernetes-worker:certificates
```

Wait a few minutes for the cluster to settle, and ensure that all services and
workloads are functioning as expected.

After the transition, you must remove **EasyRSA** to prevent it from
conflicting with **Vault**:

```bash
juju remove-application easyrsa
```

You will need to re-download the `kubectl` config file,
since it contains the certificate info for connecting to the cluster:

```bash
juju ssh kubernetes-control-plane/leader -- cat config > ~/.kube/config
```

<div class="p-notification--caution is-inline">
  <div markdown="1" class="p-notification__content">
    <span class="p-notification__title">Caution:</span>
    <p class="p-notification__message">If you have multiple clusters you will need to manage the config file rather than just
    replacing it. See the <a href="https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/">Kubernetes documentation</a> for more information on managing multiple clusters.</p>
  </div>
</div>

## Using Vault as an intermediary CA

If you don't wish **Vault** to act as a self-signed root CA, you can remove the
`auto-generate-root-ca-cert: true` option from the overlay and follow [these
instructions][vault-guide-csr] to generate a
[Certificate Signing Request (CSR)][csr], have it signed by a trusted root CA,
and upload it back to **Vault**.

## Using Vault in Auto-Unseal mode

The **Vault** charm supports the ability to store and manage the unseal keys and
root token using **Juju** [leadership data][leadership]. Note that this means that
the unseal keys can be accessed at any time from the machine that **Vault** is
running on, significantly reducing the security of **Vault**, particularly with
respect to serving as a secure secrets store. If you are comfortable with this
reduction in security and don't want to have to deal with the manual steps
involved in managing the unseal keys and root token, you can add the following
to the `options` section of `vault` in the overlay above:

```yaml
    totally-unsecure-auto-unlock: true
```

## Using Vault with HA

To enable HA for **Vault**, you will need to first bring up the deployment with
**Vault** in non-HA mode using the instructions above, wait for everything
to settle, and then transition **Vault** to HA mode. This is necessary
because **Vault** requires **etcd** to be running to enter HA mode, but
**etcd** requires PKI certificates to get up and running, leading to a
chicken-and-egg conflict.

Once the deployment is up and settled according to the instructions above,
with **Vault** unsealed and everything functioning, you can then transition
**Vault** to HA mode with the following commands:

```bash
juju integrate vault:etcd etcd:db
juju add-unit vault
```

Once the second unit of **Vault** is up, you will need to unseal it
using the same instructions above with any three of the five unseal keys
and the root token you generated previously. For example, assuming
`vault/1` is the new **Vault** unit, run the following:

```bash
juju ssh vault/1
export HISTCONTROL=ignorespace  # enable leading space to suppress command history
export VAULT_ADDR='http://localhost:8200'
 vault operator unseal {key1}
 vault operator unseal {key2}
 vault operator unseal {key3}
 VAULT_TOKEN={root token} vault token create -ttl 10m  # outputs a new {charm token} to auth the charm
exit
juju run vault/1 authorize-charm token={charm token}
```

At this point, `vault/1` will be running in HA mode with configuration data
from `etcd`. The `vault/0` unit will need to be restarted and unsealed again
to pick up this new configuration:

```bash
juju run vault/0 restart
juju ssh vault/0
export HISTCONTROL=ignorespace  # enable leading space to suppress command history
export VAULT_ADDR='http://localhost:8200'
 vault operator unseal {key1}
 vault operator unseal {key2}
 vault operator unseal {key3}
exit
```

After unsealing, it may take up to 5 minutes for `vault/0` to return to active status.
Once it does, verify **Vault** is in HA mode with one active unit and one standby unit:

```bash
juju status vault
  vault/0* ... Unit is ready (active: false, mlock: enabled)
  vault/1 ... Unit is ready (active: true, mlock: enabled)
```

<!-- LINKS -->
[vault-pki-yaml]: https://raw.githubusercontent.com/charmed-kubernetes/bundle/main/overlays/vault-pki-overlay.yaml
[certs-doc]: /kubernetes/docs/certs-and-trust
[encryption-doc]: /kubernetes/docs/encryption-at-rest
[vault]: https://www.vaultproject.io
[expose]: https://juju.is/docs/juju/expose-a-deployed-application
[hacluster]: https://charmhub.io/hacluster/
[vault-guide-csr]: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-certificate-management.html
[vault-charm-unseal]: https://opendev.org/openstack/charm-vault/src/branch/master/src/README.md#post-deployment-tasks
[csr]: https://en.wikipedia.org/wiki/Certificate_signing_request
[leadership]: https://discourse.jujucharms.com/t/implementing-leadership/1124
[cdk-bundle]: https://charmhub.io/charmed-kubernetes
[vault charm]: https://charmhub.io/vault

<!-- FEEDBACK -->
<div class="p-notification--information">
  <div class="p-notification__content">
    <p class="p-notification__message">We appreciate your feedback on the documentation. You can
    <a href="https://github.com/charmed-kubernetes/kubernetes-docs/edit/main/pages/k8s/using-vault.md" >edit this page</a>
    or
    <a href="https://github.com/charmed-kubernetes/kubernetes-docs/issues/new">file a bug here</a>.</p>
    <p>See the guide to <a href="/kubernetes/docs/how-to-contribute"> contributing </a> or discuss these docs in our <a href="https://chat.charmhub.io/charmhub/channels/kubernetes"> public Mattermost channel</a>.</p>
  </div>
</div>

