starting Hashicorp vault in server mode under docker container

Running Hashicorp vault in development mode is really easy, but starting the vault in server mode under a docker container may have some changes described in this article.

There are several simple steps, which is hard to get in one place, to run a Hashicorp vault in server mode (under docker):

1. Prepare the directories to map in the docker. The data in the directories will be safe and won’t be deleted if the container is deleted.
2. Prepare an initial base configuration to start the server. Without it, the server won’t startup. Even it is really simple.
3. Start the Hashicorp vault process in a docker container.
4. Initiliza the vault. During this step, the server will generate the database backend storage (files or in-memory or cloud backends) and 5 unseal keys and an administrative root token will be generated. To manage the vault an administrative user is required.
5. Unseal the vault. Unencrypt the database backend to use the service with at least three commands and three different unseal keys generated during the initialization step.
6. Login with the administrative user and enable vault engine to store values (or generate tokens, passwords, and so on). The example here enables the secret engine to store key:value backend. Check out the secrets engines – https://www.vaultproject.io/docs/secrets

STEP 1) Summary of the mapped directories in the docker

Three directories are preserved:

• /vault/config – contains configuration files in HCL or JSON format.
• /vault/data – the place, where the encrypted database files will be kept only if a similar storage engine is used like “file” or “raft” storages. More information here – https://www.vaultproject.io/docs/configuration/storage
• /vault/log – writing persistent audit logs. This feature should be enabled explicitly in the configuration.

The base directory used is /srv/vault/. And the three directories are created as follow and will be mapped in the docker container:

mkdir -p /srv/vault/config /srv/vault/data /srv/vault/log
chmod 777 /srv/vault/data

The server’s directory /srv/vault/config will be mapped in docker’s directory /vault/config and the other two, too.

STEP 2) Initial base configuration

The initial configuration file is placed in /vault/config/config.hcl and is using HCL format – https://github.com/hashicorp/hcl. The initial configuration is minimal:

storage "raft" {
  path    = "/vault/data"
  node_id = "node1"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1
}

disable_mlock = true

api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
ui = true

Place the file in /srv/vault/config/config.hcl

STEP 3) Start the Hashicorp vault server in docker

Mapping the three directories.

root@srv ~ # docker run --cap-add=IPC_LOCK -v /srv/vault/config:/vault/config -v /srv/vault/data:/vault/data -v /srv/vault/logs:/vault/logs --name=srv-vault vault server
==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.14.7
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: raft (HA available)
                 Version: Vault v1.5.3
             Version Sha: 9fcd81405feb320390b9d71e15a691c3bc1daeef

==> Vault server started! Log data will stream in below:

STEP 4) Initialize the vault.

Enter the vault’s shell to execute several commands to initialize the vault. Initialization will generate the database, which will be sealed (encrypted and the server cannot do read or write anything in it). To use the server the next step is to unsealed the database.

root@srv ~ # docker exec -it srv-vault /bin/sh
/ # export VAULT_ADDR=http://127.0.0.1:8200
/ # vault operator init
Unseal Key 1: 2O8np5LBNVvN5VMkCVuDyfalLWm9qMqnCx4Vfio/3Sql
Unseal Key 2: +Iurjirwi28dmEbZrUMMg/sN4Jrz230fYjji+YsNc9A6
Unseal Key 3: 9CKwqPXbv+O70Fafz0bXIB7z4dLvrcYGvhYDPopA7icb
Unseal Key 4: ZjlrRNxR1suNi2rqpaaT4x8EgGs2A1IkC/BY++UsZ1oO
Unseal Key 5: kT28W8wIFcWwjnnF7Cvn6AtFtXdPua6ukiMnFBVPOvUl

Initial Root Token: s.swMd9i9e6xL2ZA1q8L4hOpkf

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 key to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.

Enter the docker container and export the HTTP vault management address. It’s listening in local address so it is safe to be HTTP (HTTPS is not configured, it needs additional configuration). Then initialize the vault. It will generate 5 unseal keys and a root token (token to access the vault after it is unsealed i.e. decrypt, this token is a login credential to manage the vault by an administrative user).

Here is the vault daemon output:

2020-10-01T09:50:40.536Z [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
2020-10-01T10:00:48.346Z [INFO]  core: security barrier not initialized
2020-10-01T10:00:48.382Z [INFO]  storage.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:node1 Address:127.0.0.1:8201}]"
2020-10-01T10:00:48.382Z [INFO]  storage.raft: entering follower state: follower="Node at node1 [Follower]" leader=
2020-10-01T10:00:56.659Z [WARN]  storage.raft: heartbeat timeout reached, starting election: last-leader=
2020-10-01T10:00:56.659Z [INFO]  storage.raft: entering candidate state: node="Node at node1 [Candidate]" term=2
2020-10-01T10:00:56.688Z [INFO]  storage.raft: election won: tally=1
2020-10-01T10:00:56.688Z [INFO]  storage.raft: entering leader state: leader="Node at node1 [Leader]"
2020-10-01T10:00:56.764Z [INFO]  core: security barrier initialized: stored=1 shares=5 threshold=3
2020-10-01T10:00:56.919Z [INFO]  core: post-unseal setup starting
2020-10-01T10:00:57.000Z [INFO]  core: loaded wrapping token key
2020-10-01T10:00:57.000Z [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-10-01T10:00:57.000Z [INFO]  core: no mounts; adding default mount table
2020-10-01T10:00:57.042Z [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-10-01T10:00:57.043Z [INFO]  core: successfully mounted backend: type=system path=sys/
2020-10-01T10:00:57.043Z [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-10-01T10:00:57.175Z [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-10-01T10:00:57.175Z [INFO]  core: restoring leases
2020-10-01T10:00:57.175Z [INFO]  rollback: starting rollback manager
2020-10-01T10:00:57.176Z [INFO]  expiration: lease restore complete
2020-10-01T10:00:57.218Z [INFO]  identity: entities restored
2020-10-01T10:00:57.218Z [INFO]  identity: groups restored
2020-10-01T10:00:57.218Z [INFO]  core: usage gauge collection is disabled
2020-10-01T10:00:57.238Z [INFO]  core: post-unseal setup complete
2020-10-01T10:00:57.315Z [INFO]  core: root token generated
2020-10-01T10:00:57.383Z [INFO]  core: pre-seal teardown starting
2020-10-01T10:00:57.383Z [INFO]  rollback: stopping rollback manager
2020-10-01T10:00:57.383Z [INFO]  core: pre-seal teardown complete
[C2020-10-01T10:03:57.668Z [INFO]  core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:8201
2020-10-01T10:03:57.668Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:8201
2020-10-01T10:03:57.680Z [INFO]  storage.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:node1 Address:127.0.0.1:8201}]"
2020-10-01T10:03:57.680Z [INFO]  core: vault is unsealed
2020-10-01T10:03:57.680Z [INFO]  storage.raft: entering follower state: follower="Node at 127.0.0.1:8201 [Follower]" leader=
2020-10-01T10:03:57.680Z [INFO]  core: entering standby mode

These files and directories should appear after the initialization:

/ # find /vault/data/
/vault/data/
/vault/data/vault.db
/vault/data/raft
/vault/data/raft/snapshots
/vault/data/raft/raft.db

STEP 5) Unseal the database to use the vault service.

The database should be unsealed with at least 3 of the generated during the inital setup unseal keys. Execute three times unseal command and enter three different unseal keys:

/ # vault operator unseal
Unseal Key (will be hidden): 
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       61d9aa84-eb31-824c-566b-36cc27e960b8
Version            1.5.3
HA Enabled         true
/ # vault operator unseal
Unseal Key (will be hidden): 
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       61d9aa84-eb31-824c-566b-36cc27e960b8
Version            1.5.3
HA Enabled         true
/ # vault operator unseal
Unseal Key (will be hidden): 
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            5
Threshold               3
Version                 1.5.3
Cluster Name            vault-cluster-d0255d95
Cluster ID              c424ff62-5031-13d2-1793-3de721b4154f
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24

The third output shows the vault is no more sealed and it can be used.

The logs in the daemon when unsealing:

2020-10-01T10:04:05.450Z [WARN]  storage.raft: heartbeat timeout reached, starting election: last-leader=
2020-10-01T10:04:05.450Z [INFO]  storage.raft: entering candidate state: node="Node at 127.0.0.1:8201 [Candidate]" term=3
2020-10-01T10:04:05.485Z [INFO]  storage.raft: election won: tally=1
2020-10-01T10:04:05.485Z [INFO]  storage.raft: entering leader state: leader="Node at 127.0.0.1:8201 [Leader]"
2020-10-01T10:04:05.517Z [INFO]  core: acquired lock, enabling active operation
2020-10-01T10:04:05.573Z [INFO]  core: post-unseal setup starting
2020-10-01T10:04:05.594Z [INFO]  core: loaded wrapping token key
2020-10-01T10:04:05.594Z [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-10-01T10:04:05.594Z [INFO]  core: successfully mounted backend: type=system path=sys/
2020-10-01T10:04:05.594Z [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-10-01T10:04:05.594Z [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-10-01T10:04:05.595Z [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-10-01T10:04:05.595Z [INFO]  core: restoring leases
2020-10-01T10:04:05.595Z [INFO]  rollback: starting rollback manager
2020-10-01T10:04:05.595Z [INFO]  identity: entities restored
2020-10-01T10:04:05.595Z [INFO]  expiration: lease restore complete
2020-10-01T10:04:05.595Z [INFO]  identity: groups restored
2020-10-01T10:04:05.596Z [INFO]  core: usage gauge collection is disabled
2020-10-01T10:04:05.624Z [INFO]  core: post-unseal setup complete

STEP 6) Login in the server with the root administrative account and enable a secret engine to store values.

Enter the root key generated during the initialization process above.

/ # vault login s.swMd9i9e6xL2ZA1q8L4hOpkf
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.swMd9i9e6xL2ZA1q8L4hOpkf
token_accessor       kFdYrgscUokHhI5eAg2ijwMZ
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

And enable the secret engine to use the key:value backend:

/ # vault secrets enable -path=secret/ kv
Success! Enabled the kv secrets engine at: secret/

Last one, just set and get values example:

/ # vault kv put secret/hello foo=world excited=yes
Success! Data written to: secret/hello
/ # vault kv get secret/hello 
===== Data =====
Key        Value
---        -----
excited    yes
foo        world

Bonus: Expose the vault port out of the docker container

Two things should be changed if the user wants to expose the vault port out of the docker container (to load the vault UI in the host’s browser, for example):

1. Change the vault configuration file. Vault should listen on 0.0.0.0:8200

   storage "raft" {
     path    = "/vault/data"
     node_id = "node1"
   }
   
   listener "tcp" {
     address     = "0.0.0.0:8200"
     tls_disable = 1
   }
   
   disable_mlock = true
   
   api_addr = "http://127.0.0.1:8200"
   cluster_addr = "https://127.0.0.1:8201"
   ui = true
   

2. Run the docker command with mapped port:

   docker run --cap-add=IPC_LOCK -p 127.0.0.1:8200:8200 -v /srv/vault/config:/vault/config -v /srv/vault/data:/vault/data -v /srv/vault/logs:/vault/logs --name=srv-vault vault server
   

   And now, on the host machine, the URL http://127.0.0.1:8200/ui/ will load the vault user interface in the browser. Do not forget, it is dangerous to expose vault port out of user’s local machine or network!!! Especially, when an HTTP configuration is used!