Supporting rotating keys with KeyVault

Azure Key Vault is a good place to store sensitive information such as encryption keys.

For better security, we should periodically rotate those keys. Rotation is easy right up until you have messages in flight that were encrypted with the old key, and consumers who have no idea a new one exists.

We handled that with Key Vault's secret versioning, using a scenario from my time at ASOS.

The use case

Within our customer API, whenever a customer changes their email address, we publish an event onto Azure Service Bus so that downstream systems can update accordingly.

In JSON form, that event looks a little something like this:

{
  "customerId": 12345,
  "oldEmailAddress": "[email protected]",
  "updatedEmailAddress": "[email protected]"
}

Very simplistic, but it contains all the required information.

To improve the security of customer data, we encrypt the oldEmailAddress and updatedEmailAddress properties. The encryption key lives in a Key Vault, and downstream systems get access to it via Azure Active Directory.

We don't publish the key with the event. We publish the identifier for the key. To be precise, the Secret Identifier.

Anatomy of a Secret Identifier

That identifier is a URI, and its structure is what makes rotation work.

https://alexb-keyvault-test.vault.azure.net/secrets/secret-1/1a884628c2274023a45436797379bc48

Broken down:

That last segment does the work. It lets us encrypt with a specific version of the secret, and tell downstream systems exactly which version we used.

So the event on the wire becomes:

{
  "customerId": 12345,
  "oldEmailAddress": "6RiKk1uSsRJHl0lF1x+jVA==",
  "updatedEmailAddress": "cGRhc2QzMjFhc2Rhc2Q5ODc=",
  "encryptionKeyId": "https://alexb-keyvault-test.vault.azure.net/secrets/secret-1/1a884628c2274023a45436797379bc48"
}

Why this makes rotation a non-event

Set a new value for an existing secret and Key Vault doesn't overwrite it. It creates a new version, and the old versions stay retrievable by their full identifier, until you disable or purge them.

That's the whole trick. You can rotate whenever you like:

1. Write a new version of the secret to the vault.
2. Start encrypting new events with the new version, stamping the new identifier into each event.
3. Consumers carry on as normal. Every message tells them which version to fetch, so a message encrypted an hour before rotation and processed an hour after still decrypts.

You don't need to drain the queue first, coordinate a release, or pick a flag day. Messages sitting in a dead-letter queue for a fortnight still work when you replay them.

Compare that with a secret called email-encryption-key where consumers read whatever's current. Rotate it and everything sitting in the topic subscription turns into undecryptable garbage.

Consumer side

The important part of the consumer is caching by version, not by name.

public class VersionedSecretProvider(SecretClient secretClient, IMemoryCache cache)
{
    public async Task<byte[]> GetKeyAsync(string secretIdentifier, CancellationToken cancellationToken)
    {
        return await cache.GetOrCreateAsync(secretIdentifier, async entry =>
        {
            entry.SlidingExpiration = TimeSpan.FromHours(1);

            var identifier = new KeyVaultSecretIdentifier(new Uri(secretIdentifier));

            var secret = await secretClient.GetSecretAsync(
                identifier.Name,
                identifier.Version,
                cancellationToken);

            return Convert.FromBase64String(secret.Value.Value);
        });
    }
}

KeyVaultSecretIdentifier does the URI parsing for you, which saves a fiddly bit of string splitting. The cache key is the full identifier including the version, so a rotation costs you one cache miss and one extra vault call, then business as usual.

Two things to do here:

Validate the vault URI. The identifier arrives in a message payload. If an attacker can influence what goes on the bus, they can point you at a vault they control. Check the host is one you expect before you go and fetch anything from it.

Don't cache forever. A sliding expiry gives you a way to pick up a secret being disabled, without hammering the vault on every message.

Retiring old versions

Leaving old versions in place forever defeats half the point of rotating.

Our approach: after rotating, wait for longer than the maximum time a message can live in the system. That's message TTL, plus dead-letter retention, plus however long your ops team takes in practice to replay something. Then disable the old version rather than deleting it.

Disabling is reversible. Disable too early and something breaks, you re-enable and nothing is lost. Delete, with soft-delete off, and you've destroyed data.

A consumer trying to fetch a disabled version gets a RequestFailedException with a 403, which is a good thing to alert on. It also tells you how long your real message lifetime is, which in our case was a lot longer than anyone had estimated.

In summary

Publish the version, not the key. The Secret Identifier URI already carries everything a consumer needs, Key Vault keeps old versions around for free, and you get to rotate on a schedule instead of planning a release around it.