Secrets Management - With the emphasis on the Management

Introduction

Secrets are everywhere in today’s IT architectures in addition automation is a requirement of today’s IT architectures. Therefore, in a somewhat contradictory way, secrets need to be readily available and accessible everywhere.

Enabling availability and ubiquitous access is the job of a Secrets Management system.

In this article I discuss why secure storage is a necessary but not sufficient requirement for managing secrets and why secrets management is important beyond simple storage.

What is a Secret

Despite what some security professionals might say, there isn’t much that is actually truly secret in IT security. For many things in the IT security realm there are grandiose or spooky (in the spy sense) names for things. Secrets is one of these. But in this case it’s probably the perfect term.

There are bits of data that need to be kept secret in that they shouldn’t be exposed ‘in the clear’ outside of the controlled environment they are required to be used in.

My definition of a secret is:

• Any sort of data that should not be exposed in the clear
• Generally not made visible to human operators
• Primarily consumable and consumed by machines

Classic examples of secrets are:

• SSH Keys
• Private Keys
• API Keys
• Passwords
• Tokens
• Symmetric Keys

One thing that probably isn’t a secret in this sense is a user password. If you find you are mostly putting user passwords into secrets management systems for use by human operators you probably need a Privileged Access Management system (PAM) instead of or (more likely) as well as a secrets management system. This is what distinguishes a secret from user credentials. While user credentials are secret, they are not secrets in the sense of this definition.

This is a worthwhile distinction to make because the operational concerns around user credential management and secrets management are two different use cases with different solutions. In this article we are talking specifically about secrets as I define them above.

What is Secrets Management

In the past we might have used simple key stores local to applications, secrets baked into applications or, heaven forbid, secrets stored directly in config. None of these are satisfactory management strategies.

Secrets Management is the secure storage, access control, audit and lifecycle management of secrets. A Secrets Management System a specific solution designed to cater for these needs in an automatable API driven, machine friendly.

The best example of this (in my opinion) is Hashicorp’s Vault product.

Management Matters

Secure storage of secrets is pretty easy. There are myriad ways to cryptographically secure a bunch of data.

But does securing your data behind one more lock make it more secure? And isn’t the access to the secrets management system just a key to all the keys?

When securing secrets is viewed as just a store without management, then no, it’s probably no more secure. Worse still it adds a risk (a key to all the keys) and doesn’t mitigate any of the big risks.

This is why management is a very important component to get in place and it’s often what is missing.

It’s also often what doesn’t quite get articulated correctly when pitching secrets management to key decisions makers who may not have a clear view of the risks that management mitigates. I am sure this is often what leads to things like secrets management being deprioritised by many IT organisations.

Here is what why management matters.

Cascading Risks

A big operational risk is the loss of access to secrets due to mismanagement. If this happens it of course means a loss of access to the thing they unlock. In a worst cast scenario, if these lost keys are decryption keys, for example, you will have crypto-wiped whatever data they secure.

Conversely there is over exposure where due care isn’t taken to protect secrets.

Exposure of secrets to a bad actor is double problematic without secrets management in place because it’s unlikely you will know where an exposed secret is being used in the organisation.

Without knowing where secrets are used, it’s extremely difficult to rotate them without breaking something. You may not know what will break until you do rotate them. Now you have not only a security incident, but a production incident to deal with.

This is, of course, even if you can easily rotate them. Without management in place rotation will be more difficult.

On top of that, not knowing where a potentially exposed secret is used, probably means having to rotate all secrets ‘just to be safe’. Suddenly a security response turns into a massive operational exercise enterprise wide.

All of this may not be an issue, because you may not even know a secret has been exposed. That database administrator the left 6 months ago? They had the API key for the data warehouse system in the cloud account they managed. Without a way to rotate the API key and account keys easily, there is no choice but to accept the risk that former employees with privileged access are forever trustworthy.

These are just a few of the cascading set of risks that happen when management is missing.

Management of secrets is far more important than just the secure storage alone.

Beyond secure storage, the sections below discuss a few of the key things good secrets management provides that mitigate all of these cascading risks.

Make Secrets Visible

It’s a mouthful to say, but secret data needs to be secret whereas the secrets objects should be highly visible. At least to the operations people.

It should be very easy to see where secrets are used, by whom and when. It’s plain to see that this alone mitigates a lot of the risks discussed above.

Visibility is also critical for catching exposure. Requiring systems that need to use secrets to go via secrets management, means tracking their use is by default part of the process. As such anomalous activity can be spotted and in conjunction with other security tooling, out of band usage can be identified as well. Blindly rotating secrets in the hope that an exposure is mitigated is less likely to be required. This greatly lessens the impact of a security incident all round when a secret is exposed.

Remove Key Person Risk and Single Points of Failure

Key person risk and isolated storage of secrets are the principle causes of loss of access to secrets.

Having robust secrets management in place means that there is a place to store and manage secrets that doesn’t need to rely on only certain key people knowing and manually managing them.

A management system will enable appropriate access to secrets by systems that require it and should remove the need for remote islands of unmanaged secrets storage that can lead to single points of failure.

Rotation as a Business as Usual Operation

Secret rotation should not be a ‘stop the world’ activity. Most secrets should be able to be rotated at any time with relative ease. Secrets management systems make this possible.

Too many systems have been designed without considering how to rotate secrets properly. For example, systems that use long expiry certificates. This is a convenient way to put off and make less frequent the operational headache of renewing them. But it adds yet another risk, namely the risk of forgetting to renew. Many a production incident has been caused by a forgotten certificate expiry.

Building in secrets rotation as a regular business as usual practice eliminates the operational headache, eliminates long expiry problems and ensures you can rotate secrets as and when required quickly and with ease. There should be no major security exercise required to rotate secrets in bulk. If systems are engineered with rotation in mind, there is no operational risk when doing it.

Finally, this further mitigates exposure risk and privileged former employee/insider risk. Critical secrets can be rotated as part of the joiner, leaver, mover processes for example.

Devaluing Secrets

Short of not having any secrets, which is a practical impossibility, the next best thing is to devalue them. This means making secrets as useless as possible to a bad actor. Management is the only way to do this effectively.

The first way to devalue a secret is to make sure its domain of effectiveness is as small as possible. In other words, don’t use secrets in more than one place. Analogous to having a different password for every website login. This is only possible by having somewhere to manage all those secrets.

The best way to devalue a secret is to make it as dynamic as possible, to the point where they are basically single use. This can be done by:

• Changing a secret after every use
• Using time bound expiring secrets
• Just in time creation of secrets

Again the only way to do this is to have management in place.

Dynamic secrets can be difficult to implement into an existing system and sometimes scare operations people. What do you mean we will change the service account passwords every time we restart the service? It’s often the feature of a secrets management system that people put off using. But the sooner it can be used the better.

Automated creation and management of secrets is the best way to devalue them and limit exposure. Adopting business as usual rotation and baking in the handling of rotation into systems is a stepping stone to enabling complete automated management and laying the ground work for dynamic secrets.

While visibility on where secrets are being used is critical (as discussed above), visibility of the actual secrets should be reduced in order to further limit exposure. In conjunction with dynamic secrets, automation of secrets creation and rotation, secrets management systems can completely remove the need for human operators to see or even access the actual data. This further mitigates key person and exposure risks.

Conclusion

Management of secrets is just as important as storing them securely. In fact, I would argue that just having a secure store without any management is not an effective security practice.

Secrets Management mitigates risk by:

• Making secrets visible
• Removing key person risk and single points of failure
• Enabling secret rotation as a business as usual activity
• Devaluing secrets as much as possible

These aspects should be a key part of convincing organisations to that secrets management systems and processes are the table stakes for enabling effective, scalable and high performing technology organisations in the fast moving, fast growing and highly automated IT environments required today.