Secrets are a critical part of any software application. They are used to store sensitive information, such as passwords, API keys, and encryption keys. However, the way we manage secrets can also be a major security risk. If they are not stored and managed securely, they can be easily compromised, leading to data breaches and other security incidents.
Secrets sprawl is a common antipattern in organizational secrets management. It occurs when secrets are stored in a variety of different locations and a variety of tools, making it difficult to track, manage, and rotate them.
What leads to secrets sprawl?
There are a number of factors that can contribute to secrets sprawl:
The use of different tools and technologies: Different teams within an organization may use different tools and technologies to manage their secrets. This can make it difficult to keep track of where secrets are stored and who has access to them.
The lack of a centralized secrets management system: Many organizations do not have a centralized secrets management system. This means that secrets are stored in a variety of different locations, making it difficult to track and manage them.
The lack of security awareness: Developers and other employees may not be aware of the security risks associated with secrets sprawl. This can lead to secrets being stored in insecure locations or being shared with unauthorized users.
So, imagine an environment where your development teams are using a wide variety of secret stores and secret vaults. And your DevOps team is looking to centralize the secrets management system, so it’s easy for them to track, manage, ingest, and rotate secrets. While your management, still wants to provide the tech stack flexibility to developers for secrets management.
Solution:
Teller is a free and open-source secret management hub that provides a secure and scalable way to store, manage, and rotate secrets. With Teller, you can fetch and populate secrets in real-time from over 10 different vault and keystore providers, run processes safely without intermediate files or external libraries, and avoid secret sprawl. Teller also lets you redact logs, code, and other data, as well as scan for vault-originating secrets easily.
Scenario:
ALERT
This type of alert in any organization will create a severity 1 incident and you will see a group of engineers not only working to fix this as quickly as possible, but also create a better design for handling secrets in code. Clearly, there is a sprawling Auth0 secret that’s not supposed to be there.
Let’s turn our hardcoded secret into an environment variable. With that, we’re also moving another step towards 12-factor apps, which is the best practice; it means that where ever we deploy this service: Heroku, Kubernetes, Docker, or any platform— a standard way to communicate variables and dynamic values would be on top of environment variables and we can be confident that it’s supported everywhere.
Let’s name the environment variable as AUTH0_SECRET, and drop the ‘client’, to not create a false expectation that ‘client’ means “OK to store on client-side code such as web and mobile”. Naming is hard in software for a good reason, and in this case, the name we chose informs users to handle with care.
Here is our app.js file:
Now that our code is clean and well-designed, we’re going to use Teller to securely fetch and populate our environment variable.
Teller:
Antipattern: We are not going to use .env. That is because the risk of creating, or misplacing a .env.production file with a secret, is almost the same as hardcoding the secret itself in code. So, try to avoid this approach.
Instead, we will use Teller for all our needs. The first step is to create a configuration mapping in a file called .teller.yml. Note that this file contains no sensitive information at all.
This is your project-specific Teller configuration.
We use Hashicorp Vault, so we will specify that provider:
project: cart-svc
opts:
stage: development
providers:
hashicorp_vault:
env_sync:
path: secret/data/{{stage}}/cart-svcNote: If you are not using Hashicorp Vault, you can pick from a wide selection of more than 10 other providers. You can also use multiple different providers in parallel.
Back to our Auth0 secret — first revoke the old one — because you never know. Then generate and copy the new secret and place it in your vault.
You can choose to place the secret in your vault manually:
or if you want, you can use the teller CLI to place your secret because it already knows where it belongs.
$ teller put AUTH0_SECRET=`pbpaste` --providers hashicorp_vaultRunning your service now takes a fresh, more secure approach:
$ teller run bin/your-serviceBy prefixing your service start-up with teller run, you are now hardening the way it runs. This benefits from just-in-time fetching and population of your secrets, which is great for a few reasons:
There’s no file or any secret written in any place on the disk.
Everything is in-memory and no secret is being held in memory for longer than it needs to be.
When you rotate keys in the future, this will be transparent for your services and require no code changes and no .env changes at all.
In addition, Teller will block out all existing environment variables that are by default visible, and only allow the variables that are explicitly visible (you’ve allowed in). So a malicious binary or piece of code will never be able to sniff out your misplaced shell variables.
As a convenience, using Teller and creating your configuration also gives you the ability, in your project directory, to scan for your secrets in order to make sure it never got misplaced back again:
$ teller scanBut there's more, Teller also allows you to do:
Scan code for the defined sensitive values and secrets, no configuration needed, just a different command: teller scan
Fix configuration drift automatically, when using more than one type of provider or multiple environments (think: dev/staging/production)
Sync stores, for example: when you’re bringing up a new temporary stack with its own fresh vault store
Many more, actually. Check the Teller Github Repo for the docs.
Comments