Skip to content

Pipelines Patterns and Practices - Secret Management

Note: This Pattern and Practice document only describes techniques that work with YAML based pipelines stored in source control. Legacy visual pipelines will at some point be deprecated and this process doesn't consider them a valid technology to continue to use.

Overview

The configuration values for a pipeline can be managed in two different ways through variables and secrets. Secrets are the variables provided to your pipeline that need their values protected from view. Since we need to think of a pipeline as source code we need to not hard code any secrets into that pipeline since it is source code. To manage secrets we need a "store" for these secrets and we are recommending the use of Azure Key Vaults as that "store".

Secrets are normally used in release and publishing activities and as such may be context sensitive to the environment or customer that the publication is intended to be used. Some secrets though are used during build, test and review activities to access resources needed by the pipeline.

Azure Key Vault best practices indicate that a key vault needs to have its access controlled.

The most important concept of how to control this access is to have an unique vault per application per environment. Application can be broadly defined as an entire system or as refined as a single executable process. Environment usually follows concepts of development, pre-production and production.

Key Vault implementation recommendations

  1. Design a key vault instance for each environment. At minimum consider one for development, one for qualified testing environments and one for production.
  2. Scope your application to be appropriate to the number of key vaults you want to manage. For example, if you have 30 repositories and 4 environments this could become 120 key vaults. You might consider defining key vaults by application and resource type. For example 3 for database secrets, 3 for certificate secrets, etc. (one for each environment type) for several repositories that share a synergy.
  3. When you use the AzureKeyVault task to access your secrets you have two options - specifying the keys you want retrieved and specifying all keys should be returned. If the vault has many keys the wild card may be appropriate to use to not have to change the pipeline often.
  4. Follow a common naming convention for keys. Keys in one vault will be overridden by keys with the same name loaded by another vault sequentially in the pipeline. You can leverage this capability to build a basic inheritance model if needed.

Variables are not secrets

Do not store all pipeline configuration as secrets

It is possible that all configuration needed by your pipelines can be stored inside Azure Key vaults and secrets but it is difficult to manage this many values. Secrets are more often the exception then the rule. Consider using JSON based files for variables that are environment and application specific and commit those to an external storage location. Retrieve these JSON files by environment and load all the configuration values into variables for the pipeline to supply to steps.

Setup of Secret management for a pipeline.

  1. An Azure DevOps project administrator should establish two service connections to the Azure Subscription holding your key vaults. One for pre-production vaults and one for production vaults.
  2. Create the key vaults.
  3. Grant to the service connection (production or pre-production) the access policy of get and list on secrets, keys and certificates in the key vault.
  4. Grant to the correct DL access to add and update the key vault as appropriate.

  5. Add an Azure Key Vault task to access your key vault. The values below are made up...

    - task: AzureKeyVault@1 inputs: azureSubscription: 'Special Team Production Resource Access' # Name of Service Connection Defined KeyVaultName: 'kv-myappshared-1' # Name of Key Vault SecretsFilter: '*' # Comma delimited list of secrets to retrieve, or * for all RunAsPreJob: false # If true, run before job so available to all steps.

  6. In later steps you need to access the secrets through capturing them as environment variables for the step to see. The following is an example of this that just writes out the values of two secrets named in the key vault (env-secret-1 and env-secret-2). This example also shows why production secret access needs to be done with a special service connection not granted to most people.

    - task: Bash@3 inputs: targetType: 'inline' script: | echo $ENVSECRETVAR1 > $PIPELINE_WORKSPACE/secrets.txt echo $ENVSECRETVAR2 >> $PIPELINE_WORKSPACE/secrets.txt env: ENVSECRETVAR1: $(env-secret-1) ENVSECRETVAR2: $(env-secret-2)

  7. Add all of your secrets to the vaults as "secrets" or "certificates".

  8. All secrets and certificates are provided to the pipeline as a string. This article documents how you can use Powershell to convert a certificate into a PFX format or store it in a password protected PFX file.

Dealing with lots of environments.

Some teams have lots of environments and three techniques can be used to make the pipeline more manageable.

  1. Add a pipeline parameter for the environment. Use this parameter to dynamically select an environment specific vault. This is best done when there are many environments and it requires a stable naming convention on the key vaults. The benefit is that this value can be supplied as the pipeline is setup to trigger. To do this add a parameter and then dynamically name your vault selected.

    parameters: - name: environment displayName: Target Environment type: string default: dev

    And then later in the pipeline.

    - task: AzureKeyVault@1 inputs: azureSubscription: 'Special Team Production Resource Access' KeyVaultName: ${{ format('kv-myapp{0}-1', parameters.environment) }} SecretsFilter: '*' RunAsPreJob: false

    Although not illustrated here it is possible to use conditionals and also switch the "azureSubscription" value to use a production vs. development service connection for security purposes of production builds.

  2. Apply a double substitution on some values. They key name applied for a step can be double replaced by doing a format of environment into a base string. This will allow you to define a set of keys within one vault but for different environments. As an example if your key is named env-device-connstring, you could create in a testing vault values named env-vt1-device-connstring, env-vt2-device-connstring, env-val1-device-connstring and do a replacement.

    parameters: - name: environment displayName: Target Environment type: string default: dev

    And then later in the pipeline.

    - task: Bash@3 inputs: targetType: 'inline' script: | echo $ENVSECRETVAR1 > $PIPELINE_WORKSPACE/secrets.txt echo $ENVSECRETVAR2 >> $PIPELINE_WORKSPACE/secrets.txt env: ENVSECRETVAR1: ${{ format('env-{0}-device-connstring', parameters.environment) }} ENVSECRETVAR2: $(env-secret-2)

  3. Create a step that does a tokenized replacement of files matching a pattern. Use a match to an attached key vault secret as the source of the replacement. This main step would move all of the logic of dealing with secrets into a reusable step and make it so a high number of secrets can be managed. At this point a reusable pipeline extension for this has not been developed by the SDA teams.

Secrets as files

If you need to store a file as a secret you should consider if the file could become a template in source control and then have the secret parts replaced with tokenized values from azure key vault within the pipeline. If you cannot do that and the entire file needs to be a secret you have two options.

  1. Base64 encode the file, put it all on one line and store as a secret. Reverse the process as you consume the file in the pipeline.
  2. Encrypt the secret file and add to a private blob storage account that the pipeline can access. Store the credentials used to encrypt the file in azure key vault as secrets that the pipeline can access to decrypt the file as part of a pipeline.

Certificate based secrets

Azure key vault can store, manage and version certificates. They come to the pipeline in a special format that can be converted with the following powershell code to a PFX.

This imports the secret as a string into an in memory certificate collection:

$kvSecretBytes = [System.Convert]::FromBase64String($(PfxSecret))
$certCollection = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2Collection
$certCollection.Import($kvSecretBytes,$null,[System.Security.Cryptography.X509Certificates.X509KeyStorageFlags]::Exportable)

This additionally stores the certificate as a file that the pipeline can use. 

#Get the file created
$password = 'your password'
$protectedCertificateBytes = $certCollection.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Pkcs12, $password)
$pfxPath = $Env:AGENT_BUILDDIRECTORY + "\\MyCert.pfx"
[System.IO.File]::WriteAllBytes($pfxPath, $protectedCertificateBytes)

Certificate Rotation

Certificates can be rotated in Azure Key vaults because each secret has a version. You can access both the current version of the secret or a specific version in the pipeline. The names can be seen inside the Azure Key Vault if you need to pin to a version of a secret. Even if your certificates are not being managed by Azure Key Vault there is a benefit to store an exportable copy of the certificate in key vault for continuous delivery operations.

Consider making a pipeline integrate with the certificate manager and store a copy of the certificate inside the attached key vault. This will allow your pipeline to always use the latest certificate without needing changes. It is possible also to re-trigger a pipeline when the key vault changes causing a built to occur and publish an update. This continuous deployment model is very efficient if you need to manage certificates in your applications.

To build this automation the following products are used together.

  1. Event grid subscribes to changes of the key vault certificates and broadcasts them.
  2. An azure function listens to these events and based on configuration decides if a pipeline should be triggered.
  3. The function calls the Azure DevOps REST API to trigger a pipeline execution run.

Managing the security of azure key vaults and their secrets

This procedure documents using an RBAC feature which is in preview at the time of writing. The steps to implement this may change over time.

  1. All key vaults for secrets management should be grouped into a resource group for the "application" and split between production and non production environments. If desired a 3rd resource group can be produced for formal testing environments and it would be modeled like the production one. In this design you will usually have 2 or 3 resource groups where the key vaults are created.
  2. Since team projects are managed by 2 DL based groups (administrators and members) we will use these groups to apply security.
  3. When an Azure Key Vault is created within the resource group make sure under Settings > Access Policies you enable "Azure role-based access control".

  4. At the resource group establish the following security guidelines using the Add role assignment under Access control (IAM)

    DL Group RBAC Role Assigned (Pre-Production Resource Group) RBAC Role Assigned (Production Resource Group)
    Project Administrators DL Group Key Vault Administrator Key Vault Administrator
    Project Members DL Group Key Vault Secrets Officer Key Vault Reader
    Service Connection Account (account) Key Vault Secrets User Key Vault Secrets User

    This will grant users access to read and modify pre-production secrets but only see the name of the production secrets. The pipelines service connection accounts can read and list all keys and access their secrets. Administrators have full control.

    Note it will take 10 minutes at time of this writing for the security RBAC to be applied to the key vaults in the resource groups.

The goal of this security design is to not allow access to production secrets (lists and keys are ok) for project members. In rare cases production secrets may need to be managed by a different group than the members of the project administrators DL group. In this case add another role assignment for that production group and give them "Key Vault Administrator" access and provide the Project Administrators DL "Key Vault Reader" access.

Protected access to secrets

Because pipelines can be shared between pre-production uses and production uses RBAC security is not enough to protect secrets for a pipeline.

If a pipeline is authorized to use a service connection, anyone who can modify that pipeline can use the pipeline to extract the secrets. While Azure DevOps protects secrets in logs and in normal use it is possible to store them in a file for later viewing if you have rights to modify the pipeline to do that.

Use the following strategies to manage protecting production pipeline secrets

  1. Review that pipelines running against development secrets are not coded to expose secrets.
  2. Add code to the pipeline to prevent using production key vaults when not running against the main branch.
  3. Use extends templates to create a base template for a pipeline that cannot be modified by people who are not to authorized to do so. In the base template you can add the logic to select key vaults and setup state to prevent production access under the wrong conditions.