Variable and Non-Secret Configuration Management

This article demonstrates pipeline variable management through variable groups and template variable reuse but will also touch on some variable basics.

Variable configuration management

Variable environments and usage

There are two places in Azure Pipelines to interact with variables, both are called environments. One is the YAML build definition called the pipeline environment. The other is within a script, used by a task within a pipeline called a script environment.

In the pipeline environment variables can be defined and used directly in the YAML through:

1. the root level, globally scoped
2. the stage level, only available within that stage
3. the job level, only available to the job

Pipeline variables can be used in the script environment executed within YAML. Using them in the script environment within a YAML pipeline can get tricky if pipeline execution phases, variable expansion and precedence are not understood.

Variables can also be defined outside of the YAML through three ways:

1. At queue time. When triggering a run the UI allows you to define and use variables and will override variables with the same name:
2. Through variable groups
3. Through templates

Brush up on the basics of variables, types and syntax by visiting the following links

• Define variables
• Predefined/system variables
• Output variables
• Secrets management
• Runtime parameters
• Pipeline tasks

Variable groups

Use groups for sharing and controlling variables across multiple pipelines. These are managed in the Library page in Azure Pipelines.

To work with variable groups, you must authorize the group. This is a security feature: if you only had to name the variable group in YAML, then anyone who can push code to your repository could extract the contents of secrets in the variable group. To do this, or if you encounter a resource authorization error in your build, use one of the following techniques:

• If you want to authorize any pipeline to use the variable group, which may be a suitable option if you do not have any secrets in the group, go to Azure Pipelines, open the Library page, choose Variable groups, select the variable group in question, and enable the setting Allow access to all pipelines.

• If you want to authorize a variable group for a specific pipeline, open the pipeline by selecting Edit and queue a build manually. You will see a resource authorization error and a "Authorize resources" action on the error. Choose this action to explicitly add the pipeline as an authorized user of the variable group.

To access defined variables from the library in YAML pipelines use the following syntax

variables:
- group: my-variable-group

Finally, once the group defined in the YAML, the pipeline can use any variable defined in the group as it would any other variable defined in the YAML pipeline.

Template variable reuse

Variables can be stored in a separate YAML file as a template and included in a pipeline or other templates. This is helpful when wanting to increase pipeline security by extending from templates.

To use variables defined in a template use the following syntax

# File: vars.yml
variables:
  favoriteVeggie: 'brussels sprouts'

# File: azure-pipelines.yml

variables:
- template: vars.yml  # Template reference

steps:
- script: echo My favorite vegetable is ${{ variables.favoriteVeggie }}.

Compliance of Non-secret configuration management

Must Comply

1. Configuration should be captured as files managed by Azure DevOps repos or generated automatically based on DevOps tooling. No configuration should need to be entered by hand in standard deployments.
2. Configuration should support replacement tags following a standard convention. These tags should be found and replaced by release stages that have environment or target awareness for context (i.e. Production Environment #2 vs. Dev Environment B, etc.).
3. Convention should follow standards for Azure DevOps Pipelines Replace Tokens Task:
   1. Start with #{
   2. End with }#
4. Release pipelines should use a common reusable pipeline task for environment replacement.
5. No secrets should appear in these configuration files or in source code. Secrets are managed differently then configuration although replacement placeholder tags for secrets can exist that will be applied at a later point.
6. Defaults should be made available that are correct for local developer environments with no changes after a get from any branch (i.e. transforms and variable replacement should only be needed for pipelines).
7. These configuration files/templates should be part of the source code repository of the components being built/deployed.
8. The DevOps tooling should maintain environment specific overrides as Json files per environment/release target in a separate Azure DevOps repo for environment specific values unless a configuration management system is purchased (like Hashi Vault).
9. Configuration and environment specific replacement should occur in pipelines based on environment selection in the pipeline through pipeline variables.
10. Configuration based on files and not values should follow the same practice as the general configuration with one exception, replacement should also be supported within the file.
11. To support replacement within a configuration file that is not the base configuration for a component a splatting pattern should be supported containing a configuration search pattern for files that need this replacement (i,e. replace values within */config-files/.json) for a component's pipeline.
12. If secrets and configuration must exist in the same file, secrets must be applied by a later stage using a different process that secures the contents of the secret.

Should Comply (Best Practice)

1. Configuration should be captured in Json files unless the runtime programming language has another text file format it needs to use (e.g. legacy ASP.NET and C# XML config files).
2. Configuration should also support override based on launch context through pipeline or container environment variables. Care needs to exist to not inject secrets that can be obtained easily.
3. Kubernetes configuration and secrets should use ConfigMaps for general configuration and Secrets for secrets.
4. These objects should be modified as part of their helm chart deployment following the same environment aware replacement rules for values inside the pipeline stage.
5. These objects should be in the same namespace as the component and unique to the component's deployment. No out of repo references should be used.

Optional Practice

1. Legacy replacement tag patterns #{ and }# can exist and be used but a preference for these default tag conventions are the goal.

General Flow

References

• Add and use variable groups
• Manage variable groups
• Understanding AzDO Variables
• variable expansion and precedence
• templates