Environments and Gating

This document covers Azure DevOps patterns and practices for Environment creation, updates and usage. This is a living document and will be updates as these patterns solidify.

Terminology and Concepts

Terminology

1. Environments : The Pipeline Objects that contain Kubernetes clusters (which is a combination of Cluster and Namespace) and VM pools (under "Virtual Machines").
2. These are used for deployment stages in Azure DevOps YAML based pipelines.
3. They cannot be used by ClassicUI pipelines.
4. Gates : The method of restricting usage to Environments.
5. They are under the setting "Approvals and Checks" for an environment
6. Presently, this includes:
   • Approvals: Users or Groups (teams) that can approve usage of this environment for deployment
   • Branch Control: Deployment limited to specific branches
   • Business Hours: Time based controls limiting deployments to time windows
   • Evaluate Artifact: Ensures artifacts match policies (e.g. container images only)
   • Exclusive Lock: limit access to a resource one stage at a time (needed for Blue Green)
   • Invoke Azure Function: invoke a function and check it's result
   • Invoke REST API: invoke a REST api with payload and verify its response
   • Query Azure Monitor Alerts: Check Azure Monitor rules for active alerts
   • Required Template: Ensure the Pipeline using this environment extends specific other pipelines (ref / path / repo)

Concepts

The Deployment stage is what uses an environment. The environment should exist before use.

Here we see a Dev4 AKS deployment step:

parameters:
- name: 'vmImage'
  default: 'ubuntu-18.04'
  type: string
- name: 'kubeEnv'
  default: 'AKSDev4.shareddev' # optional. we always will push buildid
  type: string
- name: 'buildArtifactName'
  default: 'kubernetes'
  type: string

jobs:
- deployment: kubernetesDevEnv
  condition: and(succeeded(), not(startsWith(variables['Build.SourceBranch'], 'refs/pull/')))
  displayName: 'Deploy-k8s'
  pool:
    vmImage: ${{ parameters.vmImage }}
  environment: ${{ parameters.kubeEnv }}
  strategy:
    runOnce:
      deploy:
        steps:
        - task: DownloadPipelineArtifact@2
          displayName: 'Download Pipeline Artifact src'
          inputs:
            artifactName: ${{ parameters.buildArtifactName }}
            targetPath: '$(Pipeline.Workspace)/manifests'
        - bash: |
            set -x
            ls -l
            ls -l $(Pipeline.Workspace)/manifests
            export
            pwd
          workingDirectory: '$(Pipeline.Workspace)'
          displayName: Debug2
        - task: KubernetesManifest@0
          displayName: Deploy to Kubernetes cluster
          inputs:
            action: deploy
            manifests: |
              $(Pipeline.Workspace)/manifests/**/*.yaml

Environment Usage

Creation

Kubernetes clusters and VM Pools are created by Infrastructure teams and/or the Build and Deploy Team. However, once created, they can be added as an environment. The Name should clearly distinguish the environment type and usage.

Some environment examples: - AKSCI : Azure Kubernetes Service CI Environment (assume only 1) - AKSDev4 / AKSDev9 : The Dev 4 and Dev 9 versions of the AKS services - EKSMLifeDev : The MLifeDev cluster

Let's take a look at MLifeDev for instance. We see two distinct namespaces exposed: azdo and default

Here we see the "default" namespace and "azdo" namespaces exposed. Presently there are no approvals and checks. This means, for first time usage, a Project Admin needs to approve the usage.

Update

We can add and delete Cluster entries (Cluster and Namespace), however they cannot be edited after the fact.

However, if we select an Cluster and chose "Validate", we can see the details for this entry. for example,dev4

Removal

If used, we should not Delete an environment or Cluster. However, we can enable restricted permissions with no permitted pipelines to effectively disable a cluster from future usage.

Naming and usage

There are several types of "Environments". At a high level we have:

1. Dev : These are used by developers and should be open to usage by any valid pipeline
2. shareddev : This is the name for the slightly restricted namespace intended for only "develop" branch deployments - only merged code - code from "develop", "main", or "projects/*" should end up here
3. CI : This is also sometimes called VT . This again, should come from stable main. For R2 this has been "develop" but should be "main" for most projects
4. Prod : These are tightly controlled and will have manual approval checks. DL backed Teams should line up (e.g. AKSProd would have an AKSProd-Deployment-Approvers team) that requires an approver to manually approve deployment

POC Environments

There can be a need for a temporary environment to validate work. Perhaps this is a local on-prem pool of machines or a variety of different class hosts for performance. These will be added via a support ticket and enabled for specific pipelines at the users request. Only the requestor can expand usage (which is why we require the traceability of a ticket)

Boundaries

The Infrastructure Team or Group creates Infrastructure such as Kubernetes Clusters, Container Registries or VM Pools. They should put connectivity information into AKV for sharing to the DevOps Team that can create the required Environment entries. The DevOps Team provides the support (SLA) that the environment entry exists and is accessible, but the fundamental uptime and sizing of the underlying infrastructure remains the purview of the Infrastructure Team or Group