Skip to content

Agent Images Patterns and Practices

Document describes a general approach to creating images that will be used to deploy systems that are intended to be customized Azure DevOps Agents.

Overview

In most cases, Microsoft Hosted or generic systems through Azure VMSS will meet the needs of a AzDo Agent. Please see Agents-and-Agent-Pools.md for more details on generic use cases.

If you already have a system and simply want to add it as an AzDo Agent, please see the Self-Hosted other VM/Hardware section in Agents-and-Agent-Pools.md.

This document describes the process in creating images that require customized configurations. For example, a test system that requires specific third-party tools and will be used in a test pipeline as an AzDo agent.

The scope of this document is for creating the images intended to run as AzDo Agents in AWS. The general principals can be applied to other cloud providers as well.

Base Image

Strongly recommend using the CIS hardened base images located in the cvg-ops-mdt 050061447862 AWS account as your base image.

If you require additional tools and/or configuration, please build an additional image using the CIS hardened image as your base image. If the change is common enough to be added to the base image, please open a PR to the carelink-build-image repo.

If the CIS image does not meet your needs, please continue and follow the best practices from the rest of this document.

Prerequisites

  1. General understanding of system requirements for your use case. Some of these can be noted during Manual Discovery phase.
    • OS specifications and configuration
    • Third party tools to be installed on system
    • Network requirements
    • Security
    • User access/RBAC
    • Domain access and security
    • Active Directory requirements
    • Storage requirements
    • etc
  2. AWS account access to deploy EC2 instances and create/publish AMI
  3. AWS networking and security groups configured for your required connectivity . Secrets management for image configuration
  4. Artifact storage for packages, scripts, and binaries that are used during image build
  5. Artifact storage and processes for handling image storage and consumption
  6. Understanding of Packer
  7. If applicable, understanding of Ansible and/or other configuration management tools

Manual Discovery

Oftentimes, it is easier to manually deploy a system and manually configure it to understand all the required pieces for the system. This approach allows focus on the actual configuration management before involving the infrastructure as code elements.

Process

  1. Deploy a base OS image as a VM
  2. Manually configure system with all requirements
  3. Test out intended workloads on system to ensure it meets requirements
  4. Document requirements

In short, an AMI can be built from this existing VM and used as the base for future deployments. Although this works for the short-term, it is highly recommended to implement infrastructure as code to allow for better tracking and maintenance.

Infrastructure as Code

Infrastructure as code is imperative in having a well managed infrastucture environment. It enables confidence, repeatability, troubleshooting, auditability, visibility, security, etc. [[1]],[[2]]

There are situations where automation is not possible. It is recommended to write as much infrastructure as code and only leave the processes that require manual intervention and configuration.

For image builds, it is recommended that Packer be the tool used. Within Packer, a configuration management tool such as Ansible, is recommended for portability and reuse.

Process

In general, this process follows the Getting Started docs from Packer. Your specific use case may deviate from this process depending on your requirements.

Please see official Packer Documentation for references.

It may also be beneficial to not build an image from scratch, but use an existing image as your base. Please check the public AWS AMI list or your approved list of images for possible image re-use.

  1. Using document of requirements, write Packer script that creates image
  2. Using scripts or Ansible, write the configuration automation
    • Install third-party tools from web or local artifact store
    • Configure network
    • Pull and configure secrets
    • Join domains
    • Setup security
    • et
  3. Implement semantic versioning of images
  4. Publish images to an artifact storage, e.g. S3, Artifactory, etc
    • Location and images should be accessible by user or process that deploys system.

Automation

  • The image build should ultimately be placed in a pipeline for automated builds.
  • Image builds should be triggered on branches pushed upstream
  • Image pushes/publishes should be triggered on defined branches such as main and release
  • Image tagging should be used to ensure differentiation between production vs staging vs dev
  • Use semantic versioning for versioning images
  • Have different pipelines for building images vs deploying images