Skip to content

Frequently Asked Questions

If your question is not answered in any of the available documentation or this FAQ, or if you have any other questions or concerns, please reach out on the DevEx Teams Channel.

Agents

General

Are there agents available in the MDTCRHF Organization?

Unfortunately, no. MDTProductDevelopment is where CRM wanted to migrate, so that is where the DevEx team set up the pools. Azure DevOps does not allow cross-organizational pool sharing, so your choices are:

  1. Migrate to MDTProductDevelopment (recommended)
  2. Use Azure Pipelines pool instead (not recommended, but advisable)
  3. Self-host your own agent pool (not recommended or advisable)

Are the agents under global pools always ON (i.e. a minimum number of agents)?

Yes, there will always be a minimum number of agents. That number can be adjusted, if needed, to reduce cost or increase available agents. The current minimum number is 8 agents.

What is spin-up time of a new agent?

From the time the alarm triggers a scale-out event to the moment the last of the 4 agents of the EC2 instance are registered with the pool takes ~3min.

Do the agents go down when there are updates to the AMI's requirements and/or capabilities?

When requirements are updated, the instance refresh follows the same termination process that scale-in events do. Meaning, an instance will only be terminated once all agents on that instance are not running jobs and are de-registered from the pool. The agents don't "go down" in a sense that they go offline. They will disappear when Idle. New ones will take their place once the instance with the new AMI is launched. And because an instance refresh only executes on one or two EC2 instances at a time, there should always be at least 4 agents in the pool at all times.

Network Access

Can I access the EKS cluster in mlife-dev?

Yes! We specifically engineered the networking so not only would you be able to access the dev EKS cluster, but also all the available CLD endpoints at *.ad.dev2.mlife.

I'm getting Network Connection Timeouts.

While we absolutely sympathize, we do not possess the power to control or change the Firewall. If you'd like to submit a firewall change request to IT, use this link to understand the process, as it can be a bit cumbersome.

We are in talks with GSO/IT about utilizing a more modern process, but getting the powers-that-be to approve changes like this are akin to turning the Titanic. All we ask is your patience and understanding.

Hardware and Performance

Based on this doc note, running multiple agents on the same instance will impact the pipeline performance. Have you compared running one agent per instance and multiple agents per instance?

That note says it may impact performance, not that is necessarily will. That all depends on the amount of consumed resources the agents use from the host. We have not seen this to be the case, as each EC2 instance is pretty beefy. You're looking at 16 vCPU, 32GiB of memory, and a 10Gbps NAT. That means each agent can effectively consume 4 vCPUs at 100%, 8GiB of memory space, and transmit/receive at 2.5Gbps without ever overlapping resources (which can be done at the hardware level based on CPU scheduling, memory re-addressing, and other such firmware/kernel management that is a little too in-depth for this FAQ).

Have we compared the performance metrics between running a pipeline using Microsoft-hosted agents and self-hosted agents on AWS EC2 instances?

There's no good way to get performance metrics from running Agents, so it's next to impossible to compare. That's an oft requested feature that is consistently disregarded by Microsoft. If you just want to compare hardware, the hosted agents we designed will win hands-down. The standard hardware on a Microsoft-hosted agent is 2 vCPU & 7GiB of RAM which is less than what DevEx offers. Not only that, but Microsoft themselves say self-hosted agents have, in many cases, performance advantages over their own hosted agents.

Security

Are the agents public?

Meaning, do they have access to the public internet? Yes, but only through the Checkpoint firewall. Direct access to public internet is not advised, as these agents build and test intellectual property code. It's significantly safer to use these than Microsoft-hosted ones because of that.

Are the maintenance activities on the EC2 instances, like security patches, updates, etc, automated?

Kind of? That's a deeper discussion with the Security Team, but that being said, the AMIs get re-built every 4 days. This is to ensure that the agents have the latest security patches from new releases (e.g. log4j). It's common practice to do releases on a consistent schedule, as well as iteratively when the code-based changes (new AMI requirements). Both Microsoft and Amazon internally have an automatic release cadence. It's an easy way to catch issues in patches, software, systems, services, etc. because the releases are so frequent. It also ensures there is at least one deployment a week, which validates the entire deployment process.