Skip to content

Requesting New Azure DevOps Extensions to the Organization

Overview

According to Microsoft extensions are simple add-ons that can be used to customize and extend your experience with Azure DevOps. The marketplace is home to hundreds of extensions that can be installed to help with the following tasks:

  • Planning and tracking work items, sprints, scrums, and so on
  • Pipelines build and release flows
  • Code testing and tracking
  • Collaboration among team members.

In addition to extensions being provided for free and cost by the community we can also build extensions. Our current stance is to minimize customization of Azure DevOps that isn't provided by 3rd party extensions or the Azure DevOps product itself. This reduces operational maintenance as Azure DevOps is modified by Microsoft in a 3-week sprint cycle and the continual review of extensions is quite a lot of potential work.

Because these extensions are installed organization wide we have an official process to request, maintain and retire extensions that is described in this document.

Requesting a new extension

If you have a need not being served by the Azure DevOps product directly but find in the marketplace an extension that might help with that need this is the process you would follow to request that be added to the system.

  1. Request that the extension be added to our Sandbox environment by following our support request processes (link TBD).
  2. Evaluate the extension in the sandbox environment and if you no longer want to evaluate or consider the extension please request that it is deactivated in the sandbox org.

  3. If you after performing evaluation of the extension want to have it in the official org you must do the following steps:

  4. Develop and document a series of tests that show that the extension is performing the functions we want correctly.

  5. Submit a pull request to the official extension inventory markdown document (link TBD) by adding your extension to that list. In this pull request you are updating the inventory to point to a child markdown document with the procedure for extension testing, the contact for who will provide this testing when the extension is updated, and the purpose of the extension.
  6. The SDA teams will evaluate the extension for 4 things:
    1. Need - Does the extension duplicate functionality already present with another extension or the core Azure DevOps product.
    2. Validation Plan - Does the requester have a process of validation for the extension features in use.
    3. Ownership Plan - Does the requester have an assigned individual (and backup) who is responsible for testing the extension periodically on updates.
    4. Policies - Does the privacy, licensing, support and potential costs meet the requirements for use in regulated software development.
  7. If the SDA team finds that the extension doesn't meet the criteria above it will be rejected for addition. A "board" of managers is available to request approval variation in special situations but they will also consider the above 4 things in their evaluation.
  8. If the extension is approved for use it will be installed per the teams support SLA.
  9. The people involved in the ownership plan will be added to the "Extensions have been modified" notification e-mails.

Example of the Markdown Inventory being maintained

Extension Version Purpose Validation Link Last Performed Owner(s)
.NET Core Global Tool Installer 0.1.6 Allows installations of dotnet core global tools on build agents to be able to build using various dotnet core related technologies. val-dotnetcoretoolver.md Jan-30-2019 Paul Leska, Isaac Johnson

... and many more records.

To see the real current inventory look at the following repository document (link TBD).

Ongoing extension maintenance

As an extension is installed a notification will be added to include the people involved in the "ownership plan" in addition to the SDA support team members. It is the responsibility for the people who own the extension to re-execute their validation plan in a timely manner (30 days) and submit a pull request on the official extension inventory markdown document (link TBD) stating the date of last testing and the new version.

Do not assume the SDA support team will own the extension if requested. In most cases the person requesting the extension is responsible for producing an ownership plan for the extension. This will require the people involved in this ownership plan to periodically test and update documentation for the extensions they get approved. Some extensions will be owned by the SDA support team but not most.

If a tool falls too behind in validation support the SDA team may deactivate the extension until such time that the validation has been updated. Also, changes in the core Azure DevOps product may require re-execution of the validation work not just the tool being updated automatically by the publisher. In this case the SDA team while doing normal product vigilance of SOUP may request the team to re-execute their validation and submit a PR on the tooling status.

Extension Validation Markdown Document

Each record on the official extension inventory points to a validation markdown document. It documents the procedure and history of execution of the validations performed on this extension.

An extension validation document needs to contain the following:

  1. Purpose and Scope of the document
  2. Mention the purpose of the document that it captures evidence of the extension method validation
  3. Scope that applies to the evaluation - The Regulation / Standard Requirement described in the Design Controls
  4. A description of the functions the extension need to perform
  5. Validation evidence: Validation evidence for each criteria
  6. Criteria(Risk)
    • Risk Identification - Identify the risk caused to the product on malfunction or failure of the software tool
    • Risk Mitigation - Mention how the identified risks can be mitigated
  7. Criteria - Quality
    • Check to identify if the use of these methods will cause quality issues with any medical device(s)
  8. Criteria - Calibration
    • Identify if the method requires or contains Calibration
  9. Criteria - Statistical Techniques
    • Identify if the method requires or contains Statistical Techniques
  10. Criteria - Sampling Methods
    • Identify if the method requires or contains Sampling Methods
  11. Criteria – Requirements
    • List the Extension Requirements
  12. Criteria - Validation Protocol
    • A list of tests/test steps documenting the procedure for validating that the tooling successfully performs the functions needed. This testing should be done against the sandbox org environment to not pollute the live environment with tooling tests.
    • A journal in table form of the extension version, test date, person who executed the test and the results.
  13. Criteria - Configuration
    • Mention the Source Control, Documentation, Licensing details and how the Future Changes will be handled
  14. Criteria - Quality System
    • Mention the Quality System Tool where the extension will be retained
  15. History
    • A journal in table form of date, tool version and test results

The following is an example of this:


1. Purpose
This document captures evidence that the .NET Core Global Tool Installer, version 0.1.6, has been validated.

2. Scope
This document applies to Regulation / Standard Requirement as described in Design Controls: Method Validation, DOC000643 Rev J.

3. Method Description
Azure Devops does not provide a built in way to install .NET core global tools used in CI/CD pipelines. This tool uses the dotnet cli to install a .NET Core Global Tool and adds that to the agent's tool cache.

4. Validation Evidence
This section describes the validation evidence for each criterion specific to this method.

4.1. Criteria - Risk
4.1.a. Risk Identification
Will a malfunction or failure of the software tool:
- Cause an adverse impact on the product? **No**
- Cause unintended or unexpected changes to product? **No**
- Cause an incorrect quality record? **No**
- Cause failure to detect product non-conformities or problems? **No**
- Cause an adverse impact on product quality? **No**
- Cause incorrectly recorded test results, particularly false positives? **No**
- Cause lost or changed records? **No**
- Result in patient, customer or employee harm? **No**

4.1.b. Risk Mitigation

This tool only installs tooling. The actual tooling is run in the CI/CD pipeline and produces logs of results. Any failures or miscommunications will be detected by the tool installed and not this extension. 

4.2. Criteria - Quality
The use of these methods will not result in quality issues with any medical device(s). 

4.3. Criteria - Calibration
N/A. this method does not contain or require calibration

4.4. Criteria - Statistical Techniques
N/A. This method does not contain any or require Statistical Techniques.

4.5. Criteria - Sampling Methods
N/A. This method does not contain or require any Sampling Methods.

4.6. Criteria – Requirements
.NET Core Global Tool installer must be able to
1. Install a public global .net core tool and add to the agent's tool cache. 

4.6.1 Additional Requirements for Spreadsheets:
N/A. This method is not a Spreadsheet.

4.6.2 Additional Requirements for Quality Systems:
N/A. This method is not a Quality System.

4.7. Criteria - Validation Protocol

1. Install a public global .net core tool
    > Precondition: Extension is installed and available for use
    - Steps:
        1. Create a pipeline that installs the global tool dotnetsay with the syntax documented below. 
        2. Verify that the dotnetsay global tool can be run after the tool was installed. 

            pool:
              vmImage: windows-latest

            steps: 
            - task: DotnetGlobalToolInstaller@0
              inputs:
                name: 'dotnetsay'
                versionSpec: '2.1.*'
                checkLatest: true

            - task: CmdLine@2
              inputs:
                script: 'dotnetsay'

*Validation Results*

All tests passed. 

__Tested Configuration__: Azure DevOps as of July-2-2021, .NET Core Global Tool Installer version 0.1.6 

| Test Case | Last Performed | Tester | Test Outcome | Evidence |
|-----------|----------------|--------|--------------|----------|
| Install a public global .net core tool | July-2-2021 | *Tester Name* | Passed  | *Screenshot of evidences* |


4.8. Criteria - Configuration
The method is located in Azure DevOps
- Source code can be found in [Github](https://github.com/petersendev/azure-devops-dotnet-tool)
- Documentation: Refer [Marketplace](https://marketplace.visualstudio.com/items?itemName=petersendev.dotnet-global-tool-installer) for user documentation
- Licensing information can be found [here](https://marketplace.visualstudio.com/items/petersendev.dotnet-global-tool-installer/license)
- Future Changes: Future changes will be managed through Azure DevOps and validated through the Design Controls: Method Validation, DOC000643 process


4.9. Criteria - Quality System
N/A. This method is not a Quality System.

5 Conclusion
The __.NET Core Global Tool Installer 0.1.6__ is valid for its intended use and capable of producing valid results.

6. Abbreviations:
N/A

7. History:

| Date        | Tool Version | Test Results           |
|-------------|--------------|------------------------|
| Jan-27-2019 | 0.1.6        | All tests PASSED - PJL |
| Oct-22-2018 | 0.1.5        | All tests PASSED - PJL |

Extension end of life

Extensions may no longer be used when they are no longer being maintained by their publisher and/or we have an alternative extension or built in feature that replaces the need for the extension.

If you have an extension that has reached end of life please follow this process:

  1. Submit a PR removing the extension and related validation record from the official inventory records. In the PR state the reason why the tool is no longer should be supported. This should be done by someone who is listed as the extension owner.
  2. The SDA support team will give in our official teams channel a 30 day notice (if possible) of the date the extension will be deactivated.
  3. At the planned end of life date the tool will be deactivated but not removed. This is to serve as a reminder that we no longer wish to use this tool (since it is no longer on our inventory).
  4. The SDA support team will then send a notification to the official teams channel that the extension has been officially retired.

One complex scenario is when a group wants to replace an existing extension with another extension. It is suggested that the people wanting to do this perform the evaluation of the new extension in the sandbox org and then setup a meeting with the SDA Azure DevOps admins, the ownership team of the current extension and the people who want the new extension to discuss the option of switching extensions.