Skip to content

Overview on Reporting

Introduction

This document intends to capture the details associated with the traceability and the test execution reporting. Information about the EA requirements, the data that is available in the warehouse, the source of the data, and the report generation.

EA requirements

  • Repository carelink-ea-requirements in the carelink project of MDTProductDevelopment organization acts as a placeholder to store the changed requirements exported out of Enterprise Architect tool.
  • Pipeline devops-ea-requirements-export in the enablement project has been established to trigger whenever file has been uploaded to the repository's main branch (via PR approval).
  • Pipeline runs the associated powershell script, that scans the ea requirements in the csv file and uploads the details to the data warehouse (Cosmos DB).

How to update EA Requirements in the cosmos datawarehouse

Repository location - carelink-ea-requirements 1. Latest EA extract(CSV file) can be replaced with the carelink.csv file in the ea-extract-requirements folder. 2. User will have to create a branch under this repository, replace the CSV file, and merge it to the main branch via a pull request. 2. Pipeline associated with this repository carelink-ea-requirements will be triggered on commit to the main branch (i.e. only when the pull request is approved) 3. Pipeline devops-ea-requirements-export has been established in the enablement project to automatically trigger after the pipeline 'carelink-ea-requirements' has completed its run. 4. Pipeline devops-ea-requirements-export in the enablement project will read the CSV file content and export details to the Cosmos DB container named specifications-feed in azdo-metrics-cosmos account in Microsoft Azure. 5. Please note that on every pipeline run, the data is erased from the Cosmos DB specifications-feed* container and inserted from scratch.

Sample data from the Cosmos DB container (specifications-feed)

"guid": "434A74ED-2EFB-4946-843B-1234-ZYXW", 
"name": "User Profile Access",
"type": "Requirement",
"status": "Approved ",
"feed": "carelink",  --- *name of the CSV file from which data is read*
"id": "c7403708-d6c3-458e-af7d-5552346678"

Cosmos DB setup

  1. Data warehouse has been setup in Microsoft Azure Cosmos DB account.
  2. Cosmos DB account azdo-metrics-cosmos* contains the containers required for traceability reporting.
  3. Database azdo-metrics-db-prod has the follwing containers
  4. specifications-feed : ea requirements from the CSV
  5. workitems-type : workitems in the carelink project (currently only the workitems of type 'Test Case' and 'Test Suite' are present)
  6. workitemrevisions-type : work item revisions(history) for all the workitems
  7. testruns-doctype : test runs available in the carelink project
  8. testresults-testrunid : associated test results of all the test runs
  9. Data from these containers can be loaded into reporting tools like PowerBI to create traceability reports.

Work-Items and Work-Item revisions for Test Case and Test Suites

  • Service Hook has been established on the carelink project to trigger on work item created, updated, deleted or restored events.

servicehook

  • Service Hook trigger has been setup to insert the event message into the storage account queue in Azure.

storageaccount

  • A function app has been created, that gets triggered whenever a message is inserted into the storage account queue.

functionapp

  • Function App reads the message and inserts the workitem and workitem revision details into the cosmos db.

cosmosdb

  • Function App can only handle the work item events that occurred after its deployment. For the already existing workitems, an exe will be run to insert the details (work items and work item revisions) to the cosmos db via pipeline.
  • Source code of the exe
  • Pipeline devops-import-workitems-onetime can be executed once to sync the details of work items that were created before the service hook was established.

For Test Runs and associated Test Results

  • Pipeline devops-import-testruns-carelink has been established to run every two hours.
  • This pipeline captures the test runs that were created/updated between the last pipeline run time and the current time.
  • When there are no records in the cosmos db, pipeline processes the details of all the existing test runs and associated test results.

Pipeline setup

  • Utility requires secret values (PAT, Cosmos DB Endpoint URL, Cosmos DB Key, and the Storage Account Key) to be stored in the Azure KeyVault.
  • A PAT with read permissions on Test management and workitems is required to fetch details from AzDO.
  • Following environment variables has to be set in the pipeline devops-import-workitems-onetime that runs the exe for one time sync -
    • containerToCurrentlyPopulate : Due to large number of workitems, either workitems or workitem revisions are captured at once. Specify the particular cosmos db container for either workitems or workitem revisions
    • cosmoscontainerwiname : Cosmos DB container that captures work item details.
    • cosmoscontainerwirevname : Cosmos DB container that captures work item revision details.
    • cosmosdbendpointurl : Endpoint url of cosmos db account.
    • cosmosDBId : Database Id of cosmos db.
    • organizationname : Azure DevOps organization of the project whose work item details are captured.
    • projectName : Project within the DevOps organization whose work item details are captured.
    • wiType : Type of the work item
  • Following environment variables has to be set in the pipeline devops-import-testruns-carelink that runs the one-time script for the existing work items-
    • organizationname : Azure DevOps organization of the project whose test run & results details needs to be captured.
    • projectName : Project within the DevOps organization whose test run & results details needs to be captured.
    • cosmosdbendpointurl : Endpoint url of cosmos db account.
    • cosmosDBId : Database Id of cosmos db.
    • cosmoscontainertestruns : Cosmos DB container that captures test run details.
    • cosmoscontainertestresults : Cosmos DB container that captures test result details.

Traceability report generation

  • Please follow the How-To-Guide to create traceability reports using Power BI.
  • Please follow the How-To-Guide to create test execution reports using Cosmos DB ODBC driver and Microsoft Excel.