Pull Request Documentation and Process

Overview

Having good documentation practices captured on pull requests is critical for both having high quality software and also maintaining a history for regulators to look at if an investigation is ever launched on work performed. This process discusses both good documentation practices on pull requests and a generalized process for documenting Pull Requests with Azure DevOps using Repos. Note that this process documents also the use of pull request templates to help people remember the contents and format for good documentation practices.

Generally, when documenting a pull request the following should be captured.

The changes made/work performed in this pull request. This can be a summary of all the pull request commit messages with a short statement to tie them all together.
If tests other than automated tests run by the CI pipeline associated with this branch were run. Document the testing in the pull request or mention that the testing is capture in the associated task(s).
Capture any notes for the reviewer that are important in reviewing this work.
Document which quality records exist inside this pull request and are a part of this review.
All pull request feedback closed with any unresolved issues marked for resolution with a bug, RPI or associated work item (user story/task).

Do not recreate the content of the Task documentation, only link to it with the pull request.

Description of the pull request

In the description section add a section that lists a bulleted or numbered list of all changes. When the pull request is created all of the commit messages associated with the request can be added to the description to give you a starting point. An example of that section is the following:

Improved error handling code and updated design. 
1. Updated base framework code to handle errors more gracefully. 
2. Updated design documentation to indicate how the error handler works. 
3. Corrected a defect in logging when attributes are missing.

Optionally, in this section you can add the update SHA from the commits tab in the PR in your documentation to help better explain each commit. This is the same as on TFS where a changeset ID was mentioned.

Improved error handling code and updated design. 
1. Updated base framework code to handle errors more gracefully. (dcdecfda) 
2. Updated design documentation to indicate how the error handler works. (dcdeeeda) 
3. Corrected a defect in logging when attributes are missing. (1sw3seew)

It is often useful to open this PR as draft at the beginning of your work and add these messages and notes as you do work.

Testing of the pull request

The second thing after the list of work performed that should be considered are any testing notes that should be captured. This can be as simple as the following:

Testing:
Automated unit testing executed and passed as part of this PR.

But if you also needed to execute manual tests as part of the PR you have two options. You can link to the related tasks and say the testing documentation is there or you can follow the same manual testing good documentation practices and document the testing inside the PR. If you document within the PR it is recommended inside the task's Integration Testing section to indicate all tests documented within the PR. Given manual tests documented within the task, a task link could be used to indicate these tests. The following would be what the testing section might look like in that case.

Testing:
1. Automated unit testing executed and passed as part of this PR. 
2. Manual testing documented in Task 232324 (Link)
3. Manual testing documented in Task 232325 (Link)

Or if in the event you choose to capture the testing in the PR, which is preferred this would be the format.

Testing:
1. Automated unit testing executed and passed as part of this PR.

2. Manual Test Executed
    Scenario: home page displays correctly, "Environment CL-Chaska7 CLN22 TRR6, OneApp v5.2 TRR 6"
    1. Login to Web site at https://www.helloworld.com using credential MyAccount, Default Password for Testing
    2. Select Menu Option "Things I want to test". 
    3. Verify, On the top of the page it indicates "Lots of testing to do".

    PASSED, AUGUST-22-2021, PJL

3. Manual Test Executed
    Scenario: Other page is correct, "Environment CL-Chaska7 CLN22 TRR6, OneApp v5.2 TRR 6"
    1. Login to Web site at https://www.helloworld.com using credential MyAccount, Default Password for Testing
    2. Select Menu Option "Things I don't want to test". 
    3. Verify, On the bottom of the page it shows "And that is all I don't want to test.".

    FAILED, AUGUST-22-2021, PJL, Issue documented in RPI 232322 (Link)

Manual Test Documentation Format

When running manual tests you need to document the test protocol/steps. The following is an example of one of these tests.

Scenario: Web pages display correctly, "Environment CL-Chaska7 CLN22 TRR6, OneApp v5.2 TRR 6"
1. Login to Web site at https://www.helloworld.com using credential MyAccount, Default Password for Testing
2. Select Menu Option "Things I want to test". 
3. Verify, On the top of the page it indicates "Lots of testing to do".

   PASSED, AUGUST-22-2021, PJL

4. Select Menu Option "Things I don't want to test". 
5. Verify, On the bottom of the page it shows "And that is all I don't want to test.".

   FAILED, AUGUST-22-2021, PJL, Issue documented in RPI 232322 (Link)

Just like automated tests, manual test verification steps should show the environment/tools in use (if applicable), if the test passed or failed, the date executed, initials of who ran the test and if it failed links to work items for repair or future analysis.

Other Notes in the pull request

A process cannot capture any notes that are valuable but this section is useful to indicate notes to the reviewer that will help them perform the review. An example might be a block of code you wish to explain. The section should look like

Notes:
* Inside PresentationComplexifier.cs, Line 34 you will see that I hard coded an environment URL. A backlog user story 342343 (link) is planning on migrating the entire code base to be multiple environment aware so I didn't consider that right now. 
* It was decided that part of the user story will be completed in a later story so code is being submitted in this PR that is not yet referenced. It is intended on being used in that later phase of the project.

Quality record notes

If your team is following the recommended convention based review process you should list each type of quality record that is contained in this PR. This can be done with a section like the following:

Note: Contains Quality Records
* Component Specifications
* Software Code

Teams can decide if they leave this out when the contents under review are software code and nothing else. Common quality record types per our LDC process are the following:

System Architecture Specifications
Finished Device Specifications
HFE Quality Record
Component Specifications (Requirements)
Component Specifications (Architecture)
Component Specifications (Design)
Traceability Matrix
Software Code
Firmware Code
Build Audit Report
Verification Plan
Verification Protocol
Verification Matrix
Verification Report
Verification Results
Method Validation Record
Software Archive and Transfer Media
Firmware Archive and Transfer Media
Software Version Description Document (VDD)
Software Deployment Procedure
SOUP Item List

Feedback documentation

Good documentation practices are also required on feedback to questions during the review.

If the request is resolved through a change indicate that as you resolve and close the comment with a statement like:

Changes requested made.
If the request will be resolved in a future work item indicate that and link to the work item using the hashtag link technique.

Changes deferred until later work. Will be addressed in User Story 23323 (Link)
If the request/comment will not be resolved, document why using clear language.

Not making these changes as they would impact the ability to support the system.

Using Pull Request templates

One or more pull request templates can be saved at a Azure DevOps repository default branch level. These can be used to initialize the work item following the notes and template the team wants to use. A base template called "pull_request_template.md" will be provided for all teams to use by the Simplified DevOps Architecture teams.

To create a template do the following

If updating the global pull request template create a file named pull_request_template.md with the template and check it into the develop branch (assuming this is the repo's default branch).
If creating a pull request template for a sub/branch create in the develop branch (assuming this is the repo's default branch) create the folder /.azuredevops/pull_request_template/branches and add a md document named after the branch. For example for all feature branches create the file /.azuredevops/pull_request_template/branches/feature.md
Any template in these folders not named after the default name pull_request_template.md or a branch specific name can be selected optionally after the PR is opened with the "Add a template" option in the pull request.

Our default pull request template

The built in markdown document template would look like the following:

Summary of work performed documented here.

* First Change Made
* Second Change Made

**Testing:**
    Automated unit testing executed and passed as part of this PR.

**Notes:**
* First note to the reviewer. 
* Second note to the reviewer.

**Note: Contains Quality Records:**
* Software/Firmware Code

**Author Check List:**
- [ ] Changes made documented
- [ ] Testing Documented
    - [ ] Automated unit testing
    - [ ] Links to tasks with documentation of manual/integration testing, if any
    - [ ] Manual testing documented in this PR, if any
- [ ] Quality records under review noted. 
- [ ] Notes to help reviewer documented. 
- [ ] All tests pass or work items are linked for resolution.
- [ ] All comments are resolved or work items are linked for future resolution.

This would produce a description with the following check boxes process and layout for the user to edit.

Other references

Online documentation on using templates with pull requests can be found on the Online Azure DeVops documentation.