{Regression test} Enhance Bump Version Pipeline to be Regression Test…

… Pipeline (Azure#23931)

* accept custom repo & branch

* condition variable

* login password

* docs

doc/how_to_bump_SDK_version_in_cli.md

@@ -5,39 +5,58 @@ This article aims to provide a guide for CLI developers and contributors to bump
 ## Overview
 
 Developers need to do several things for bumping version:
-1. Upgrade the SDK version in CLI dependency
-2. Update the used API version if defined in Cli Core
+1. Upgrade CLI dependency
+    - Upgrade the SDK version in [setup.py](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/setup.py), [requirements.py3.windows.txt](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.windows.txt), [requirements.py3.Linux.txt](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.Linux.txt) and [requirements.py3.Darwin.txt](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.Darwin.txt)
+2. Update the used API version
+    - If it's not multi-api SDK, then nothing needed in this step
+    - If the SDK contains multiple APIs, the specified API version needs to be updated in CLI Core [AZURE_API_PROFILES](https://github.com/Azure/azure-cli/blob/ce74ae358b51aedfdfb6c32042b515d949618e33/src/azure-cli-core/azure/cli/core/profiles/_shared.py#L147) for 'latest' profile
 3. Regression test to check if it breaks existing features
+    - Run `azdev test --no-exitfirst` to figure out all test failures in all modules after bumping version
+    - Run `azdev test --live --lf --no-exitfirst` to rerun all the failed tests in live mode
+    - Tests failed in live run may be affected by breaking changes in new SDK/API version and need to be fixed manually in next step
 4. Make the code changes to
     - fix the regression
     - add new features
 
-We can divide all the work into 2 phases.
+Considering bumping version is time consuming and most of the work is repetitive, we have `Regression Test Pipeline` to help with step 1-3.
 
-## Phase 1 - Trigger Bump Version Pipeline
+## Regression Test Pipeline
 
-The [Bump Version Pipeline](https://dev.azure.com/azure-sdk/internal/_build?definitionId=4949) is designed to handle all the repetitive work. It will:
-- Upgrade the SDK version in [setup.py](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/setup.py), [requirements.py3.windows.txt](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.windows.txt), [requirements.py3.Linux.txt](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.Linux.txt) and [requirements.py3.Darwin.txt](https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.Darwin.txt)
-- Update the API version defined in CLI Core [AZURE_API_PROFILES](https://github.com/Azure/azure-cli/blob/ce74ae358b51aedfdfb6c32042b515d949618e33/src/azure-cli-core/azure/cli/core/profiles/_shared.py#L147) for 'latest' profile
-- Run automation full test for all cli command modules in recording mode
-- Rerun failed tests of last step in live mode, keep the new recording yaml files for those successful tests during live rerun
-- Create a PR to CLI repo with all version changes and new test recordings
+The [Regression Test Pipeline](https://dev.azure.com/azclitools/internal/_build?definitionId=220) is designed to handle all the repetitive work. It will take care of step #1 ~ #3 with a PR created to CLI repo with all code changes and test recordings.
 
-To trigger a new execution of Bump Version Pipeline, developers need to fulfill these variables:
-- PACKAGE: **Required**. The SDK package name, eg. `azure-mgmt-network`.
-- TARGET_PACKAGE_VERSION: **Required**. The SDK new version, eg. `19.3.0`.
-- RESOURCE_TYPE: Required if you want to update the API version definition in CLI Core. The resource type enum name define in [CLI Core](https://github.com/Azure/azure-cli/blob/ce74ae358b51aedfdfb6c32042b515d949618e33/src/azure-cli-core/azure/cli/core/profiles/_shared.py#L38), eg. `MGMT_NETWORK`.
-- TARGET_API_VERSION: Required if you want to update the API version definition in CLI Core. The new API version or API versions:
-    - If the AZURE_API_PROFILES['latest'][RESOURCE_TYPE] definition is `str` type, then just fulfill this variable with new API version. For example, to update API version for [ResourceType.MGMT_NETWORK](https://github.com/Azure/azure-cli/blob/ce74ae358b51aedfdfb6c32042b515d949618e33/src/azure-cli-core/azure/cli/core/profiles/_shared.py#L150), use `2021-10-01` as variable value.
-    - If the  AZURE_API_PROFILES['latest'][RESOURCE_TYPE] definition is `SDKProfile` type, then fulfill this variable with 'operation=version' pairs separated by space. For example, to update API version for [ResourceType.MGMT_COMPUTE](https://github.com/Azure/azure-cli/blob/ce74ae358b51aedfdfb6c32042b515d949618e33/src/azure-cli-core/azure/cli/core/profiles/_shared.py#L151-L164), use `default=2022-05-01 snapshots=2022-03-01 gallery_images=2021-12-01` as variable value. `default` means other operations except for explicitly listed ones.
+Follow below steps to trigger a new execution of Regression Test Pipeline:
+
+Step 1. Click `Run pipeline` button, leave the `Branch/tag` and `Commit` as it is, open `Variables`
+
+![](assets/regression_test_new_pipeline.jpg)
+
+![](assets/regression_test_pipeline_branch.jpg)
 
-It may take some time for the pipeline execution. After PR created, developers can check the status and test results in linked pipeline.
+Step 2. Fulfill the variables and run
 
-## Phase 2 - Manual code changes
+![](assets/regression_test_variables.jpg)
 
-Developers can review the test failures from pipeline test result report or PR CI checks. These failed test can't pass live run after bumping version.
+![](assets/regression_test_run.jpg)
+
+### Pipeline Variables
+
+Developers can a) update code in your own repo&branch and then leverage this pipeline to do regression test or b) fully rely on pipeline to update versions and then do regression test
+
+For scenario a, variables `CUSTOM_REPO`, `CUSTOM_BRANCH` and `CUSTOM_GITHUB_TOKEN` are used.
+For scenario b, variables `PACKAGE`, `TARGET_PACKAGE_VERSION`, `RESOURCE_TYPE` and `TARGET_API_VERSION` are used.
+- CUSTOM_REPO: The forked repo name: `https://github.com/{CUSTOM_REPO}/azure-cli/`, eg. `azclibot`, `evelyn-ys`, etc. **Required for scenario a**. Leave it as empty for scenario b.
+- CUSTOM_BRANCH: The branch you used in your forked repo to develop features, eg. `dev`, `fix_XXX`, etc. **Required for scenario a**. Leave it as empty for scenario b.
+- CUSTOM_GITHUB_TOKEN: Github personal access token which allows commit changes to your repo and branch. See [creating a personal access token](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token). **Required for scenario a**. Leave it as empty for scenario b.
+- PACKAGE: The SDK package name, eg. `azure-mgmt-network`. **Required for scenario b**. Leave it as empty for scenario a.
+- TARGET_PACKAGE_VERSION: The SDK new version, eg. `19.3.0`. **Required for scenario b**. Leave it as empty for scenario a.
+- RESOURCE_TYPE: The resource type enum name define in [CLI Core](https://github.com/Azure/azure-cli/blob/ce74ae358b51aedfdfb6c32042b515d949618e33/src/azure-cli-core/azure/cli/core/profiles/_shared.py#L38), eg. `MGMT_NETWORK`. Required if you want to update the API version definition in CLI Core. Usually used for multi-api SDKs. Leave it as empty for scenario a.
+- TARGET_API_VERSION: The new API version(s):
+    - If the AZURE_API_PROFILES['latest'][RESOURCE_TYPE] definition is `str` type, then just fulfill this variable with new API version. For example, to update API version for [ResourceType.MGMT_NETWORK](https://github.com/Azure/azure-cli/blob/ce74ae358b51aedfdfb6c32042b515d949618e33/src/azure-cli-core/azure/cli/core/profiles/_shared.py#L150), use `2021-10-01` as variable value.
+    - If the  AZURE_API_PROFILES['latest'][RESOURCE_TYPE] definition is `SDKProfile` type, then fulfill this variable with 'operation=version' pairs separated by space. For example, to update API version for [ResourceType.MGMT_COMPUTE](https://github.com/Azure/azure-cli/blob/ce74ae358b51aedfdfb6c32042b515d949618e33/src/azure-cli-core/azure/cli/core/profiles/_shared.py#L151-L164), use `default=2022-05-01 snapshots=2022-03-01 gallery_images=2021-12-01` as variable value. `default` means other operations except for explicitly listed ones.
+
+    Required if you want to update the API version(s) in CLI Core. Leave it as empty for scenario a.
 
-Developers need to figure out the failure root cause and fix them. After all the tests fixed, feel free to implement new features in the same PR or a separate one.
+It may take some time for the pipeline execution. After PR created, developers can check the status and test results in linked pipeline. The test results can show failed tests which can't pass live run after bumping version. Developers need to figure out the failure root cause and fix them.
 
 ## See also
 

scripts/ci/automation_full_test.py

@@ -21,7 +21,7 @@
 
 # sys.argv is passed by
 # .azure-pipelines/templates/automation_test.yml in section `Running full test`
-# scripts/bump_version/bump_version.yml in `azdev test` step
+# scripts/regression_test/regression_test.yml in section "Rerun tests"
 instance_cnt = int(sys.argv[1])
 instance_idx = int(sys.argv[2])
 profile = sys.argv[3]

scripts/bump_version/bump_version.yml → scripts/regression_test/regression_test.yml

@@ -1,4 +1,4 @@
-name: CLI BUMP VERSION $(PACKAGE) $(TARGET_PACKAGE_VERSION) $(Date:yyyyMMdd)$(Rev:.r)
+name: CLI REGRESSION TEST $(Date:yyyyMMdd)$(Rev:.r)
 
 resources:
 - repo: self
@@ -14,6 +14,7 @@ variables:
 jobs:
 - job: UpdateVersionFiles
   displayName: CLI Bump Version Update Version Files
+  condition: ne(variables['PACKAGE'], '')
   pool:
     name: ${{ variables.ubuntu_pool }}
   steps:
@@ -59,9 +60,10 @@ jobs:
           git push --set-upstream azclibot bump_version_$(Build.BuildId)
 
 - job: RerunTests
-  displayName: CLI Bump Version Rerun tests
+  displayName: CLI Regression tests
   dependsOn: UpdateVersionFiles
-  timeoutInMinutes: 9999
+  condition: in(dependencies.UpdateVersionFiles.result, 'Succeeded', 'Skipped')
+  timeoutInMinutes: 0
   strategy:
     maxParallel: 8
     matrix:
@@ -96,21 +98,28 @@ jobs:
         inlineScript: |
           set -ev
           # git config
-          GITHUB_TOKEN=$(az keyvault secret show --vault-name kv-azuresdk --name azclibot-pat --query value -otsv)
+          if [[ -n "$(CUSTOM_REPO)" && -n "$(CUSTOM_BRANCH)" && -n "$(CUSTOM_GITHUB_TOKEN)" ]]; then
+            GITHUB_REPO=$(CUSTOM_REPO)
+            GITHUB_BRANCH=$(CUSTOM_BRANCH)
+            GITHUB_TOKEN=$(CUSTOM_GITHUB_TOKEN)
+          else
+            GITHUB_REPO="azclibot"
+            GITHUB_BRANCH="bump_version_$(Build.BuildId)"
+            GITHUB_TOKEN=$(az keyvault secret show --vault-name kv-azuresdk --name azclibot-pat --query value -otsv)
+          fi
           git config --global user.email "[email protected]"
           git config --global user.name "Azure CLI Team"
-          git remote add azclibot https://azclibot:${GITHUB_TOKEN}@github.com/azclibot/azure-cli.git
-          git fetch azclibot
+          git remote add ${GITHUB_REPO} https://${GITHUB_REPO}:${GITHUB_TOKEN}@github.com/${GITHUB_REPO}/azure-cli.git
+          git fetch ${GITHUB_REPO}
 
-          git checkout -b bump_version_$(Build.BuildId) azclibot/bump_version_$(Build.BuildId)
+          git checkout -b ${GITHUB_BRANCH} ${GITHUB_REPO}/${GITHUB_BRANCH}
     - template: ../../.azure-pipelines/templates/azdev_setup.yml
     - bash: |
         set -ev
 
         source env/bin/activate
-        pip show $(PACKAGE)
 
-        az login -u $(CLI_LIVE_TEST_ACCOUNT) -p $(CLI_LIVE_TEST_PASSWORD)
+        az login -u $(CLI_LIVE_TEST_ACCOUNT) -p "$(CLI_LIVE_TEST_PASSWORD)"
         az account set -s 0b1f6471-1bf0-4dda-aec3-cb9272f09590
 
         serial_modules="appservice botservice cloud network azure-cli-core azure-cli-telemetry"
@@ -119,10 +128,10 @@ jobs:
     - task: PublishTestResults@2
       inputs:
         testResultsFiles: '/mnt/vss/.azdev/env_config/mnt/vss/_work/1/s/env/test_results_*.xml'
-        testRunTitle: 'CLI Bump Version test results of instance $(Instance_idx)'
+        testRunTitle: 'CLI Regression test results of instance $(Instance_idx)'
 
 - job: CreatePR
-  displayName: CLI Bump Version Create PR
+  displayName: CLI Regression Test Create PR
   dependsOn: RerunTests
   pool:
     name: ${{ variables.ubuntu_pool }}
@@ -135,14 +144,20 @@ jobs:
         inlineScript: |
           set -ev
           # git config
-          GITHUB_TOKEN=$(az keyvault secret show --vault-name kv-azuresdk --name azclibot-pat --query value -otsv)
+          if [[ -n "$(CUSTOM_REPO)" && -n "$(CUSTOM_BRANCH)" && -n "$(CUSTOM_GITHUB_TOKEN)" ]]; then
+            GITHUB_TOKEN=$(CUSTOM_GITHUB_TOKEN)
+            pr_title="Regression Test for $(CUSTOM_REPO)/$(CUSTOM_BRANCH)"
+            pr_body="Triggered by CLI Regression Test Pipeline - ADO_BUILD_ID=$(Build.BuildId)\n\nRegression test for $(CUSTOM_REPO)/$(CUSTOM_BRANCH)\nThere may still exist some failed tests, see [details](https://dev.azure.com/azure-sdk/internal/_build/results?buildId=$(Build.BuildId)&view=ms.vss-test-web.build-test-results-tab) in ADO pipeline"
+            pr_head="$(CUSTOM_REPO):$(CUSTOM_BRANCH)"
+          else
+            GITHUB_TOKEN=$(az keyvault secret show --vault-name kv-azuresdk --name azclibot-pat --query value -otsv)
+            pr_title="Regression Test for $(PACKAGE) to $(TARGET_PACKAGE_VERSION)"
+            pr_body="Triggered by CLI Regression Test Pipeline - ADO_BUILD_ID=$(Build.BuildId)\n\npackage: $(PACKAGE) $(TARGET_PACKAGE_VERSION)\nresource_type: $(RESOURCE_TYPE) $(TARGET_API_VERSION)\n\nThere may still exist some failed tests, see [details](https://dev.azure.com/azure-sdk/internal/_build/results?buildId=$(Build.BuildId)&view=ms.vss-test-web.build-test-results-tab) in ADO pipeline"
+            pr_head="azclibot:bump_version_$(Build.BuildId)"
+          fi
           git config --global user.email "[email protected]"
           git config --global user.name "Azure CLI Team"
 
-          pr_title="{Package} Bump version for $(PACKAGE) to $(TARGET_PACKAGE_VERSION)"
-          pr_body="Triggered by CLI Bump Version Pipeline - ADO_BUILD_ID=$(Build.BuildId)\n\npackage: $(PACKAGE) $(TARGET_PACKAGE_VERSION)\nresource_type: $(RESOURCE_TYPE) $(TARGET_API_VERSION)\n\nThere may still exist some failed tests, see [details](https://dev.azure.com/azure-sdk/internal/_build/results?buildId=$(Build.BuildId)&view=ms.vss-test-web.build-test-results-tab) in ADO pipeline"
-          pr_head="azclibot:bump_version_$(Build.BuildId)"
-
           curl \
           -H "Authorization: token ${GITHUB_TOKEN}" \
           -d "{\"title\": \"${pr_title}\", \"body\": \"${pr_body}\", \"head\": \"${pr_head}\", \"base\": \"dev\"}" \