Skip to content

Commit

Permalink
[BE] Migrate Anaconda Prune jobs from CircleCI to GHA (pytorch#93876)
Browse files Browse the repository at this point in the history
We need periodical anaconda prune jobs to remove older packages (e.g. pytorch, torchvision, torchaudio, torchtext etc) from channels like pytorch-nightly and pytorch-test.
Currently it is done in circleci (e.g. https://app.circleci.com/pipelines/github/pytorch/pytorch/647201/workflows/72e5af30-0d54-44c1-8d9b-4c5502d27c9d/jobs/17260775) and triggered by postnightly update (https://github.com/pytorch/pytorch/tree/postnightly)

However, this postnightly branch triggers so many useless jobs (dozens of them failed due to docker command too long. Why? Because change history was part of docker command and it exceeds max INT).

<img width="756" alt="image" src="https://user-images.githubusercontent.com/109318740/216139179-3c913094-82cb-4605-99b7-23a21b4cbb36.png">

Therefore, we should stop the postnightly jobs (waste of resources) but save anaconda prune jobs.
This PR attempts to achieve this.
Pull Request resolved: pytorch#93876
Approved by: https://github.com/atalman
  • Loading branch information
weiwangmeta authored and pytorchmergebot committed Feb 2, 2023
1 parent ca9ebf9 commit dd8662d
Show file tree
Hide file tree
Showing 2 changed files with 76 additions and 0 deletions.
37 changes: 37 additions & 0 deletions .github/workflows/_prune-anaconda-packages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Prune Anaconda Binaries

on:
workflow_call:
inputs:
packages:
required: true
type: string
description: The packages to prune
channel:
required: true
type: string
description: The channel to prune packages
secrets:
conda-pytorchbot-token:
required: true
description: Conda PyTorchBot token
jobs:
build:
runs-on: ubuntu-22.04
container:
image: continuumio/miniconda3:4.12.0
steps:
- name: Checkout PyTorch
uses: pytorch/pytorch/.github/actions/checkout-pytorch@master
with:
no-sudo: true

- name: Prune binaries
env:
CHANNEl: ${{ inputs.channel }}
PACKAGES: ${{ inputs.packages }}
ANACONDA_API_TOKEN: ${{ secrets.conda-pytorchbot-token }}
run: |
set -ex
conda install -yq anaconda-client
bash ./scripts/release/anaconda-prune/run.sh
39 changes: 39 additions & 0 deletions .github/workflows/anaconda-prune.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: anaconda-prune

on:
schedule:
- cron: 45 1,7,13,19 * * *
push:
branches:
- postnightly
- weiwangmeta/migrate_anaconda_prune_to_gha
pull_request:
paths:
- .github/workflows/anaconda-prune.yml
- .github/workflows/_prune-anaconda-packages.yml
- scripts/release/anaconda-prune/run.sh
- scripts/release/anaconda-prune/prune.sh
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}-${{ github.event_name == 'workflow_dispatch' }}
cancel-in-progress: true

jobs:
anaconda-prune-pytorch-nightly:
name: anaconda-prune-pytorch-nightly
uses: ./.github/workflows/_prune-anaconda-packages.yml
with:
packages: "pytorch torchvision torchaudio torchtext ignite torchcsprng"
channel: pytorch-nightly
secrets:
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}

anaconda-prune-pytorch-test:
name: anaconda-prune-pytorch-test
uses: ./.github/workflows/_prune-anaconda-packages.yml
with:
packages: "pytorch torchvision torchaudio torchtext ignite torchcsprng"
channel: pytorch-test
secrets:
conda-pytorchbot-token: ${{ secrets.CONDA_PYTORCHBOT_TOKEN }}

0 comments on commit dd8662d

Please sign in to comment.