Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Mariner Arm64 docker image distribution. #2891

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker/DockerfileMariner
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Create container based on Ubuntu-22.04 Jammy Jellyfish image
# Create container based on mariner 2.0 image
FROM mcr.microsoft.com/cbl-mariner/base/core:2.0

# Install dependencies
Expand Down
15 changes: 15 additions & 0 deletions docker/DockerfileMarinerArm64
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Create container based on mariner 2.0 image
FROM --platform=linux/arm64 mcr.microsoft.com/cbl-mariner/base/core:2.0

# Install dependencies
RUN tdnf update -y
RUN tdnf install -y ca-certificates

# Copy azcopy binary to executable path
COPY ./azcopy /usr/local/bin/

# Make azcopy executable
RUN chmod +x /usr/local/bin/azcopy

WORKDIR /azcopy
CMD [ "azcopy" ]
107 changes: 19 additions & 88 deletions release-pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1837,98 +1837,29 @@ stages:
dependsOn: TestArtifacts
condition: succeeded('TestArtifacts')
jobs:
- job: Set_1_Ubuntu_Mariner_AMD64
strategy:
- template: setup/build_publish_docker.yml
parameters:
jobName: Set_1_Ubuntu_Mariner_AMD64
useDemands: false
matrix:
Ubuntu_amd64:
agentName: "blobfuse-ubuntu22"
vmImage: 'Ubuntu-22.04'
container: 'test-cnt-ubn-22'
pool:
vmImage: $(vmImage)

variables:
- group: AZCOPY_SECRET_VAULT
- name: root_dir
value: '$(System.DefaultWorkingDirectory)'
- name: work_dir
value: '$(System.DefaultWorkingDirectory)/azure-storage-azcopy'

steps:
- checkout: none
- script: |
git clone https://github.com/Azure/azure-storage-azcopy
displayName: 'Checkout Code'
workingDirectory: $(root_dir)

- script: |
git checkout `echo $(Build.SourceBranch) | cut -d "/" -f 1,2 --complement`
displayName: 'Checkout branch'
workingDirectory: $(work_dir)

- script: |
chmod 777 *.sh
./dockerinstall.sh

./buildcontainer.sh Dockerfile ubuntu-x86_64
./publishcontainer.sh $(AZCOPY_DOCKER_REG_USER) $(AZCOPY_DOCKER_REG_PWD) ubuntu-x86_64

./buildcontainer.sh DockerfileMariner mariner-x86_64
./publishcontainer.sh $(AZCOPY_DOCKER_REG_USER) $(AZCOPY_DOCKER_REG_PWD) mariner-x86_64

displayName: "Create docker image and push to the containers registry"
workingDirectory: $(work_dir)/docker

- job: Set_2_Ubuntu_ARM64
timeoutInMinutes: 120
strategy:
vmImage: "Ubuntu-22.04"
container: "test-cnt-ubn-22"
ubuntu_tag: "ubuntu-x86_64"
mariner_tag: "mariner-x86_64"

- template: setup/build_publish_docker.yml
parameters:
jobName: Set_2_Ubuntu_Mariner_ARM64
useDemands: true
matrix:
Ubuntu-22-ARM64:
vmImage: 'Ubuntu-22.04'
container: 'test-cnt-ubn-22-arm64'
AgentName: "blobfuse-ubn22-arm64"
pool:
name: "blobfuse-ubn-arm64-pool"
demands:
- ImageOverride -equals $(AgentName)

variables:
- group: AZCOPY_SECRET_VAULT
- name: root_dir
value: '$(System.DefaultWorkingDirectory)'
- name: work_dir
value: '$(System.DefaultWorkingDirectory)/azure-storage-azcopy'

steps:
- checkout: none
- script: |
git clone https://github.com/Azure/azure-storage-azcopy
displayName: 'Checkout Code'
workingDirectory: $(root_dir)

- script: |
git checkout `echo $(Build.SourceBranch) | cut -d "/" -f 1,2 --complement`
displayName: 'Checkout branch'
workingDirectory: $(work_dir)

- task: ShellScript@2
inputs:
scriptPath: "$(work_dir)/go_installer.sh"
args: "$(root_dir)/ $(AZCOPY_GOLANG_VERSION)"
displayName: "Installing Go tools"

- script: |
sudo apt update
sudo apt --fix-broken install
displayName: "Install dependencies"

- script: |
chmod 777 *.sh
./dockerinstall.sh
./buildcontainer.sh DockerfileArm64 ubuntu-arm64
./publishcontainer.sh $(AZCOPY_DOCKER_REG_USER) $(AZCOPY_DOCKER_REG_PWD) ubuntu-arm64
displayName: "Create docker image for arm64 and push to the containers registry"
workingDirectory: $(work_dir)/docker
Ubuntu_arm64:
agentName: "blobfuse-ubn22-arm64"
vmImage: "Ubuntu-22.04"
container: "test-cnt-ubn-22-arm64"
ubuntu_tag: "ubuntu-arm64"
mariner_tag: "mariner-arm64"

- ${{ if eq(parameters.post_release, true) }}:
- stage: ReleaseToGithub
Expand Down
60 changes: 60 additions & 0 deletions setup/build_publish_docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
parameters:
- name: jobName
type: string
- name: useDemands
type: boolean
- name: matrix
type: object
- name: ubuntu_tag
type: string
- name: mariner_tag
type: string

jobs:
- job: ${{ parameters.jobName }}
timeoutInMinutes: 120
strategy:
matrix: ${{ parameters.matrix }}
pool:
${{ if eq(parameters.useDemands, true) }}:
name: "blobfuse-ubn-arm64-pool"
demands:
- ImageOverride -equals ${{ parameters.matrix['Ubuntu_arm64'].agentName }}
${{ if eq(parameters.useDemands, false) }}:
vmImage: ${{ parameters.matrix['Ubuntu_amd64'].vmImage }}

variables:
- group: AZCOPY_SECRET_VAULT
- name: root_dir
value: '$(System.DefaultWorkingDirectory)'
- name: work_dir
value: '$(System.DefaultWorkingDirectory)/azure-storage-azcopy'

steps:
- checkout: self

- ${{ if eq(parameters.jobName, 'Set_2_Ubuntu_Mariner_ARM64') }}:
- task: ShellScript@2
inputs:
scriptPath: "$(work_dir)/go_installer.sh"
args: "$(root_dir)/ $(AZCOPY_GOLANG_VERSION)"
displayName: "Installing Go tools"

- ${{ if eq(parameters.jobName, 'Set_2_Ubuntu_Mariner_ARM64') }}:
- script: |
sudo apt update
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can probably do this for all the cases, this should be safe to do even in the distros that we didn't need this for

sudo apt --fix-broken install
displayName: "Install dependencies"

- script: |
chmod 777 *.sh
./dockerinstall.sh

# Build and publish Docker containers
./buildcontainer.sh Dockerfile ${{ parameters.ubuntu_tag }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the Dockerfile names be passed in as an arg? Same with Mariner below

./publishcontainer.sh $(AZCOPY_DOCKER_REG_USER) $(AZCOPY_DOCKER_REG_PWD) ${{ parameters.ubuntu_tag }}

./buildcontainer.sh DockerfileMariner ${{ parameters.mariner_tag }}
./publishcontainer.sh $(AZCOPY_DOCKER_REG_USER) $(AZCOPY_DOCKER_REG_PWD) ${{ parameters.mariner_tag }}
displayName: "Create docker image and push to the containers registry"
workingDirectory: $(work_dir)/docker
Loading