WIP: adding Pure CBS support to AVS ESLZ

Only Terraform is in progress
Not working at this stage:

> performing CreateOrUpdate: unexpected status 500 with error: InternalBillingError:
> There was a problem communicating with the billing service.

.github/config/.wordlist.txt

@@ -16,6 +16,7 @@ azurecli
 AzureCLI
 azurepowershell
 backend
+CBS
 cd
 cli
 CLI
@@ -76,6 +77,7 @@ privatecloud
 PrivateCloud
 PrivateCloudWithHCX
 ps
+PureStorage
 Readme
 repo
 resourcegroups

BrownField/Storage/AVS-to-PureCBSdatastore-NewVNET/README.md

@@ -0,0 +1,112 @@
+# Connect Azure VMware Solution to an Pure Cloud Block Store™ datastore with new dedicated Azure Virtual Network and Express Route Gateway
+
+This tutorial walks through the scenario of connecting Azure VMware Solution Private Cloud to an Pure Cloud Block Store™ datastore with a ***new*** dedicated Azure Virtual Network and Express Route Gateway.
+
+## What will be deployed?
+
+This set of workbooks will deploy the following components:
+
+* a Virtual network
+  * 5 subnets
+* a Virtual network gateway
+  * a Public IP address
+  * an Express Route connection to AVS
+* a Managed Identity
+  * a Custom Role
+  * a Role Assignment
+* Marketplace based deployement of Pure Cloud Block Store™
+
+<!-- TODO: redo the diagram for Pure Storage  -->
+![ANF Datastores](../../../docs/images/anf-datastores.png)
+
+## Prerequisited
+
+* Steps as outlined in [Create Private Cloud](../../PrivateCloud/AVS-PrivateCloud/readme.md) or [Create Private Cloud with HCX](../../PrivateCloud/AVS-PrivateCloud-WithHCX/readme.md) section are completed.
+* Be aware of the [limit on number of authorization keys](https://docs.microsoft.com/azure/expressroute/expressroute-faqs#can-i-link-to-more-than-one-virtual-network-to-an-expressroute-circuit) that can be generated per ExpressRoute circuit: 100 for AVS.
+* **Be aware of the costs associated with [Pure Cloud Block Store™](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/purestoragemarketplaceadmin.cbs_subscription?tab=overview) and the ExpressRoute gateway using the 'Ultra Performance' or 'ErGw3Az' SKU.**
+
+## Deployment Steps
+
+* Update the parameter values in appropriate parameter file.
+* Run **one of** the following scripts.
+
+### Bicep
+
+```bash
+cd Bicep
+az deployment group create -g AVS-Step-By-Step-RG -n AVS-PCBS-datastore-Deployment -c -f "PCBSdatastoreWithNewVNet.bicep" -p "@PCBSdatastoreWithNewVNet.parameters.json"
+```
+
+### ARM
+
+```bash
+cd ARM
+
+az deployment group create -g AVS-Step-By-Step-RG -n AVS-PCBS-datastore-Deployment -c -f "PCBSdatastoreWithNewVNet.deploy.json" -p "@PCBSdatastoreWithNewVNet.parameters.json"
+```
+
+### Terraform
+
+* If deploying stand-alone, update the sample `.tfvars.sample` file in the Terraform directory with the deployment values, remove the `.sample` extension, and run the terraform workflow that fits your environment.
+
+```bash
+terraform init
+terraform plan -var-file="AVS-to-PCBSdatastore-NewVNet.tfvars"
+terraform apply -var-file="AVS-to-PCBSdatastore-NewVNet.tfvars"
+```
+
+* If deploying as a module within a larger implementation, use a module block similar to the following sample and follow your organization's Terraform workflow:
+
+<!-- TODO: Review this -->
+
+```terraform
+module "AVS-to-PCBSdatastore-NewVnet" {
+    source = "../AVS-to-PureCBSdatastore-NewVNet/Terraform/"
+    
+    DeploymentResourceGroupName = "ExampleRG"
+    PrivateCloudName = "MyPrivateCloud"
+    PrivateCloudResourceGroup = "MyPrivateCloudRG"
+    PrivateCloudSubscriptionId = "<private cloud subscription id value (not full resource id)>"
+    Location = "example region"
+
+    # vNET configuration
+    VNetName = "PureCBS-VNet"
+    VNetAddressSpaceCIDR = ["10.2.3.0/24",]
+
+    # ER gateway configuration
+    GatewaySubnetCIDR = ["10.2.3.0/27",]
+    GatewayName = "PureCBS-ERGW"
+    # GatewaySku = "UltraPerformance"
+    # GatewayBypassEnabled = true
+
+    # PureCBS network configuration
+    ManagementSubnetCIDR = ["10.2.3.32/27",]
+    ManagementSubnetName = "PureCBS-management"
+    SystemSubnetCIDR = ["10.2.3.64/27",]
+    SystemSubnetName = "PureCBS-system"
+    ReplicationSubnetCIDR = ["10.2.3.96/27"]
+    ReplicationSubnetName = "PureCBS-replication"
+    IscsiSubnetCIDR = ["10.2.3.128/27",]
+    IscsiSubnetName = "PureCBS-iscsi"
+
+    # Managed identity configuration
+    ManagedIdentityName = "PureCBS-identity"
+    PureCBSManagedIdentityRoleName = "PureCBS-role"
+
+    # PureCBS configuration
+    PcbsAlertRecipientEmail = "[email protected]"
+    PcbsArrayName           = "PureCBS-arrayName"
+    PcbsLicenseKey          = "CBS-TRIAL-LICENSE"
+    PcbsOrgDomain           = "demo.avs"
+    # PcbsUserPublicKey       = ""
+    PcbsSku                 = "V10MUR1"
+    # PcbsJitEnabled          = true
+}
+```
+## Post-deployment Steps
+
+* Navigate to "Azure Monitor", click "Networks" and select "Network health" tab. Apply filter with `Type=ER and VPN Connections`. ER Connection with Azure VMware Solution should show "Available" under "Health" column.
+
+## Next Steps
+
+[Understand External Storage solutions for AVS datastore best practices](/azure/azure-vmware/ecosystem-external-storage-solutions#external-storage-solutions-for-azure-vmware-solution-preview)

BrownField/Storage/AVS-to-PureCBSdatastore-NewVNET/Terraform/AVS-to-PureCBSdatastore-NewVNET.tfvars.sample

@@ -0,0 +1,42 @@
+# populate this file and remove the `.sample` extension when running this module standalone
+# if calling this module from another module leave this file commented out
+# Commented key/values can be customized, otherwise, a default value will be used.
+
+DeploymentResourceGroupName = "ExampleRG"
+PrivateCloudName = "MyPrivateCloud"
+PrivateCloudResourceGroup = "MyPrivateCloudRG"
+PrivateCloudSubscriptionId = "<private cloud subscription id value (not full resource id)>"
+Location = "example region"
+
+# vNET configuration
+VNetName = "PureCBS-VNet"
+VNetAddressSpaceCIDR = ["10.2.3.0/24",]
+
+# ER gateway configuration
+GatewaySubnetCIDR = ["10.2.3.0/27",]
+GatewayName = "PureCBS-ERGW"
+# GatewaySku = "UltraPerformance"
+# GatewayBypassEnabled = true
+
+# PureCBS network configuration
+ManagementSubnetCIDR = ["10.2.3.32/27",]
+ManagementSubnetName = "PureCBS-management"
+SystemSubnetCIDR = ["10.2.3.64/27",]
+SystemSubnetName = "PureCBS-system"
+ReplicationSubnetCIDR = ["10.2.3.96/27"]
+ReplicationSubnetName = "PureCBS-replication"
+IscsiSubnetCIDR = ["10.2.3.128/27",]
+IscsiSubnetName = "PureCBS-iscsi"
+
+# Managed identity configuration
+ManagedIdentityName = "PureCBS-identity"
+PureCBSManagedIdentityRoleName = "PureCBS-role"
+
+# PureCBS configuration
+PcbsAlertRecipientEmail = "[email protected]"
+PcbsArrayName           = "PureCBS-arrayName"
+PcbsLicenseKey          = "CBS-TRIAL-LICENSE"
+PcbsOrgDomain           = "demo.avs"
+# PcbsUserPublicKey       = ""
+PcbsSku                 = "V10MUR1"
+# PcbsJitEnabled          = true

BrownField/Storage/AVS-to-PureCBSdatastore-NewVNET/Terraform/main.tf

@@ -0,0 +1,220 @@
+terraform {
+  required_providers {
+    azurerm = {
+      source = "hashicorp/azurerm"
+    }
+    azapi = {
+      source = "azure/azapi"
+    }
+  }
+}
+
+provider "azurerm" {
+  features {}
+  alias                      = "AVS-to-PCBSdatastore"
+  partner_id                 = "938cd838-e22a-47da-8a6f-bdda923e3edb"
+  skip_provider_registration = "true"
+}
+
+provider "azurerm" {
+  features {}
+  alias                      = "AVS-to-PCBSdatastore-avs"
+  partner_id                 = "938cd838-e22a-47da-8a6f-bdda923e3edb"
+  skip_provider_registration = "true"
+  subscription_id            = var.PrivateCloudSubscriptionId
+}
+
+provider "azapi" {
+  skip_provider_registration = "true"
+}
+
+
+
+# Resource group creation
+resource "azurerm_resource_group" "deploymentRG" {
+  provider = azurerm.AVS-to-PCBSdatastore
+  name     = var.DeploymentResourceGroupName
+  location = var.Location
+}
+
+
+# # vNET configuration
+resource "azurerm_virtual_network" "vnetPureCBS" {
+  provider            = azurerm.AVS-to-PCBSdatastore
+  name                = var.VNetName
+  location            = azurerm_resource_group.deploymentRG.location
+  resource_group_name = azurerm_resource_group.deploymentRG.name
+  address_space       = var.VNetAddressSpaceCIDR
+}
+
+resource "azurerm_subnet" "gatewaySubnet" {
+  provider             = azurerm.AVS-to-PCBSdatastore
+  name                 = "GatewaySubnet"
+  resource_group_name  = azurerm_resource_group.deploymentRG.name
+  virtual_network_name = azurerm_virtual_network.vnetPureCBS.name
+  address_prefixes     = var.GatewaySubnetCIDR
+}
+
+# PureCBS network configuration
+resource "azurerm_subnet" "managementSubnet" {
+  provider             = azurerm.AVS-to-PCBSdatastore
+  name                 = var.ManagementSubnetName
+  resource_group_name  = azurerm_resource_group.deploymentRG.name
+  virtual_network_name = azurerm_virtual_network.vnetPureCBS.name
+  address_prefixes     = var.ManagementSubnetCIDR
+}
+
+resource "azurerm_subnet" "systemSubnet" {
+  provider             = azurerm.AVS-to-PCBSdatastore
+  name                 = var.SystemSubnetName
+  resource_group_name  = azurerm_resource_group.deploymentRG.name
+  virtual_network_name = azurerm_virtual_network.vnetPureCBS.name
+  address_prefixes     = var.SystemSubnetCIDR
+
+  service_endpoints = [
+    "Microsoft.KeyVault",
+    "Microsoft.AzureCosmosDB",
+  ]
+}
+
+resource "azurerm_subnet" "replicationSubnet" {
+  provider             = azurerm.AVS-to-PCBSdatastore
+  name                 = var.ReplicationSubnetName
+  resource_group_name  = azurerm_resource_group.deploymentRG.name
+  virtual_network_name = azurerm_virtual_network.vnetPureCBS.name
+  address_prefixes     = var.ReplicationSubnetCIDR
+}
+
+resource "azurerm_subnet" "iscsiSubnet" {
+  provider             = azurerm.AVS-to-PCBSdatastore
+  name                 = var.IscsiSubnetName
+  resource_group_name  = azurerm_resource_group.deploymentRG.name
+  virtual_network_name = azurerm_virtual_network.vnetPureCBS.name
+  address_prefixes     = var.IscsiSubnetCIDR
+}
+
+# Express Route gateway creation
+resource "azurerm_public_ip" "gatewayIP" {
+  provider            = azurerm.AVS-to-PCBSdatastore
+  name                = "${var.GatewayName}-PIP"
+  resource_group_name = azurerm_resource_group.deploymentRG.name
+  location            = azurerm_resource_group.deploymentRG.location
+  allocation_method   = "Static"
+  sku                 = "Standard"
+  sku_tier            = "Regional"
+}
+
+resource "azurerm_virtual_network_gateway" "ERGateway" {
+  provider            = azurerm.AVS-to-PCBSdatastore
+  name                = var.GatewayName
+  location            = azurerm_resource_group.deploymentRG.location
+  resource_group_name = azurerm_resource_group.deploymentRG.name
+
+  type = "ExpressRoute"
+  sku  = var.GatewaySku
+
+  ip_configuration {
+    name                          = "vnetGatewayConfig"
+    public_ip_address_id          = azurerm_public_ip.gatewayIP.id
+    private_ip_address_allocation = "Dynamic"
+    subnet_id                     = azurerm_subnet.gatewaySubnet.id
+  }
+}
+
+# Express Route authorization key
+data "azurerm_vmware_private_cloud" "sddc" {
+  provider            = azurerm.AVS-to-PCBSdatastore-avs
+  name                = var.PrivateCloudName
+  resource_group_name = var.PrivateCloudResourceGroup
+}
+
+#check this is the proper way to name the authorization
+resource "azurerm_vmware_express_route_authorization" "pcbs-to-avs" {
+  provider         = azurerm.AVS-to-PCBSdatastore-avs
+  name             = azurerm_virtual_network.vnetPureCBS.name
+  private_cloud_id = data.azurerm_vmware_private_cloud.sddc.id
+}
+
+resource "azurerm_virtual_network_gateway_connection" "pcbs-to-avs" {
+  provider            = azurerm.AVS-to-PCBSdatastore
+  name                = var.PrivateCloudName
+  location            = azurerm_resource_group.deploymentRG.location
+  resource_group_name = azurerm_resource_group.deploymentRG.name
+
+  type                         = "ExpressRoute"
+  virtual_network_gateway_id   = azurerm_virtual_network_gateway.ERGateway.id
+  express_route_circuit_id     = data.azurerm_vmware_private_cloud.sddc.circuit[0].express_route_id
+  authorization_key            = azurerm_vmware_express_route_authorization.pcbs-to-avs.express_route_authorization_key
+  routing_weight               = 0
+  express_route_gateway_bypass = var.GatewayBypassEnabled
+}
+
+# Azure VMware Solution features registration
+#  Cannot use azurerm_resource_provider_registration as it could already be registered with other set of features
+#  and we don't want to overwrite them.
+#  Using PUT on API directly will ensure the registration of required features without overwriting existing ones.
+
+# Azure VMware Solution scripting preview
+resource "azapi_resource_action" "avs-scriptingPreview" {
+  type        = "Microsoft.Features/featureProviders/subscriptionFeatureRegistrations@2021-07-01"
+  resource_id = "/subscriptions/${var.PrivateCloudSubscriptionId}/providers/Microsoft.Features/featureProviders/Microsoft.AVS/subscriptionFeatureRegistrations/scriptingPreview"
+
+  method = "PUT"
+  body = jsonencode({
+    properties = {}
+  })
+}
+
+# Azure VMware Solution iSCSI Multipath IO (MPIO)
+resource "azapi_resource_action" "avs-iSCSIMultipath" {
+  type        = "Microsoft.Features/featureProviders/subscriptionFeatureRegistrations@2021-07-01"
+  resource_id = "/subscriptions/${var.PrivateCloudSubscriptionId}/providers/Microsoft.Features/featureProviders/Microsoft.AVS/subscriptionFeatureRegistrations/iSCSIMultipath"
+
+  method = "PUT"
+  body = jsonencode({
+    properties = {}
+  })
+}
+
+# Azure VMware Solution external storage address block
+resource "azapi_resource_action" "avs-iSCSIMultipath-CIDR" {
+  type        = "Microsoft.AVS/privateClouds/iscsiPaths@2023-09-01"
+  resource_id = "${data.azurerm_vmware_private_cloud.sddc.id}/iscsiPaths/default"
+
+  method = "PUT"
+  body = jsonencode({
+    properties = {
+      "networkBlock" = "10.100.120.0/24"
+    }
+  })
+}
+
+# Create a managed identity for Cloud Block Store
+resource "azurerm_user_assigned_identity" "PCBS-Identity" {
+  provider            = azurerm.AVS-to-PCBSdatastore
+  name                = var.ManagedIdentityName
+  resource_group_name = azurerm_resource_group.deploymentRG.name
+  location            = azurerm_resource_group.deploymentRG.location
+}
+
+# Custom role definition for Cloud Block Store
+resource "azurerm_role_definition" "PCBS-Role" {
+  provider    = azurerm.AVS-to-PCBSdatastore
+  name        = var.PureCBSManagedIdentityRoleName
+  scope       = azurerm_resource_group.deploymentRG.id
+  description = "Custom Role for Cloud Block Store"
+  permissions {
+    actions = [
+      "Microsoft.Network/virtualNetworks/subnets/joinViaServiceEndpoint/action"
+    ]
+    not_actions = []
+  }
+}
+
+# Assign the custom role to the managed identity
+resource "azurerm_role_assignment" "PCBS-RoleAssignment" {
+  provider           = azurerm.AVS-to-PCBSdatastore
+  scope              = azurerm_virtual_network.vnetPureCBS.id
+  role_definition_id = azurerm_role_definition.PCBS-Role.role_definition_resource_id
+  principal_id       = azurerm_user_assigned_identity.PCBS-Identity.principal_id
+}