Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: PowerVS cluster creation fails with no fixed IP addresses available #1877

Open
Amulyam24 opened this issue Jul 10, 2024 · 1 comment · May be fixed by #2001
Open

CI: PowerVS cluster creation fails with no fixed IP addresses available #1877

Amulyam24 opened this issue Jul 10, 2024 · 1 comment · May be fixed by #2001
Assignees
@Amulyam24
Milestone
Next

Comments

@Amulyam24
Copy link
Contributor

In CI, we create 2 clusters as a part of the test suite. After the first cluster is created successfully, during the the second cluster deployment, first machine creation fails with

"message":"Build of instance 70daa327-9052-4168-90ea-27677aac45bc aborted: Failed to allocate the network(s) with error No fixed IP addresses available for network: 8b7e4d36-7142-4216-83d1-4a4849c229c0, not rescheduling."

This is due to improper cleanup of VMs created as a part of the first cluster deployment.

{
  "pvmInstances": [
    {
      "addresses": [],
      "creationDate": "2024-07-08T21:46:35.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.427994",
        "reason": "PENDING",
        "status": "PENDING"
      },
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/70daa327-9052-4168-90ea-27677aac45bc",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "memory": 0,
      "networks": [],
      "osType": "",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "",
      "processors": 0,
      "pvmInstanceID": "70daa327-9052-4168-90ea-27677aac45bc",
      "serverName": "capibm-e2e-ri4t2i-control-plane-4dl6k",
      "srcs": [],
      "status": "BUILD",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "spawning",
      "updatedDate": "2024-07-08T21:46:35.000Z",
      "virtualCores": {
        "assigned": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89/networks/capi-net-CzwSz",
          "ip": "192.168.157.165",
          "ipAddress": "192.168.157.165",
          "macAddress": "fa:7b:e2:cc:61:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:36:00.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428057",
        "reason": "rmc_state - inactive",
        "status": "WARNING"
      },
      "hostID": 4,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/5fbf4b71-caad-4e49-9541-bff8b6d6af89/networks/capi-net-CzwSz",
          "ip": "192.168.157.165",
          "ipAddress": "192.168.157.165",
          "macAddress": "fa:7b:e2:cc:61:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Unknown",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "5fbf4b71-caad-4e49-9541-bff8b6d6af89",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-sbcrc",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:36:00.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60/networks/capi-net-CzwSz",
          "ip": "192.168.157.163",
          "ipAddress": "192.168.157.163",
          "macAddress": "fa:63:9b:99:6b:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:35:48.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428106",
        "reason": "rmc_state - inactive",
        "status": "WARNING"
      },
      "hostID": 29,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/936e6efa-05d4-47dd-ba6c-fca24240bc60/networks/capi-net-CzwSz",
          "ip": "192.168.157.163",
          "ipAddress": "192.168.157.163",
          "macAddress": "fa:63:9b:99:6b:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "936e6efa-05d4-47dd-ba6c-fca24240bc60",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-4mf49",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:35:48.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d/networks/capi-net-CzwSz",
          "ip": "192.168.157.162",
          "ipAddress": "192.168.157.162",
          "macAddress": "fa:19:94:45:15:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:25:49.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428142",
        "status": "OK"
      },
      "hostID": 4,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/4d7910d6-abe7-4a17-ad30-ea27b46e987d/networks/capi-net-CzwSz",
          "ip": "192.168.157.162",
          "ipAddress": "192.168.157.162",
          "macAddress": "fa:19:94:45:15:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "4d7910d6-abe7-4a17-ad30-ea27b46e987d",
      "serverName": "capibm-e2e-jb3h95-md-0-b7kp5-crx6p",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:25:49.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    },
    {
      "addresses": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f/networks/capi-net-CzwSz",
          "ip": "192.168.157.164",
          "ipAddress": "192.168.157.164",
          "macAddress": "fa:fa:b2:69:da:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "creationDate": "2024-07-08T21:16:45.000Z",
      "diskSize": 20,
      "health": {
        "lastUpdate": "2024-07-08T21:47:23.428172",
        "status": "OK"
      },
      "hostID": 29,
      "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f",
      "imageID": "758854ce-9b0a-42f8-9471-3aa9870795e3",
      "maxmem": 32,
      "maxproc": 2,
      "memory": 4,
      "minmem": 2,
      "minproc": 0.25,
      "networks": [
        {
          "href": "/pcloud/v1/cloud-instances/e7207381-53be-4a6a-b2c1-127580e97466/pvm-instances/316dc448-77ac-4bdb-8342-7605d95b9c9f/networks/capi-net-CzwSz",
          "ip": "192.168.157.164",
          "ipAddress": "192.168.157.164",
          "macAddress": "fa:fa:b2:69:da:20",
          "networkName": "capi-net-CzwSz",
          "type": "fixed",
          "version": 4
        }
      ],
      "operatingSystem": "Linux/CentOS Stream 4.18.0-552.1.1.el8.ppc64l8, 8",
      "osType": "rhel",
      "pinPolicy": "none",
      "placementGroup": "none",
      "procType": "shared",
      "processors": 0.25,
      "pvmInstanceID": "316dc448-77ac-4bdb-8342-7605d95b9c9f",
      "serverName": "capibm-e2e-jb3h95-control-plane-4w5sw",
      "srcs": [],
      "status": "ACTIVE",
      "storagePool": "General-Flash-72",
      "storagePoolAffinity": true,
      "storageType": "tier3",
      "sysType": "s922",
      "taskState": "deleting",
      "updatedDate": "2024-07-08T21:16:45.000Z",
      "virtualCores": {
        "assigned": 1,
        "max": 5,
        "min": 1
      }
    }
  ]
}

Job link - https://prow.ppc64le-cloud.cis.ibm.net/view/gs/ppc64le-kubernetes/logs/periodic-capi-provider-ibmcloud-e2e-powervs/1810418740110561280
@mkumatag mkumatag added this to the Next milestone Jul 23, 2024
@mkumatag mkumatag modified the milestones: Next, v0.9.0 Sep 17, 2024
@Amulyam24 Amulyam24 linked a pull request Oct 9, 2024 that will close this issue
Open
@mkumatag
Copy link
Member

mkumatag commented Nov 14, 2024 •
edited
Loading

is this happening because vm deleting is taking some time and not waiting for vm to be disappear?

@mkumatag mkumatag modified the milestones: v0.9.0, Next Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Amulyam24 Amulyam24

Labels
None yet
Projects
None yet
Milestone
Next
2 participants
@mkumatag @Amulyam24