Occasionally missing Kubernetes Events

[gemblerz]

It happens rarely but when Kubernetes events are missed (i.e., not received in the scheduler) the particular plugin can get stuck.

For example, the car-sampler plugin was initialized and the scheduler did not get any following events from Kubernetes. Even after the plugin was completed in Kubernetes, the scheduler still thought it was being initialized.

INFO: 2024/10/23 23:06:21 nodescheduler.go:311: Plugin "car-sampler-26" is scheduled
INFO: 2024/10/23 23:06:21 nodescheduler.go:248: Plugin "avian-diversity-monitoring-2198" is created
INFO: 2024/10/23 23:06:21 nodescheduler.go:248: Plugin "car-sampler-26" is created
INFO: 2024/10/23 23:06:21 nodescheduler.go:311: Plugin "avian-diversity-monitoring-2198" is scheduled
INFO: 2024/10/23 23:06:21 nodescheduler.go:342: plugin "car-sampler-26" is being initialized
INFO: 2024/10/23 23:06:21 nodescheduler.go:342: plugin "avian-diversity-monitoring-2198" is being initialized

I removed the plugin Pod manually and the scheduler received a Pod removal event, causing an iligal transition from initialized to completed,

ERROR: 2024/10/24 15:35:31 nodescheduler.go:422: plugin "car-sampler" failed to transition from initializing to completed: event completed inappropriate in current state initializing
INFO: 2024/10/24 15:35:31 nodescheduler.go:480: Plugin "car-sampler-26" removed
INFO: 2024/10/24 15:35:31 nodescheduler.go:188: Reason for (re)scheduling "sys.scheduler.status.plugin.failed"

We need a better mechanism to monitor states of plugins, in case Kubernetes events are missed. How?