Adding support for PMD auto load balancing

Port rx queues that have not been statically assigned to PMDs are currently
assigned based on periodically sampled load measurements.
The assignment is performed at specific instances – port addition, port
deletion, upon reassignment request via CLI etc.

Due to change in traffic pattern over time it can cause uneven load among
the PMDs and thus resulting in lower overall throughout.

This patch enables the support of auto load balancing of PMDs based on
measured load of RX queues. Each PMD measures the processing load for each
of its associated queues every 10 seconds. If the aggregated PMD load reaches
95% for 6 consecutive intervals then PMD considers itself to be overloaded.

If any PMD is overloaded, a dry-run of the PMD assignment algorithm is
performed by OVS main thread. The dry-run does NOT change the existing
queue to PMD assignments.

If the resultant mapping of dry-run indicates an improved distribution
of the load then the actual reassignment will be performed.

The automatic rebalancing will be disabled by default and has to be
enabled via configuration option. The interval (in minutes) between
two consecutive rebalancing can also be configured via CLI, default
is 1 min.

Following example commands can be used to set the auto-lb params:
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb="true"
ovs-vsctl set open_vswitch . other_config:pmd-auto-lb-rebalance-intvl="5"

Co-authored-by: Rohith Basavaraja <[email protected]>
Co-authored-by: Venkatesan Pradeep <[email protected]>
Signed-off-by: Rohith Basavaraja <[email protected]>
Signed-off-by: Venkatesan Pradeep <[email protected]>
Signed-off-by: Nitin Katiyar <[email protected]>
Acked-by: Kevin Traynor <[email protected]>
Tested-by: Kevin Traynor <[email protected]>
Signed-off-by: Ian Stokes <[email protected]>

Documentation/topics/dpdk/pmd.rst

@@ -183,3 +183,66 @@ or can be triggered by using::
    In addition, the output of ``pmd-rxq-show`` was modified to include
    Rx queue utilization of the PMD as a percentage. Prior to this, tracking of
    stats was not available.
+
+Automatic assignment of Port/Rx Queue to PMD Threads (experimental)
+-------------------------------------------------------------------
+
+Cycle or utilization based allocation of Rx queues to PMDs gives efficient
+load distribution but it is not adaptive to change in traffic pattern
+occurring over the time. This causes uneven load among the PMDs which results
+in overall lower throughput.
+
+To address this automatic load balancing of PMDs can be set by::
+
+    $ ovs-vsctl set open_vswitch . other_config:pmd-auto-lb="true"
+
+If pmd-auto-lb is set to true AND cycle based assignment is enabled then auto
+load balancing of PMDs is enabled provided there are 2 or more non-isolated
+PMDs and at least one of these PMDs is polling more than one RX queue. So,
+following conditions need to be met to have Auto Load balancing enabled:
+
+1. cycle based assignment of RX queues to PMD is enabled.
+2. pmd-auto-lb is set to true.
+3. There are two or more non-isolated PMDs present.
+4. And at least one of the non-isolated PMD has more than one queue polling.
+
+If any of above is not met PMD Auto Load Balancing is disabled.
+
+Once auto load balancing is set, each non-isolated PMD measures the processing
+load for each of its associated queues every 10 seconds. If the aggregated PMD
+load reaches 95% for 6 consecutive intervals then PMD considers itself to be
+overloaded.
+
+If any PMD is overloaded, a dry-run of the PMD assignment algorithm is
+performed by OVS main thread. The dry-run does NOT change the existing queue
+to PMD assignments.
+
+If the resultant mapping of dry-run indicates an improved distribution of the
+load then the actual reassignment will be performed.
+
+.. note::
+
+    PMD Auto Load Balancing doesn't currently work if queues are assigned
+    cross NUMA as actual processing load could get worse after assignment
+    as compared to what dry run predicts.
+
+The minimum time between 2 consecutive PMD auto load balancing iterations can
+also be configured by::
+
+    $ ovs-vsctl set open_vswitch .\
+        other_config:pmd-auto-lb-rebal-interval="<interval>"
+
+where ``<interval>`` is a value in minutes. The default interval is 1 minute
+and setting it to 0 will also result in default value i.e. 1 min.
+
+A user can use this option to avoid frequent trigger of Auto Load Balancing of
+PMDs. For e.g. set this (in min) such that it occurs once in few hours or a day
+or a week.
+
+.. note::
+    In some scenarios it may not be desired to have Auto Load Balancing
+    triggerred. For example, if traffic profile for specific RX queue is
+    changing dramatically very frequently which in turn thrashes CPU cache
+    due to changes required in dpctl flows and EMC for newly added flows.
+    In such scenarios user should configure rebalance interval accordingly
+    to avoid frequent rebalancing happening.

NEWS

@@ -27,6 +27,7 @@ Post-v2.10.0
      * Add option for simple round-robin based Rxq to PMD assignment.
        It can be set with pmd-rxq-assign.
      * Add support for DPDK 18.11
+     * Add support for Auto load balancing of PMDs (experimental)
    - Add 'symmetric_l3' hash function.
    - OVS now honors 'updelay' and 'downdelay' for bonds with LACP configured.
    - ovs-vswitchd: