netdev-afxdp: Best-effort configuration of XDP mode.

Until now there was only two options for XDP mode in OVS: SKB or DRV.
i.e. 'generic XDP' or 'native XDP with zero-copy enabled'.

Devices like 'veth' interfaces in Linux supports native XDP, but
doesn't support zero-copy mode.  This case can not be covered by
existing API and we have to use slower generic XDP for such devices.
There are few more issues, e.g. TCP is not supported in generic XDP
mode for veth interfaces due to kernel limitations, however it is
supported in native mode.

This change introduces ability to use native XDP without zero-copy
along with best-effort configuration option that enabled by default.
In best-effort case OVS will sequentially try different modes starting
from the fastest one and will choose the first acceptable for current
interface.  This will guarantee the best possible performance.

If user will want to choose specific mode, it's still possible by
setting the 'options:xdp-mode'.

This change additionally changes the API by renaming the configuration
knob from 'xdpmode' to 'xdp-mode' and also renaming the modes
themselves to be more user-friendly.

The full list of currently supported modes:
  * native-with-zerocopy - former DRV
  * native               - new one, DRV without zero-copy
  * generic              - former SKB
  * best-effort          - new one, chooses the best available from
                           3 above modes

Since 'best-effort' is a default mode, users will not need to
explicitely set 'xdp-mode' in most cases.

TCP related tests enabled back in system afxdp testsuite, because
'best-effort' will choose 'native' mode for veth interfaces
and this mode has no issues with TCP.

Signed-off-by: Ilya Maximets <[email protected]>
Acked-by: William Tu <[email protected]>
Acked-by: Eelco Chaudron <[email protected]>

Documentation/intro/install/afxdp.rst

@@ -153,9 +153,8 @@ To kick start end-to-end autotesting::
   make check-afxdp TESTSUITEFLAGS='1'
 
 .. note::
-   Not all test cases pass at this time. Currenly all TCP related
-   tests, ex: using wget or http, are skipped due to XDP limitations
-   on veth. cvlan test is also skipped.
+   Not all test cases pass at this time. Currenly all cvlan tests are skipped
+   due to kernel issues.
 
 If a test case fails, check the log at::
 
@@ -177,33 +176,35 @@ in :doc:`general`::
   ovs-vsctl -- add-br br0 -- set Bridge br0 datapath_type=netdev
 
 Make sure your device driver support AF_XDP, netdev-afxdp supports
-the following additional options (see man ovs-vswitchd.conf.db for
+the following additional options (see ``man ovs-vswitchd.conf.db`` for
 more details):
 
- * **xdpmode**: use "drv" for driver mode, or "skb" for skb mode.
+ * ``xdp-mode``: ``best-effort``, ``native-with-zerocopy``,
+   ``native`` or ``generic``.  Defaults to ``best-effort``, i.e. best of
+   supported modes, so in most cases you don't need to change it.
 
- * **use-need-wakeup**: default "true" if libbpf supports it, otherwise false.
+ * ``use-need-wakeup``: default ``true`` if libbpf supports it,
+   otherwise ``false``.
 
 For example, to use 1 PMD (on core 4) on 1 queue (queue 0) device,
-configure these options: **pmd-cpu-mask, pmd-rxq-affinity, and n_rxq**.
-The **xdpmode** can be "drv" or "skb"::
+configure these options: ``pmd-cpu-mask``, ``pmd-rxq-affinity``, and
+``n_rxq``::
 
   ethtool -L enp2s0 combined 1
   ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x10
   ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0 type="afxdp" \
-    options:n_rxq=1 options:xdpmode=drv \
-    other_config:pmd-rxq-affinity="0:4"
+                                   other_config:pmd-rxq-affinity="0:4"
 
 Or, use 4 pmds/cores and 4 queues by doing::
 
   ethtool -L enp2s0 combined 4
   ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x36
   ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0 type="afxdp" \
-    options:n_rxq=4 options:xdpmode=drv \
-    other_config:pmd-rxq-affinity="0:1,1:2,2:3,3:4"
+    options:n_rxq=4 other_config:pmd-rxq-affinity="0:1,1:2,2:3,3:4"
 
 .. note::
-   pmd-rxq-affinity is optional. If not specified, system will auto-assign.
+   ``pmd-rxq-affinity`` is optional. If not specified, system will auto-assign.
+   ``n_rxq`` equals ``1`` by default.
 
 To validate that the bridge has successfully instantiated, you can use the::
 
@@ -214,12 +215,21 @@ Should show something like::
   Port "ens802f0"
    Interface "ens802f0"
       type: afxdp
-      options: {n_rxq="1", xdpmode=drv}
+      options: {n_rxq="1"}
 
 Otherwise, enable debugging by::
 
   ovs-appctl vlog/set netdev_afxdp::dbg
 
+To check which XDP mode was chosen by ``best-effort``, you can look for
+``xdp-mode-in-use`` in the output of ``ovs-appctl dpctl/show``::
+
+  # ovs-appctl dpctl/show
+  netdev@ovs-netdev:
+    <...>
+    port 2: ens802f0 (afxdp: n_rxq=1, use-need-wakeup=true,
+                      xdp-mode=best-effort,
+                      xdp-mode-in-use=native-with-zerocopy)
 
 References
 ----------
@@ -323,8 +333,13 @@ Limitations/Known Issues
 #. Most of the tests are done using i40e single port. Multiple ports and
    also ixgbe driver also needs to be tested.
 #. No latency test result (TODO items)
-#. Due to limitations of current upstream kernel, TCP and various offloading
+#. Due to limitations of current upstream kernel, various offloading
    (vlan, cvlan) is not working over virtual interfaces (i.e. veth pair).
+   Also, TCP is not working over virtual interfaces (veth) in generic XDP mode.
+   Some more information and possible workaround available `here
+   <https://github.com/cilium/cilium/issues/3077#issuecomment-430801467>`__ .
+   For TAP interfaces generic mode seems to work fine (TCP works) and even
+   could provide better performance than native mode in some cases.
 
 
 PVP using tap device
@@ -335,8 +350,7 @@ First, start OVS, then add physical port::
   ethtool -L enp2s0 combined 1
   ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x10
   ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0 type="afxdp" \
-    options:n_rxq=1 options:xdpmode=drv \
-    other_config:pmd-rxq-affinity="0:4"
+    options:n_rxq=1 other_config:pmd-rxq-affinity="0:4"
 
 Start a VM with virtio and tap device::
 
@@ -414,13 +428,11 @@ Create namespace and veth peer devices::
 
 Attach the veth port to br0 (linux kernel mode)::
 
-  ovs-vsctl add-port br0 afxdp-p0 -- \
-    set interface afxdp-p0 options:n_rxq=1
+  ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0
 
-Or, use AF_XDP with skb mode::
+Or, use AF_XDP::
 
-  ovs-vsctl add-port br0 afxdp-p0 -- \
-    set interface afxdp-p0 type="afxdp" options:n_rxq=1 options:xdpmode=skb
+  ovs-vsctl add-port br0 afxdp-p0 -- set interface afxdp-p0 type="afxdp"
 
 Setup the OpenFlow rules::
 

NEWS

@@ -5,11 +5,19 @@ Post-v2.12.0
        separate project. You can find it at
        https://github.com/ovn-org/ovn.git
    - Userspace datapath:
+     * Add option to enable, disable and query TCP sequence checking in
+       conntrack.
+   - AF_XDP:
      * New option 'use-need-wakeup' for netdev-afxdp to control enabling
        of corresponding 'need_wakeup' flag in AF_XDP rings.  Enabled by default
        if supported by libbpf.
-     * Add option to enable, disable and query TCP sequence checking in
-       conntrack.
+     * 'xdpmode' option for netdev-afxdp renamed to 'xdp-mode'.
+       Modes also updated.  New values:
+         native-with-zerocopy  - former DRV
+         native                - new one, DRV without zero-copy
+         generic               - former SKB
+         best-effort [default] - new one, chooses the best available from
+                                 3 above modes
    - DPDK:
      * DPDK pdump packet capture support disabled by default. New configure
        option '--enable-dpdk-pdump' to enable it.