ovn-architecture.7.xml

<?xml version="1.0" encoding="utf-8"?>
<manpage program="ovn-architecture" section="7" title="OVN Architecture">
  <h1>Name</h1>
  <p>ovn-architecture -- Open Virtual Network architecture</p>

  <h1>Description</h1>

  <p>
    OVN, the Open Virtual Network, is a system to support virtual network
    abstraction.  OVN complements the existing capabilities of OVS to add
    native support for virtual network abstractions, such as virtual L2 and L3
    overlays and security groups.  Services such as DHCP are also desirable
    features.  Just like OVS, OVN's design goal is to have a production-quality
    implementation that can operate at significant scale.
  </p>

  <p>
    An OVN deployment consists of several components:
  </p>

  <ul>
    <li>
      <p>
        A <dfn>Cloud Management System</dfn> (<dfn>CMS</dfn>), which is
        OVN's ultimate client (via its users and administrators).  OVN
        integration requires installing a CMS-specific plugin and
        related software (see below).  OVN initially targets OpenStack
        as CMS.
      </p>

      <p>
        We generally speak of ``the'' CMS, but one can imagine scenarios in
        which multiple CMSes manage different parts of an OVN deployment.
      </p>
    </li>

    <li>
      An OVN Database physical or virtual node (or, eventually, cluster)
      installed in a central location.
    </li>

    <li>
      One or more (usually many) <dfn>hypervisors</dfn>.  Hypervisors must run
      Open vSwitch and implement the interface described in
      <code>IntegrationGuide.rst</code> in the OVS source tree.  Any hypervisor
      platform supported by Open vSwitch is acceptable.
    </li>

    <li>
      <p>
        Zero or more <dfn>gateways</dfn>.  A gateway extends a tunnel-based
        logical network into a physical network by bidirectionally forwarding
        packets between tunnels and a physical Ethernet port.  This allows
        non-virtualized machines to participate in logical networks.  A gateway
        may be a physical host, a virtual machine, or an ASIC-based hardware
        switch that supports the <code>vtep</code>(5) schema.  (Support for the
        latter will come later in OVN implementation.)
      </p>

      <p>
        Hypervisors and gateways are together called <dfn>transport node</dfn>
        or <dfn>chassis</dfn>.
      </p>
    </li>
  </ul>

  <p>
    The diagram below shows how the major components of OVN and related
    software interact.  Starting at the top of the diagram, we have:
  </p>

  <ul>
    <li>
      The Cloud Management System, as defined above.
    </li>

    <li>
      <p>
        The <dfn>OVN/CMS Plugin</dfn> is the component of the CMS that
        interfaces to OVN.  In OpenStack, this is a Neutron plugin.
        The plugin's main purpose is to translate the CMS's notion of logical
        network configuration, stored in the CMS's configuration database in a
        CMS-specific format, into an intermediate representation understood by
        OVN.
      </p>

      <p>
        This component is necessarily CMS-specific, so a new plugin needs to be
        developed for each CMS that is integrated with OVN.  All of the
        components below this one in the diagram are CMS-independent.
      </p>
    </li>

    <li>
      <p>
        The <dfn>OVN Northbound Database</dfn> receives the intermediate
        representation of logical network configuration passed down by the
        OVN/CMS Plugin.  The database schema is meant to be ``impedance
        matched'' with the concepts used in a CMS, so that it directly supports
        notions of logical switches, routers, ACLs, and so on.  See
        <code>ovn-nb</code>(5) for details.
      </p>

      <p>
        The OVN Northbound Database has only two clients: the OVN/CMS Plugin
        above it and <code>ovn-northd</code> below it.
      </p>
    </li>

    <li>
      <code>ovn-northd</code>(8) connects to the OVN Northbound Database
      above it and the OVN Southbound Database below it.  It translates the
      logical network configuration in terms of conventional network
      concepts, taken from the OVN Northbound Database, into logical
      datapath flows in the OVN Southbound Database below it.
    </li>

    <li>
      <p>
    The <dfn>OVN Southbound Database</dfn> is the center of the system.
    Its clients are <code>ovn-northd</code>(8) above it and
    <code>ovn-controller</code>(8) on every transport node below it.
      </p>

      <p>
        The OVN Southbound Database contains three kinds of data: <dfn>Physical
        Network</dfn> (PN) tables that specify how to reach hypervisor and
        other nodes, <dfn>Logical Network</dfn> (LN) tables that describe the
        logical network in terms of ``logical datapath flows,'' and
        <dfn>Binding</dfn> tables that link logical network components'
        locations to the physical network.  The hypervisors populate the PN and
        Port_Binding tables, whereas <code>ovn-northd</code>(8) populates the
        LN tables.
      </p>

      <p>
    OVN Southbound Database performance must scale with the number of
    transport nodes.  This will likely require some work on
    <code>ovsdb-server</code>(1) as we encounter bottlenecks.
    Clustering for availability may be needed.
      </p>
    </li>
  </ul>

  <p>
    The remaining components are replicated onto each hypervisor:
  </p>

  <ul>
    <li>
      <code>ovn-controller</code>(8) is OVN's agent on each hypervisor and
      software gateway.  Northbound, it connects to the OVN Southbound
      Database to learn about OVN configuration and status and to
      populate the PN table and the <code>Chassis</code> column in
      <code>Binding</code> table with the hypervisor's status.
      Southbound, it connects to <code>ovs-vswitchd</code>(8) as an
      OpenFlow controller, for control over network traffic, and to the
      local <code>ovsdb-server</code>(1) to allow it to monitor and
      control Open vSwitch configuration.
    </li>

    <li>
      <code>ovs-vswitchd</code>(8) and <code>ovsdb-server</code>(1) are
      conventional components of Open vSwitch.
    </li>
  </ul>

  <pre fixed="yes">
                                  CMS
                                   |
                                   |
                       +-----------|-----------+
                       |           |           |
                       |     OVN/CMS Plugin    |
                       |           |           |
                       |           |           |
                       |   OVN Northbound DB   |
                       |           |           |
                       |           |           |
                       |       ovn-northd      |
                       |           |           |
                       +-----------|-----------+
                                   |
                                   |
                         +-------------------+
                         | OVN Southbound DB |
                         +-------------------+
                                   |
                                   |
                +------------------+------------------+
                |                  |                  |
  HV 1          |                  |    HV n          |
+---------------|---------------+  .  +---------------|---------------+
|               |               |  .  |               |               |
|        ovn-controller         |  .  |        ovn-controller         |
|         |          |          |  .  |         |          |          |
|         |          |          |     |         |          |          |
|  ovs-vswitchd   ovsdb-server  |     |  ovs-vswitchd   ovsdb-server  |
|                               |     |                               |
+-------------------------------+     +-------------------------------+
  </pre>

  <h2>Information Flow in OVN</h2>

  <p>
    Configuration data in OVN flows from north to south.  The CMS, through its
    OVN/CMS plugin, passes the logical network configuration to
    <code>ovn-northd</code> via the northbound database.  In turn,
    <code>ovn-northd</code> compiles the configuration into a lower-level form
    and passes it to all of the chassis via the southbound database.
  </p>

  <p>
    Status information in OVN flows from south to north.  OVN currently
    provides only a few forms of status information.  First,
    <code>ovn-northd</code> populates the <code>up</code> column in the
    northbound <code>Logical_Switch_Port</code> table: if a logical port's
    <code>chassis</code> column in the southbound <code>Port_Binding</code>
    table is nonempty, it sets <code>up</code> to <code>true</code>, otherwise
    to <code>false</code>.  This allows the CMS to detect when a VM's
    networking has come up.
  </p>

  <p>
    Second, OVN provides feedback to the CMS on the realization of its
    configuration, that is, whether the configuration provided by the CMS has
    taken effect.  This feature requires the CMS to participate in a sequence
    number protocol, which works the following way:
  </p>

  <ol>
    <li>
      When the CMS updates the configuration in the northbound database, as
      part of the same transaction, it increments the value of the
      <code>nb_cfg</code> column in the <code>NB_Global</code> table.  (This is
      only necessary if the CMS wants to know when the configuration has been
      realized.)
    </li>

    <li>
      When <code>ovn-northd</code> updates the southbound database based on a
      given snapshot of the northbound database, it copies <code>nb_cfg</code>
      from northbound <code>NB_Global</code> into the southbound database
      <code>SB_Global</code> table, as part of the same transaction.  (Thus, an
      observer monitoring both databases can determine when the southbound
      database is caught up with the northbound.)
    </li>

    <li>
      After <code>ovn-northd</code> receives confirmation from the southbound
      database server that its changes have committed, it updates
      <code>sb_cfg</code> in the northbound <code>NB_Global</code> table to the
      <code>nb_cfg</code> version that was pushed down.  (Thus, the CMS or
      another observer can determine when the southbound database is caught up
      without a connection to the southbound database.)
    </li>

    <li>
      The <code>ovn-controller</code> process on each chassis receives the
      updated southbound database, with the updated <code>nb_cfg</code>.  This
      process in turn updates the physical flows installed in the chassis's
      Open vSwitch instances.  When it receives confirmation from Open vSwitch
      that the physical flows have been updated, it updates <code>nb_cfg</code>
      in its own <code>Chassis</code> record in the southbound database.
    </li>

    <li>
      <code>ovn-northd</code> monitors the <code>nb_cfg</code> column in all of
      the <code>Chassis</code> records in the southbound database.  It keeps
      track of the minimum value among all the records and copies it into the
      <code>hv_cfg</code> column in the northbound <code>NB_Global</code>
      table.  (Thus, the CMS or another observer can determine when all of the
      hypervisors have caught up to the northbound configuration.)
    </li>
  </ol>

  <h2>Chassis Setup</h2>

  <p>
    Each chassis in an OVN deployment must be configured with an Open vSwitch
    bridge dedicated for OVN's use, called the <dfn>integration bridge</dfn>.
    System startup scripts may create this bridge prior to starting
    <code>ovn-controller</code> if desired.  If this bridge does not exist when
    ovn-controller starts, it will be created automatically with the default
    configuration suggested below.  The ports on the integration bridge include:
  </p>

  <ul>
    <li>
      On any chassis, tunnel ports that OVN uses to maintain logical network
      connectivity.  <code>ovn-controller</code> adds, updates, and removes
      these tunnel ports.
    </li>

    <li>
      On a hypervisor, any VIFs that are to be attached to logical networks.
      The hypervisor itself, or the integration between Open vSwitch and the
      hypervisor (described in <code>IntegrationGuide.rst</code>) takes care of
      this.  (This is not part of OVN or new to OVN; this is pre-existing
      integration work that has already been done on hypervisors that support
      OVS.)
    </li>

    <li>
      On a gateway, the physical port used for logical network connectivity.
      System startup scripts add this port to the bridge prior to starting
      <code>ovn-controller</code>.  This can be a patch port to another bridge,
      instead of a physical port, in more sophisticated setups.
    </li>
  </ul>

  <p>
    Other ports should not be attached to the integration bridge.  In
    particular, physical ports attached to the underlay network (as opposed to
    gateway ports, which are physical ports attached to logical networks) must
    not be attached to the integration bridge.  Underlay physical ports should
    instead be attached to a separate Open vSwitch bridge (they need not be
    attached to any bridge at all, in fact).
  </p>

  <p>
    The integration bridge should be configured as described below.
    The effect of each of these settings is documented in
    <code>ovs-vswitchd.conf.db</code>(5):
  </p>

  <!-- Keep the following in sync with create_br_int() in
       ovn/controller/ovn-controller.c. -->
  <dl>
    <dt><code>fail-mode=secure</code></dt>
    <dd>
      Avoids switching packets between isolated logical networks before
      <code>ovn-controller</code> starts up.  See <code>Controller Failure
      Settings</code> in <code>ovs-vsctl</code>(8) for more information.
    </dd>

    <dt><code>other-config:disable-in-band=true</code></dt>
    <dd>
      Suppresses in-band control flows for the integration bridge.  It would be
      unusual for such flows to show up anyway, because OVN uses a local
      controller (over a Unix domain socket) instead of a remote controller.
      It's possible, however, for some other bridge in the same system to have
      an in-band remote controller, and in that case this suppresses the flows
      that in-band control would ordinarily set up.  See <code>In-Band
      Control</code> in <code>DESIGN.rst</code> for more information.
    </dd>
  </dl>

  <p>
    The customary name for the integration bridge is <code>br-int</code>, but
    another name may be used.
  </p>

  <h2>Logical Networks</h2>

  <p>
    A <dfn>logical network</dfn> implements the same concepts as physical
    networks, but they are insulated from the physical network with tunnels or
    other encapsulations.  This allows logical networks to have separate IP and
    other address spaces that overlap, without conflicting, with those used for
    physical networks.  Logical network topologies can be arranged without
    regard for the topologies of the physical networks on which they run.
  </p>

  <p>
    Logical network concepts in OVN include:
  </p>

  <ul>
    <li>
      <dfn>Logical switches</dfn>, the logical version of Ethernet switches.
    </li>

    <li>
      <dfn>Logical routers</dfn>, the logical version of IP routers.  Logical
      switches and routers can be connected into sophisticated topologies.
    </li>

    <li>
      <dfn>Logical datapaths</dfn> are the logical version of an OpenFlow
      switch.  Logical switches and routers are both implemented as logical
      datapaths.
    </li>
  </ul>

  <h2>Life Cycle of a VIF</h2>

  <p>
    Tables and their schemas presented in isolation are difficult to
    understand.  Here's an example.
  </p>

  <p>
    A VIF on a hypervisor is a virtual network interface attached either
    to a VM or a container running directly on that hypervisor (This is
    different from the interface of a container running inside a VM).
  </p>

  <p>
    The steps in this example refer often to details of the OVN and OVN
    Northbound database schemas.  Please see <code>ovn-sb</code>(5) and
    <code>ovn-nb</code>(5), respectively, for the full story on these
    databases.
  </p>

  <ol>
    <li>
      A VIF's life cycle begins when a CMS administrator creates a new VIF
      using the CMS user interface or API and adds it to a switch (one
      implemented by OVN as a logical switch).  The CMS updates its own
      configuration.  This includes associating unique, persistent identifier
      <var>vif-id</var> and Ethernet address <var>mac</var> with the VIF.
    </li>

    <li>
      The CMS plugin updates the OVN Northbound database to include the new
      VIF, by adding a row to the <code>Logical_Switch_Port</code>
      table.  In the new row, <code>name</code> is <var>vif-id</var>,
      <code>mac</code> is <var>mac</var>, <code>switch</code> points to
      the OVN logical switch's Logical_Switch record, and other columns
      are initialized appropriately.
    </li>

    <li>
      <code>ovn-northd</code> receives the OVN Northbound database update.  In
      turn, it makes the corresponding updates to the OVN Southbound database,
      by adding rows to the OVN Southbound database <code>Logical_Flow</code>
      table to reflect the new port, e.g. add a flow to recognize that packets
      destined to the new port's MAC address should be delivered to it, and
      update the flow that delivers broadcast and multicast packets to include
      the new port.  It also creates a record in the <code>Binding</code> table
      and populates all its columns except the column that identifies the
      <code>chassis</code>.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> receives the
      <code>Logical_Flow</code> table updates that <code>ovn-northd</code> made
      in the previous step.  As long as the VM that owns the VIF is powered
      off, <code>ovn-controller</code> cannot do much; it cannot, for example,
      arrange to send packets to or receive packets from the VIF, because the
      VIF does not actually exist anywhere.
    </li>

    <li>
      Eventually, a user powers on the VM that owns the VIF.  On the hypervisor
      where the VM is powered on, the integration between the hypervisor and
      Open vSwitch (described in <code>IntegrationGuide.rst</code>) adds the VIF
      to the OVN integration bridge and stores <var>vif-id</var> in
      <code>external_ids</code>:<code>iface-id</code> to indicate that the
      interface is an instantiation of the new VIF.  (None of this code is new
      in OVN; this is pre-existing integration work that has already been done
      on hypervisors that support OVS.)
    </li>

    <li>
      On the hypervisor where the VM is powered on, <code>ovn-controller</code>
      notices <code>external_ids</code>:<code>iface-id</code> in the new
      Interface. In response, in the OVN Southbound DB, it updates the
      <code>Binding</code> table's <code>chassis</code> column for the
      row that links the logical port from <code>external_ids</code>:<code>
      iface-id</code> to the hypervisor. Afterward, <code>ovn-controller</code>
      updates the local hypervisor's OpenFlow tables so that packets to and from
      the VIF are properly handled.
    </li>

    <li>
      Some CMS systems, including OpenStack, fully start a VM only when its
      networking is ready.  To support this, <code>ovn-northd</code> notices
      the <code>chassis</code> column updated for the row in
      <code>Binding</code> table and pushes this upward by updating the
      <ref column="up" table="Logical_Switch_Port" db="OVN_NB"/> column
      in the OVN Northbound database's <ref table="Logical_Switch_Port"
      db="OVN_NB"/> table to indicate that the VIF is now up.  The CMS,
      if it uses this feature, can then react by allowing the VM's
      execution to proceed.
    </li>

    <li>
      On every hypervisor but the one where the VIF resides,
      <code>ovn-controller</code> notices the completely populated row in the
      <code>Binding</code> table.  This provides <code>ovn-controller</code>
      the physical location of the logical port, so each instance updates the
      OpenFlow tables of its switch (based on logical datapath flows in the OVN
      DB <code>Logical_Flow</code> table) so that packets to and from the VIF
      can be properly handled via tunnels.
    </li>

    <li>
      Eventually, a user powers off the VM that owns the VIF.  On the
      hypervisor where the VM was powered off, the VIF is deleted from the OVN
      integration bridge.
    </li>

    <li>
      On the hypervisor where the VM was powered off,
      <code>ovn-controller</code> notices that the VIF was deleted.  In
      response, it removes the <code>Chassis</code> column content in the
      <code>Binding</code> table for the logical port.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> notices the empty
      <code>Chassis</code> column in the <code>Binding</code> table's row
      for the logical port.  This means that <code>ovn-controller</code> no
      longer knows the physical location of the logical port, so each instance
      updates its OpenFlow table to reflect that.
    </li>

    <li>
      Eventually, when the VIF (or its entire VM) is no longer needed by
      anyone, an administrator deletes the VIF using the CMS user interface or
      API.  The CMS updates its own configuration.
    </li>

    <li>
      The CMS plugin removes the VIF from the OVN Northbound database,
      by deleting its row in the <code>Logical_Switch_Port</code> table.
    </li>

    <li>
      <code>ovn-northd</code> receives the OVN Northbound update and in turn
      updates the OVN Southbound database accordingly, by removing or updating
      the rows from the OVN Southbound database <code>Logical_Flow</code> table
      and <code>Binding</code> table that were related to the now-destroyed
      VIF.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> receives the
      <code>Logical_Flow</code> table updates that <code>ovn-northd</code> made
      in the previous step.  <code>ovn-controller</code> updates OpenFlow
      tables to reflect the update, although there may not be much to do, since
      the VIF had already become unreachable when it was removed from the
      <code>Binding</code> table in a previous step.
    </li>
  </ol>

  <h2>Life Cycle of a Container Interface Inside a VM</h2>

  <p>
    OVN provides virtual network abstractions by converting information
    written in OVN_NB database to OpenFlow flows in each hypervisor.  Secure
    virtual networking for multi-tenants can only be provided if OVN controller
    is the only entity that can modify flows in Open vSwitch.  When the
    Open vSwitch integration bridge resides in the hypervisor, it is a
    fair assumption to make that tenant workloads running inside VMs cannot
    make any changes to Open vSwitch flows.
  </p>

  <p>
    If the infrastructure provider trusts the applications inside the
    containers not to break out and modify the Open vSwitch flows, then
    containers can be run in hypervisors.  This is also the case when
    containers are run inside the VMs and Open vSwitch integration bridge
    with flows added by OVN controller resides in the same VM.  For both
    the above cases, the workflow is the same as explained with an example
    in the previous section ("Life Cycle of a VIF").
  </p>

  <p>
    This section talks about the life cycle of a container interface (CIF)
    when containers are created in the VMs and the Open vSwitch integration
    bridge resides inside the hypervisor.  In this case, even if a container
    application breaks out, other tenants are not affected because the
    containers running inside the VMs cannot modify the flows in the
    Open vSwitch integration bridge.
  </p>

  <p>
    When multiple containers are created inside a VM, there are multiple
    CIFs associated with them.  The network traffic associated with these
    CIFs need to reach the Open vSwitch integration bridge running in the
    hypervisor for OVN to support virtual network abstractions.  OVN should
    also be able to distinguish network traffic coming from different CIFs.
    There are two ways to distinguish network traffic of CIFs.
  </p>

  <p>
    One way is to provide one VIF for every CIF (1:1 model).  This means that
    there could be a lot of network devices in the hypervisor.  This would slow
    down OVS because of all the additional CPU cycles needed for the management
    of all the VIFs.  It would also mean that the entity creating the
    containers in a VM should also be able to create the corresponding VIFs in
    the hypervisor.
  </p>

  <p>
    The second way is to provide a single VIF for all the CIFs (1:many model).
    OVN could then distinguish network traffic coming from different CIFs via
    a tag written in every packet.  OVN uses this mechanism and uses VLAN as
    the tagging mechanism.
  </p>

  <ol>
    <li>
      A CIF's life cycle begins when a container is spawned inside a VM by
      the either the same CMS that created the VM or a tenant that owns that VM
      or even a container Orchestration System that is different than the CMS
      that initially created the VM.  Whoever the entity is, it will need to
      know the <var>vif-id</var> that is associated with the network interface
      of the VM through which the container interface's network traffic is
      expected to go through.  The entity that creates the container interface
      will also need to choose an unused VLAN inside that VM.
    </li>

    <li>
      The container spawning entity (either directly or through the CMS that
      manages the underlying infrastructure) updates the OVN Northbound
      database to include the new CIF, by adding a row to the
      <code>Logical_Switch_Port</code> table.  In the new row,
      <code>name</code> is any unique identifier,
      <code>parent_name</code> is the <var>vif-id</var> of the VM
      through which the CIF's network traffic is expected to go through
      and the <code>tag</code> is the VLAN tag that identifies the
      network traffic of that CIF.
    </li>

    <li>
      <code>ovn-northd</code> receives the OVN Northbound database update.  In
      turn, it makes the corresponding updates to the OVN Southbound database,
      by adding rows to the OVN Southbound database's <code>Logical_Flow</code>
      table to reflect the new port and also by creating a new row in the
      <code>Binding</code> table and populating all its columns except the
      column that identifies the <code>chassis</code>.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> subscribes to the
      changes in the <code>Binding</code> table.  When a new row is created
      by <code>ovn-northd</code> that includes a value in
      <code>parent_port</code> column of <code>Binding</code> table, the
      <code>ovn-controller</code> in the hypervisor whose OVN integration bridge
      has that same value in <var>vif-id</var> in
      <code>external_ids</code>:<code>iface-id</code>
      updates the local hypervisor's OpenFlow tables so that packets to and
      from the VIF with the particular VLAN <code>tag</code> are properly
      handled.  Afterward it updates the <code>chassis</code> column of
      the <code>Binding</code> to reflect the physical location.
    </li>

    <li>
      One can only start the application inside the container after the
      underlying network is ready.  To support this, <code>ovn-northd</code>
      notices the updated <code>chassis</code> column in <code>Binding</code>
      table and updates the <ref column="up" table="Logical_Switch_Port"
      db="OVN_NB"/> column in the OVN Northbound database's
      <ref table="Logical_Switch_Port" db="OVN_NB"/> table to indicate that the
      CIF is now up.  The entity responsible to start the container application
      queries this value and starts the application.
    </li>

    <li>
      Eventually the entity that created and started the container, stops it.
      The entity, through the CMS (or directly) deletes its row in the
      <code>Logical_Switch_Port</code> table.
    </li>

    <li>
      <code>ovn-northd</code> receives the OVN Northbound update and in turn
      updates the OVN Southbound database accordingly, by removing or updating
      the rows from the OVN Southbound database <code>Logical_Flow</code> table
      that were related to the now-destroyed CIF.  It also deletes the row in
      the <code>Binding</code> table for that CIF.
    </li>

    <li>
      On every hypervisor, <code>ovn-controller</code> receives the
      <code>Logical_Flow</code> table updates that <code>ovn-northd</code> made
      in the previous step.  <code>ovn-controller</code> updates OpenFlow
      tables to reflect the update.
    </li>
  </ol>

  <h2>Architectural Physical Life Cycle of a Packet</h2>

  <p>
    This section describes how a packet travels from one virtual machine or
    container to another through OVN.  This description focuses on the physical
    treatment of a packet; for a description of the logical life cycle of a
    packet, please refer to the <code>Logical_Flow</code> table in
    <code>ovn-sb</code>(5).
  </p>

  <p>
    This section mentions several data and metadata fields, for clarity
    summarized here:
  </p>

  <dl>
    <dt>tunnel key</dt>
    <dd>
      When OVN encapsulates a packet in Geneve or another tunnel, it attaches
      extra data to it to allow the receiving OVN instance to process it
      correctly.  This takes different forms depending on the particular
      encapsulation, but in each case we refer to it here as the ``tunnel
      key.''  See <code>Tunnel Encapsulations</code>, below, for details.
    </dd>

    <dt>logical datapath field</dt>
    <dd>
      A field that denotes the logical datapath through which a packet is being
      processed.
      <!-- Keep the following in sync with MFF_LOG_DATAPATH in
           ovn/lib/logical-fields.h. -->
      OVN uses the field that OpenFlow 1.1+ simply (and confusingly) calls
      ``metadata'' to store the logical datapath.  (This field is passed across
      tunnels as part of the tunnel key.)
    </dd>

    <dt>logical input port field</dt>
    <dd>
      <p>
        A field that denotes the logical port from which the packet
        entered the logical datapath.
        <!-- Keep the following in sync with MFF_LOG_INPORT in
             ovn/lib/logical-fields.h. -->
        OVN stores this in Open vSwitch extension register number 14.
      </p>

      <p>
        Geneve and STT tunnels pass this field as part of the tunnel key.
        Although VXLAN tunnels do not explicitly carry a logical input port,
        OVN only uses VXLAN to communicate with gateways that from OVN's
        perspective consist of only a single logical port, so that OVN can set
        the logical input port field to this one on ingress to the OVN logical
        pipeline.
      </p>
    </dd>

    <dt>logical output port field</dt>
    <dd>
      <p>
        A field that denotes the logical port from which the packet will
        leave the logical datapath.  This is initialized to 0 at the
        beginning of the logical ingress pipeline.
        <!-- Keep the following in sync with MFF_LOG_OUTPORT in
             ovn/lib/logical-fields.h. -->
        OVN stores this in Open vSwitch extension register number 15.
      </p>

      <p>
        Geneve and STT tunnels pass this field as part of the tunnel key.
        VXLAN tunnels do not transmit the logical output port field.
        Since VXLAN tunnels do not carry a logical output port field in
        the tunnel key, when a packet is received from VXLAN tunnel by
        an OVN hypervisor, the packet is resubmitted to table 16 to
        determine the output port(s);  when the packet reaches table 32,
        these packets are resubmitted to table 33 for local delivery by
        checking a MLF_RCV_FROM_VXLAN flag, which is set when the packet
        arrives from a VXLAN tunnel.
      </p>
    </dd>

    <dt>conntrack zone field for logical ports</dt>
    <dd>
      A field that denotes the connection tracking zone for logical ports.
      The value only has local significance and is not meaningful between
      chassis.  This is initialized to 0 at the beginning of the logical
        <!-- Keep the following in sync with MFF_LOG_CT_ZONE in
             ovn/lib/logical-fields.h. -->
      ingress pipeline.  OVN stores this in Open vSwitch extension register
      number 13.
    </dd>

    <dt>conntrack zone fields for Gateway router</dt>
    <dd>
      Fields that denote the connection tracking zones for Gateway routers.
      These values only have local significance (only on chassis that have
      Gateway routers instantiated) and is not meaningful between
      chassis.  OVN stores the zone information for DNATting in Open vSwitch
        <!-- Keep the following in sync with MFF_LOG_DNAT_ZONE and
        MFF_LOG_SNAT_ZONE in ovn/lib/logical-fields.h. -->
      extension register number 11 and zone information for SNATing in
      Open vSwitch extension register number 12.
    </dd>

    <dt>logical flow flags</dt>
    <dd>
      The logical flags are intended to handle keeping context between
      tables in order to decide which rules in subsequent tables are
      matched.  These values only have local significance and are not
      meaningful between chassis.  OVN stores the logical flags in
        <!-- Keep the following in sync with MFF_LOG_FLAGS in
             ovn/lib/logical-fields.h. -->
      Open vSwitch extension register number 10.
    </dd>

    <dt>VLAN ID</dt>
    <dd>
      The VLAN ID is used as an interface between OVN and containers nested
      inside a VM (see <code>Life Cycle of a container interface inside a
      VM</code>, above, for more information).
    </dd>
  </dl>

  <p>
    Initially, a VM or container on the ingress hypervisor sends a packet on a
    port attached to the OVN integration bridge.  Then:
  </p>

  <ol>
    <li>
      <p>
        OpenFlow table 0 performs physical-to-logical translation.  It matches
        the packet's ingress port.  Its actions annotate the packet with
        logical metadata, by setting the logical datapath field to identify the
        logical datapath that the packet is traversing and the logical input
        port field to identify the ingress port.  Then it resubmits to table 16
        to enter the logical ingress pipeline.
      </p>

      <p>
        Packets that originate from a container nested within a VM are treated
        in a slightly different way.  The originating container can be
        distinguished based on the VIF-specific VLAN ID, so the
        physical-to-logical translation flows additionally match on VLAN ID and
        the actions strip the VLAN header.  Following this step, OVN treats
        packets from containers just like any other packets.
      </p>

      <p>
        Table 0 also processes packets that arrive from other chassis.  It
        distinguishes them from other packets by ingress port, which is a
        tunnel.  As with packets just entering the OVN pipeline, the actions
        annotate these packets with logical datapath and logical ingress port
        metadata.  In addition, the actions set the logical output port field,
        which is available because in OVN tunneling occurs after the logical
        output port is known.  These three pieces of information are obtained
        from the tunnel encapsulation metadata (see <code>Tunnel
        Encapsulations</code> for encoding details).  Then the actions resubmit
        to table 33 to enter the logical egress pipeline.
      </p>
    </li>

    <li>
      <p>
        OpenFlow tables 16 through 31 execute the logical ingress pipeline from
        the <code>Logical_Flow</code> table in the OVN Southbound database.
        These tables are expressed entirely in terms of logical concepts like
        logical ports and logical datapaths.  A big part of
        <code>ovn-controller</code>'s job is to translate them into equivalent
        OpenFlow (in particular it translates the table numbers:
        <code>Logical_Flow</code> tables 0 through 15 become OpenFlow tables 16
        through 31).
      </p>

      <p>
        Most OVN actions have fairly obvious implementations in OpenFlow (with
        OVS extensions), e.g. <code>next;</code> is implemented as
        <code>resubmit</code>, <code><var>field</var> =
        <var>constant</var>;</code> as <code>set_field</code>.  A few are worth
        describing in more detail:
      </p>

      <dl>
        <dt><code>output:</code></dt>
        <dd>
          Implemented by resubmitting the packet to table 32.  If the pipeline
          executes more than one <code>output</code> action, then each one is
          separately resubmitted to table 32.  This can be used to send
          multiple copies of the packet to multiple ports.  (If the packet was
          not modified between the <code>output</code> actions, and some of the
          copies are destined to the same hypervisor, then using a logical
          multicast output port would save bandwidth between hypervisors.)
        </dd>

        <dt><code>get_arp(<var>P</var>, <var>A</var>);</code></dt>
        <dt><code>get_nd(<var>P</var>, <var>A</var>);</code></dt>
        <dd>
          <p>
            Implemented by storing arguments into OpenFlow fields, then
            resubmitting to table 66, which <code>ovn-controller</code>
            populates with flows generated from the <code>MAC_Binding</code>
            table in the OVN Southbound database.  If there is a match in table
            66, then its actions store the bound MAC in the Ethernet
            destination address field.
          </p>

          <p>
            (The OpenFlow actions save and restore the OpenFlow fields used for
            the arguments, so that the OVN actions do not have to be aware of
            this temporary use.)
          </p>
        </dd>

        <dt><code>put_arp(<var>P</var>, <var>A</var>, <var>E</var>);</code></dt>
        <dt><code>put_nd(<var>P</var>, <var>A</var>, <var>E</var>);</code></dt>
        <dd>
          <p>
            Implemented by storing the arguments into OpenFlow fields, then
            outputting a packet to <code>ovn-controller</code>, which updates
            the <code>MAC_Binding</code> table.
          </p>

          <p>
            (The OpenFlow actions save and restore the OpenFlow fields used for
            the arguments, so that the OVN actions do not have to be aware of
            this temporary use.)
          </p>
        </dd>
      </dl>
    </li>

    <li>
      <p>
        OpenFlow tables 32 through 47 implement the <code>output</code> action
        in the logical ingress pipeline.  Specifically, table 32 handles
        packets to remote hypervisors, table 33 handles packets to the local
        hypervisor, and table 34 checks whether packets whose logical ingress
        and egress port are the same should be discarded.
      </p>

      <p>
        Logical patch ports are a special case.  Logical patch ports do not
        have a physical location and effectively reside on every hypervisor.
        Thus, flow table 33, for output to ports on the local hypervisor,
        naturally implements output to unicast logical patch ports too.
        However, applying the same logic to a logical patch port that is part
        of a logical multicast group yields packet duplication, because each
        hypervisor that contains a logical port in the multicast group will
        also output the packet to the logical patch port.  Thus, multicast
        groups implement output to logical patch ports in table 32.
      </p>

      <p>
        Each flow in table 32 matches on a logical output port for unicast or
        multicast logical ports that include a logical port on a remote
        hypervisor.  Each flow's actions implement sending a packet to the port
        it matches.  For unicast logical output ports on remote hypervisors,
        the actions set the tunnel key to the correct value, then send the
        packet on the tunnel port to the correct hypervisor.  (When the remote
        hypervisor receives the packet, table 0 there will recognize it as a
        tunneled packet and pass it along to table 33.)  For multicast logical
        output ports, the actions send one copy of the packet to each remote
        hypervisor, in the same way as for unicast destinations.  If a
        multicast group includes a logical port or ports on the local
        hypervisor, then its actions also resubmit to table 33.  Table 32 also
        includes a fallback flow that resubmits to table 33 if there is no
        other match.  Table 32 also contains a higher priority rule to match
        packets received from VXLAN tunnels, based on flag MLF_RCV_FROM_VXLAN
        and resubmit these packets to table 33 for local delivery. Packets
        received from VXLAN tunnels reach here because of a lack of logical
        output port field in the tunnel key and thus these packets needed to
        be submitted to table 16 to determine the output port.
      </p>

      <p>
        Flows in table 33 resemble those in table 32 but for logical ports that
        reside locally rather than remotely.  For unicast logical output ports
        on the local hypervisor, the actions just resubmit to table 34.  For
        multicast output ports that include one or more logical ports on the
        local hypervisor, for each such logical port <var>P</var>, the actions
        change the logical output port to <var>P</var>, then resubmit to table
        34.
      </p>

      <p>
        A special case is that when a localnet port exists on the datapath,
        remote port is connected by switching to the localnet port. In this
        case, instead of adding a flow in table 32 to reach the remote port, a
        flow is added in table 33 to switch the logical outport to the localnet
        port, and resubmit to table 33 as if it were unicasted to a logical
        port on the local hypervisor.
      </p>

      <p>
        Table 34 matches and drops packets for which the logical input and
        output ports are the same and the MLF_ALLOW_LOOPBACK flag is not
        set.  It resubmits other packets to table 48.
      </p>
    </li>

    <li>
      <p>
        OpenFlow tables 48 through 63 execute the logical egress pipeline from
        the <code>Logical_Flow</code> table in the OVN Southbound database.
        The egress pipeline can perform a final stage of validation before
        packet delivery.  Eventually, it may execute an <code>output</code>
        action, which <code>ovn-controller</code> implements by resubmitting to
        table 64.  A packet for which the pipeline never executes
        <code>output</code> is effectively dropped (although it may have been
        transmitted through a tunnel across a physical network).
      </p>

      <p>
        The egress pipeline cannot change the logical output port or cause
        further tunneling.
      </p>
    </li>

    <li>
     <p>
       Table 64 bypasses OpenFlow loopback when MLF_ALLOW_LOOPBACK is set.
       Logical loopback was handled in table 34, but OpenFlow by default also
       prevents loopback to the OpenFlow ingress port.  Thus, when
       MLF_ALLOW_LOOPBACK is set, OpenFlow table 64 saves the OpenFlow ingress
       port, sets it to zero, resubmits to table 65 for logical-to-physical
       transformation, and then restores the OpenFlow ingress port,
       effectively disabling OpenFlow loopback prevents.  When
       MLF_ALLOW_LOOPBACK is unset, table 64 flow simply resubmits to table
       65.
     </p>
    </li>

    <li>
      <p>
        OpenFlow table 65 performs logical-to-physical translation, the
        opposite of table 0.  It matches the packet's logical egress port.  Its
        actions output the packet to the port attached to the OVN integration
        bridge that represents that logical port.  If the logical egress port
        is a container nested with a VM, then before sending the packet the
        actions push on a VLAN header with an appropriate VLAN ID.
      </p>

      <p>
        If the logical egress port is a logical patch port, then table 65
        outputs to an OVS patch port that represents the logical patch port.
        The packet re-enters the OpenFlow flow table from the OVS patch port's
        peer in table 0, which identifies the logical datapath and logical
        input port based on the OVS patch port's OpenFlow port number.
      </p>
    </li>
  </ol>

  <h2>Life Cycle of a VTEP gateway</h2>

  <p>
    A gateway is a chassis that forwards traffic between the OVN-managed
    part of a logical network and a physical VLAN,  extending a
    tunnel-based logical network into a physical network.
  </p>

  <p>
    The steps below refer often to details of the OVN and VTEP database
    schemas.  Please see <code>ovn-sb</code>(5), <code>ovn-nb</code>(5)
    and <code>vtep</code>(5), respectively, for the full story on these
    databases.
  </p>

  <ol>
    <li>
      A VTEP gateway's life cycle begins with the administrator registering
      the VTEP gateway as a <code>Physical_Switch</code> table entry in the
      <code>VTEP</code> database.  The <code>ovn-controller-vtep</code>
      connected to this VTEP database, will recognize the new VTEP gateway
      and create a new <code>Chassis</code> table entry for it in the
      <code>OVN_Southbound</code> database.
    </li>

    <li>
      The administrator can then create a new <code>Logical_Switch</code>
      table entry, and bind a particular vlan on a VTEP gateway's port to
      any VTEP logical switch.  Once a VTEP logical switch is bound to
      a VTEP gateway, the <code>ovn-controller-vtep</code> will detect
      it and add its name to the <var>vtep_logical_switches</var>
      column of the <code>Chassis</code> table in the <code>
      OVN_Southbound</code> database.  Note, the <var>tunnel_key</var>
      column of VTEP logical switch is not filled at creation.  The
      <code>ovn-controller-vtep</code> will set the column when the
      correponding vtep logical switch is bound to an OVN logical network.
    </li>

    <li>
      Now, the administrator can use the CMS to add a VTEP logical switch
      to the OVN logical network.  To do that, the CMS must first create a
      new <code>Logical_Switch_Port</code> table entry in the <code>
      OVN_Northbound</code> database.  Then, the <var>type</var> column
      of this entry must be set to "vtep".  Next, the <var>
      vtep-logical-switch</var> and <var>vtep-physical-switch</var> keys
      in the <var>options</var> column must also be specified, since
      multiple VTEP gateways can attach to the same VTEP logical switch.
    </li>

    <li>
      The newly created logical port in the <code>OVN_Northbound</code>
      database and its configuration will be passed down to the <code>
      OVN_Southbound</code> database as a new <code>Port_Binding</code>
      table entry.  The <code>ovn-controller-vtep</code> will recognize the
      change and bind the logical port to the corresponding VTEP gateway
      chassis.  Configuration of binding the same VTEP logical switch to
      a different OVN logical networks is not allowed and a warning will be
      generated in the log.
    </li>

    <li>
      Beside binding to the VTEP gateway chassis, the <code>
      ovn-controller-vtep</code> will update the <var>tunnel_key</var>
      column of the VTEP logical switch to the corresponding <code>
      Datapath_Binding</code> table entry's <var>tunnel_key</var> for the
      bound OVN logical network.
    </li>

    <li>
      Next, the <code>ovn-controller-vtep</code> will keep reacting to the
      configuration change in the <code>Port_Binding</code> in the
      <code>OVN_Northbound</code> database, and updating the
      <code>Ucast_Macs_Remote</code> table in the <code>VTEP</code> database.
      This allows the VTEP gateway to understand where to forward the unicast
      traffic coming from the extended external network.
    </li>

    <li>
      Eventually, the VTEP gateway's life cycle ends when the administrator
      unregisters the VTEP gateway from the <code>VTEP</code> database.
      The <code>ovn-controller-vtep</code> will recognize the event and
      remove all related configurations (<code>Chassis</code> table entry
      and port bindings) in the <code>OVN_Southbound</code> database.
    </li>

    <li>
      When the <code>ovn-controller-vtep</code> is terminated, all related
      configurations in the <code>OVN_Southbound</code> database and
      the <code>VTEP</code> database will be cleaned, including
      <code>Chassis</code> table entries for all registered VTEP gateways
      and their port bindings, and all <code>Ucast_Macs_Remote</code> table
      entries and the <code>Logical_Switch</code> tunnel keys.
    </li>
  </ol>

  <h1>Design Decisions</h1>

  <h2>Tunnel Encapsulations</h2>

  <p>
    OVN annotates logical network packets that it sends from one hypervisor to
    another with the following three pieces of metadata, which are encoded in
    an encapsulation-specific fashion:
  </p>

  <ul>
    <li>
      24-bit logical datapath identifier, from the <code>tunnel_key</code>
      column in the OVN Southbound <code>Datapath_Binding</code> table.
    </li>

    <li>
      15-bit logical ingress port identifier.  ID 0 is reserved for internal
      use within OVN.  IDs 1 through 32767, inclusive, may be assigned to
      logical ports (see the <code>tunnel_key</code> column in the OVN
      Southbound <code>Port_Binding</code> table).
    </li>

    <li>
      16-bit logical egress port identifier.  IDs 0 through 32767 have the same
      meaning as for logical ingress ports.  IDs 32768 through 65535,
      inclusive, may be assigned to logical multicast groups (see the
      <code>tunnel_key</code> column in the OVN Southbound
      <code>Multicast_Group</code> table).
    </li>
  </ul>

  <p>
    For hypervisor-to-hypervisor traffic, OVN supports only Geneve and STT
    encapsulations, for the following reasons:
  </p>

  <ul>
    <li>
      Only STT and Geneve support the large amounts of metadata (over 32 bits
      per packet) that OVN uses (as described above).
    </li>

    <li>
      STT and Geneve use randomized UDP or TCP source ports that allows
      efficient distribution among multiple paths in environments that use ECMP
      in their underlay.
    </li>

    <li>
      NICs are available to offload STT and Geneve encapsulation and
      decapsulation.
    </li>
  </ul>

  <p>
    Due to its flexibility, the preferred encapsulation between hypervisors is
    Geneve.  For Geneve encapsulation, OVN transmits the logical datapath
    identifier in the Geneve VNI.

    <!-- Keep the following in sync with ovn/controller/physical.h. -->
    OVN transmits the logical ingress and logical egress ports in a TLV with
    class 0x0102, type 0x80, and a 32-bit value encoded as follows, from MSB to
    LSB:
  </p>

  <diagram>
    <header name="">
      <bits name="rsv" above="1" below="0" width=".25"/>
      <bits name="ingress port" above="15" width=".75"/>
      <bits name="egress port" above="16" width=".75"/>
    </header>
  </diagram>

  <p>
    Environments whose NICs lack Geneve offload may prefer STT encapsulation
    for performance reasons.  For STT encapsulation, OVN encodes all three
    pieces of logical metadata in the STT 64-bit tunnel ID as follows, from MSB
    to LSB:
  </p>

  <diagram>
    <header name="">
      <bits name="reserved" above="9" below="0" width=".5"/>
      <bits name="ingress port" above="15" width=".75"/>
      <bits name="egress port" above="16" width=".75"/>
      <bits name="datapath" above="24" width="1.25"/>
    </header>
  </diagram>

  <p>
    For connecting to gateways, in addition to Geneve and STT, OVN supports
    VXLAN, because only VXLAN support is common on top-of-rack (ToR) switches.
    Currently, gateways have a feature set that matches the capabilities as
    defined by the VTEP schema, so fewer bits of metadata are necessary.  In
    the future, gateways that do not support encapsulations with large amounts
    of metadata may continue to have a reduced feature set.
  </p>
</manpage>