Skip to content

Commit

Permalink
ovn-northd: Force SNAT for multiple gateway routers.
Browse files Browse the repository at this point in the history 
When multiple gateway routers exist, a packet can
enter any gateway router. Once the packet reaches its
destination, its reverse direction should be via the
same gateway router.  This is achieved by doing a SNAT
of the packet in the inward direction (towards logical space)
with a IP address of the gateway router such that packet travels back
to the same gateway router.

To do the above, we introduce two new options in the logical router.

options:dnat_force_snat_ip=$IP will force SNAT any packet to $IP if
it has been previously DNATted.

options:lb_force_snat_ip=$IP will force SNAT any packet to $IP if
it has been previously load-balanced.

Signed-off-by: Gurucharan Shetty <[email protected]>
Acked-by: Mickey Spiegel <[email protected]>
  • Loading branch information
@shettyg
shettyg committed Nov 29, 2016
1 parent 50da3c7 commit 65d8810
Show file tree
Hide file tree
Showing 6 changed files with 637 additions and 21 deletions.
8 changes: 8 additions & 0 deletions ovn/lib/logical-fields.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,14 @@ ovn_init_symtab(struct shash *symtab)
char flags_str[16];
snprintf(flags_str, sizeof flags_str, "flags[%d]", MLF_ALLOW_LOOPBACK_BIT);
expr_symtab_add_subfield(symtab, "flags.loopback", NULL, flags_str);
snprintf(flags_str, sizeof flags_str, "flags[%d]",
MLF_FORCE_SNAT_FOR_DNAT_BIT);
expr_symtab_add_subfield(symtab, "flags.force_snat_for_dnat", NULL,
flags_str);
snprintf(flags_str, sizeof flags_str, "flags[%d]",
MLF_FORCE_SNAT_FOR_LB_BIT);
expr_symtab_add_subfield(symtab, "flags.force_snat_for_lb", NULL,
flags_str);

/* Connection tracking state. */
expr_symtab_add_field(symtab, "ct_mark", MFF_CT_MARK, NULL, false);
Expand Down
10 changes: 10 additions & 0 deletions ovn/lib/logical-fields.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ void ovn_init_symtab(struct shash *symtab);
enum mff_log_flags_bits {
MLF_ALLOW_LOOPBACK_BIT = 0,
MLF_RCV_FROM_VXLAN_BIT = 1,
MLF_FORCE_SNAT_FOR_DNAT_BIT = 2,
MLF_FORCE_SNAT_FOR_LB_BIT = 3,
};

/* MFF_LOG_FLAGS_REG flag assignments */
Expand All @@ -59,6 +61,14 @@ enum mff_log_flags {
* VXLAN encapsulation. Egress port information is available for
* Geneve and STT tunnel types. */
MLF_RCV_FROM_VXLAN = (1 << MLF_RCV_FROM_VXLAN_BIT),

/* Indicate that a packet needs a force SNAT in the gateway router when
* DNAT has taken place. */
MLF_FORCE_SNAT_FOR_DNAT = (1 << MLF_FORCE_SNAT_FOR_DNAT_BIT),

/* Indicate that a packet needs a force SNAT in the gateway router when
* load-balancing has taken place. */
MLF_FORCE_SNAT_FOR_LB = (1 << MLF_FORCE_SNAT_FOR_LB_BIT),
};

#endif /* ovn/lib/logical-fields.h */
72 changes: 67 additions & 5 deletions ovn/northd/ovn-northd.8.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1200,9 +1200,23 @@ icmp4 {
<ul>
<li>
<p>
For each configuration in the OVN Northbound database, that asks
If the Gateway router has been configured to force SNAT any
previously DNATted packets to <var>B</var>, a priority-110 flow
matches <code>ip &amp;&amp; ip4.dst == <var>B</var></code> with
an action <code>ct_snat; next;</code>.
</p>

<p>
If the Gateway router has been configured to force SNAT any
previously load-balanced packets to <var>B</var>, a priority-100 flow
matches <code>ip &amp;&amp; ip4.dst == <var>B</var></code> with
an action <code>ct_snat; next;</code>.
</p>

<p>
For each NAT configuration in the OVN Northbound database, that asks
to change the source IP address of a packet from <var>A</var> to
<var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
<var>B</var>, a priority-90 flow matches <code>ip &amp;&amp;
ip4.dst == <var>B</var></code> with an action
<code>ct_snat; next;</code>.
</p>
Expand Down Expand Up @@ -1231,7 +1245,23 @@ icmp4 {
&amp;&amp; <var>P</var> &amp;&amp; <var>P</var>.dst == <var>PORT
</var></code> with an action of <code>ct_lb(<var>args</var>)</code>,
where <var>args</var> contains comma separated IPv4 addresses (and
optional port numbers) to load balance to.
optional port numbers) to load balance to. If the Gateway router
is configured to force SNAT any load-balanced packets, the above
action will be replaced by <code>flags.force_snat_for_lb = 1;
ct_lb(<var>args</var>);</code>.
</li>

<li>
For all the configured load balancing rules for Gateway router in
<code>OVN_Northbound</code> database that includes a L4 port
<var>PORT</var> of protocol <var>P</var> and IPv4 address
<var>VIP</var>, a priority-120 flow that matches on
<code>ct.est &amp;&amp; ip &amp;&amp; ip4.dst == <var>VIP</var>
&amp;&amp; <var>P</var> &amp;&amp; <var>P</var>.dst == <var>PORT
</var></code> with an action of <code>ct_dnat;</code>.
If the Gateway router is configured to force SNAT any load-balanced
packets, the above action will be replaced by
<code>flags.force_snat_for_lb = 1; ct_dnat;</code>.
</li>

<li>
Expand All @@ -1241,15 +1271,33 @@ icmp4 {
<code>ct.new &amp;&amp; ip &amp;&amp; ip4.dst ==
<var>VIP</var></code> with an action of
<code>ct_lb(<var>args</var>)</code>, where <var>args</var> contains
comma separated IPv4 addresses.
comma separated IPv4 addresses. If the Gateway router
is configured to force SNAT any load-balanced packets, the above
action will be replaced by <code>flags.force_snat_for_lb = 1;
ct_lb(<var>args</var>);</code>.
</li>

<li>
For all the configured load balancing rules for Gateway router in
<code>OVN_Northbound</code> database that includes just an IP address
<var>VIP</var> to match on, a priority-110 flow that matches on
<code>ct.est &amp;&amp; ip &amp;&amp; ip4.dst ==
<var>VIP</var></code> with an action of <code>ct_dnat;</code>.
If the Gateway router is configured to force SNAT any load-balanced
packets, the above action will be replaced by
<code>flags.force_snat_for_lb = 1; ct_dnat;</code>.
</li>

<li>
For each configuration in the OVN Northbound database, that asks
to change the destination IP address of a packet from <var>A</var> to
<var>B</var>, a priority-100 flow matches <code>ip &amp;&amp;
ip4.dst == <var>A</var></code> with an action
<code>flags.loopback = 1; ct_dnat(<var>B</var>);</code>.
<code>flags.loopback = 1; ct_dnat(<var>B</var>);</code>. If the
Gateway router is configured to force SNAT any DNATed packet,
the above action will be replaced by
<code>flags.force_snat_for_dnat = 1; flags.loopback = 1;
ct_dnat(<var>B</var>);</code>.
</li>

<li>
Expand Down Expand Up @@ -1487,6 +1535,20 @@ arp {
</p>
<ul>
<li>
<p>
If the Gateway router in the OVN Northbound database has been
configured to force SNAT a packet (that has been previously DNATted)
to <var>B</var>, a priority-100 flow matches
<code>flags.force_snat_for_dnat == 1 &amp;&amp; ip</code> with an
action <code>ct_snat(<var>B</var>);</code>.
</p>
<p>
If the Gateway router in the OVN Northbound database has been
configured to force SNAT a packet (that has been previously
load-balanced) to <var>B</var>, a priority-100 flow matches
<code>flags.force_snat_for_lb == 1 &amp;&amp; ip</code> with an
action <code>ct_snat(<var>B</var>);</code>.
</p>
<p>
For each configuration in the OVN Northbound database, that asks
to change the source IP address of a packet from an IP address of
Expand Down
157 changes: 141 additions & 16 deletions ovn/northd/ovn-northd.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3454,6 +3454,63 @@ op_put_v6_networks(struct ds *ds, const struct ovn_port *op)
ds_put_cstr(ds, "}");
}

static const char *
get_force_snat_ip(struct ovn_datapath *od, const char *key_type, ovs_be32 *ip)
{
char *key = xasprintf("%s_force_snat_ip", key_type);
const char *ip_address = smap_get(&od->nbr->options, key);
free(key);

if (ip_address) {
ovs_be32 mask;
char *error = ip_parse_masked(ip_address, ip, &mask);
if (error || mask != OVS_BE32_MAX) {
static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1);
VLOG_WARN_RL(&rl, "bad ip %s in options of router "UUID_FMT"",
ip_address, UUID_ARGS(&od->key));
free(error);
*ip = 0;
return NULL;
}
return ip_address;
}

*ip = 0;
return NULL;
}

static void
add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od,
struct ds *match, struct ds *actions, int priority,
const char *lb_force_snat_ip)
{
/* A match and actions for new connections. */
char *new_match = xasprintf("ct.new && %s", ds_cstr(match));
if (lb_force_snat_ip) {
char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s",
ds_cstr(actions));
ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, priority, new_match,
new_actions);
free(new_actions);
} else {
ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, priority, new_match,
ds_cstr(actions));
}

/* A match and actions for established connections. */
char *est_match = xasprintf("ct.est && %s", ds_cstr(match));
if (lb_force_snat_ip) {
ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, priority, est_match,
"flags.force_snat_for_lb = 1; ct_dnat;");
} else {
ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, priority, est_match,
"ct_dnat;");
}

free(new_match);
free(est_match);
}

static void
build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
struct hmap *lflows)
Expand Down Expand Up @@ -3675,8 +3732,26 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,

sset_destroy(&all_ips);

ovs_be32 *snat_ips = xmalloc(sizeof *snat_ips * op->od->nbr->n_nat);
/* A gateway router can have 2 SNAT IP addresses to force DNATed and
* LBed traffic respectively to be SNATed. In addition, there can be
* a number of SNAT rules in the NAT table. */
ovs_be32 *snat_ips = xmalloc(sizeof *snat_ips *
(op->od->nbr->n_nat + 2));
size_t n_snat_ips = 0;

ovs_be32 snat_ip;
const char *dnat_force_snat_ip = get_force_snat_ip(op->od, "dnat",
&snat_ip);
if (dnat_force_snat_ip) {
snat_ips[n_snat_ips++] = snat_ip;
}

const char *lb_force_snat_ip = get_force_snat_ip(op->od, "lb",
&snat_ip);
if (lb_force_snat_ip) {
snat_ips[n_snat_ips++] = snat_ip;
}

for (int i = 0; i < op->od->nbr->n_nat; i++) {
const struct nbrec_nat *nat;

Expand Down Expand Up @@ -3847,6 +3922,12 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
continue;
}

ovs_be32 snat_ip;
const char *dnat_force_snat_ip = get_force_snat_ip(od, "dnat",
&snat_ip);
const char *lb_force_snat_ip = get_force_snat_ip(od, "lb",
&snat_ip);

/* A set to hold all ips that need defragmentation and tracking. */
struct sset all_ips = SSET_INITIALIZER(&all_ips);

Expand All @@ -3869,14 +3950,16 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
sset_add(&all_ips, ip_address);
}

/* Higher priority rules are added in DNAT table to match on
* ct.new which in-turn have group id as an action for load
* balancing. */
/* Higher priority rules are added for load-balancing in DNAT
* table. For every match (on a VIP[:port]), we add two flows
* via add_router_lb_flow(). One flow is for specific matching
* on ct.new with an action of "ct_lb($targets);". The other
* flow is for ct.est with an action of "ct_dnat;". */
ds_clear(&actions);
ds_put_format(&actions, "ct_lb(%s);", node->value);

ds_clear(&match);
ds_put_format(&match, "ct.new && ip && ip4.dst == %s",
ds_put_format(&match, "ip && ip4.dst == %s",
ip_address);
free(ip_address);

Expand All @@ -3888,11 +3971,11 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
ds_put_format(&match, " && tcp && tcp.dst == %d",
port);
}
ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT,
120, ds_cstr(&match), ds_cstr(&actions));
add_router_lb_flow(lflows, od, &match, &actions, 120,
lb_force_snat_ip);
} else {
ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT,
110, ds_cstr(&match), ds_cstr(&actions));
add_router_lb_flow(lflows, od, &match, &actions, 110,
lb_force_snat_ip);
}
}
}
Expand Down Expand Up @@ -3968,7 +4051,7 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
|| !strcmp(nat->type, "dnat_and_snat")) {
ds_clear(&match);
ds_put_format(&match, "ip && ip4.dst == %s", nat->external_ip);
ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 100,
ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 90,
ds_cstr(&match), "ct_snat; next;");
}

Expand All @@ -3983,7 +4066,13 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
ds_clear(&match);
ds_put_format(&match, "ip && ip4.dst == %s", nat->external_ip);
ds_clear(&actions);
ds_put_format(&actions,"flags.loopback = 1; ct_dnat(%s);",
if (dnat_force_snat_ip) {
/* Indicate to the future tables that a DNAT has taken
* place and a force SNAT needs to be done in the Egress
* SNAT table. */
ds_put_format(&actions, "flags.force_snat_for_dnat = 1; ");
}
ds_put_format(&actions, "flags.loopback = 1; ct_dnat(%s);",
nat->logical_ip);
ovn_lflow_add(lflows, od, S_ROUTER_IN_DNAT, 100,
ds_cstr(&match), ds_cstr(&actions));
Expand All @@ -4008,8 +4097,47 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
}
}

/* Handle force SNAT options set in the gateway router. */
if (dnat_force_snat_ip) {
/* If a packet with destination IP address as that of the
* gateway router (as set in options:dnat_force_snat_ip) is seen,
* UNSNAT it. */
ds_clear(&match);
ds_put_format(&match, "ip && ip4.dst == %s", dnat_force_snat_ip);
ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 110,
ds_cstr(&match), "ct_snat; next;");

/* Higher priority rules to force SNAT with the IP addresses
* configured in the Gateway router. This only takes effect
* when the packet has already been DNATed once. */
ds_clear(&match);
ds_put_format(&match, "flags.force_snat_for_dnat == 1 && ip");
ds_clear(&actions);
ds_put_format(&actions, "ct_snat(%s);", dnat_force_snat_ip);
ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 100,
ds_cstr(&match), ds_cstr(&actions));
}
if (lb_force_snat_ip) {
/* If a packet with destination IP address as that of the
* gateway router (as set in options:lb_force_snat_ip) is seen,
* UNSNAT it. */
ds_clear(&match);
ds_put_format(&match, "ip && ip4.dst == %s", lb_force_snat_ip);
ovn_lflow_add(lflows, od, S_ROUTER_IN_UNSNAT, 100,
ds_cstr(&match), "ct_snat; next;");

/* Load balanced traffic will have flags.force_snat_for_lb set.
* Force SNAT it. */
ds_clear(&match);
ds_put_format(&match, "flags.force_snat_for_lb == 1 && ip");
ds_clear(&actions);
ds_put_format(&actions, "ct_snat(%s);", lb_force_snat_ip);
ovn_lflow_add(lflows, od, S_ROUTER_OUT_SNAT, 100,
ds_cstr(&match), ds_cstr(&actions));
}

/* Re-circulate every packet through the DNAT zone.
* This helps with three things.
* This helps with two things.
*
* 1. Any packet that needs to be unDNATed in the reverse
* direction gets unDNATed. Ideally this could be done in
Expand All @@ -4018,10 +4146,7 @@ build_lrouter_flows(struct hmap *datapaths, struct hmap *ports,
* ip address being external IP address for IP routing,
* we can do it here, saving a future re-circulation.
*
* 2. Established load-balanced connections automatically get
* DNATed.
*
* 3. Any packet that was sent through SNAT zone in the
* 2. Any packet that was sent through SNAT zone in the
* previous table automatically gets re-circulated to get
* back the new destination IP address that is needed for
* routing in the openflow pipeline. */
Expand Down
Loading

0 comments on commit 65d8810

Please sign in to comment.