* [dpdk-dev] [PATCH] net/mlx5: fix VXLAN port registration race condition
@ 2019-01-23 7:51 Viacheslav Ovsiienko
2019-01-23 13:19 ` [dpdk-dev] [PATCH v2] " Viacheslav Ovsiienko
0 siblings, 1 reply; 3+ messages in thread
From: Viacheslav Ovsiienko @ 2019-01-23 7:51 UTC (permalink / raw)
To: dev; +Cc: shahafs
E-Switch VXLAN tunneling rules require virtual VXLAN network
devices be created. These devices are managed by MLX5 PMD and
created/deleted dynamically.
Kernel creates the VXLAN devices and registers VXLAN UDP ports
to be hardware offloaded within the NIC kernel drivers. The
registration process is being performed into context of working
kernel thread and the race conditions might happen.
The VXLAN device is created and success code is returned to calling
application, but the UDP port registration process is not completed
yet and the next applied rule might be rejected by the driver with
ENOSUP code. This patch adds some timeout for new created devices,
allowing port registration process to be completed. The waiting
is performed once after device been created and first rule is being
applied.
Fixes: 95a464cecc21 ("net/mlx5: add E-switch VXLAN tunnel devices management")
Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
drivers/net/mlx5/mlx5_flow_tcf.c | 49 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 47 insertions(+), 2 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_flow_tcf.c b/drivers/net/mlx5/mlx5_flow_tcf.c
index 859e99a..b8204df 100644
--- a/drivers/net/mlx5/mlx5_flow_tcf.c
+++ b/drivers/net/mlx5/mlx5_flow_tcf.c
@@ -354,6 +354,11 @@ struct tc_tunnel_key {
/** Parameters of VXLAN devices created by driver. */
#define MLX5_VXLAN_DEFAULT_VNI 1
#define MLX5_VXLAN_DEVICE_PFX "vmlx_"
+/**
+ * Timeout in milliseconds to wait VXLAN UDP offloaded port
+ * registration completed within the mlx5 driver.
+ */
+#define MLX5_VXLAN_WAIT_PORT_REG_MS 250
/** Tunnel action type, used for @p type in header structure. */
enum flow_tcf_tunact_type {
@@ -445,7 +450,8 @@ struct tcf_vtep {
uint32_t refcnt;
unsigned int ifindex; /**< Own interface index. */
uint16_t port;
- uint8_t created;
+ uint32_t created:1; /**< Actually created by PMD. */
+ uint32_t waitreg:1; /**< Wait for VXLAN UDP port registration. */
};
/** Tunnel descriptor header, common for all tunnel types. */
@@ -5167,6 +5173,7 @@ struct tcf_nlcb_context {
* when we do not need it anymore.
*/
vtep->created = 1;
+ vtep->waitreg = 1;
}
/* Try to get ifindex of created of pre-existing device. */
ret = if_nametoindex(name);
@@ -5648,6 +5655,8 @@ struct tcf_nlcb_query {
struct mlx5_flow *dev_flow;
struct nlmsghdr *nlh;
struct tcmsg *tcm;
+ uint64_t start = 0;
+ uint64_t twait = 0;
int ret;
dev_flow = LIST_FIRST(&flow->dev_flows);
@@ -5681,8 +5690,44 @@ struct tcf_nlcb_query {
dev_flow->tcf.tunnel->ifindex_org);
*dev_flow->tcf.tunnel->ifindex_ptr =
dev_flow->tcf.tunnel->vtep->ifindex;
+ if (dev_flow->tcf.tunnel->vtep->waitreg) {
+ /* Clear wait flag for VXLAN port registration. */
+ dev_flow->tcf.tunnel->vtep->waitreg = 0;
+ twait = rte_get_timer_hz();
+ assert(twait > MS_PER_S);
+ twait = twait * MLX5_VXLAN_WAIT_PORT_REG_MS;
+ twait = twait / MS_PER_S;
+ start = rte_get_timer_cycles();
+ }
}
- ret = flow_tcf_nl_ack(ctx, nlh, flow_tcf_collect_apply_cb, nlh);
+ /*
+ * Kernel creates the VXLAN devices and registers UDP ports to
+ * be hardware offloaded within the NIC kernel drivers. The
+ * registration process is being performed into context of
+ * working kernel thread and the race conditions might happen.
+ * The VXLAN device is created and success is returned to
+ * calling application, but the UDP port registration process
+ * is not completed yet. The next applied rule may be rejected
+ * by the driver with ENOSUP code. We are going to wait a bit,
+ * allowing registration process to be completed. The waiting
+ * is performed once after device been created.
+ */
+ do {
+ struct timespec onems;
+
+ ret = flow_tcf_nl_ack(ctx, nlh,
+ flow_tcf_collect_apply_cb, nlh);
+ if (!ret || ret != -ENOTSUP || !twait)
+ break;
+ /* Wait one millisecond and try again till timeout. */
+ onems.tv_sec = 0;
+ onems.tv_nsec = NS_PER_S / MS_PER_S;
+ nanosleep(&onems, 0);
+ if ((rte_get_timer_cycles() - start) > twait) {
+ /* Timeout elapsed, try once more and exit. */
+ twait = 0;
+ }
+ } while (true);
if (!ret) {
if (!tcm->tcm_handle) {
flow_tcf_remove(dev, flow);
--
1.8.3.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [dpdk-dev] [PATCH v2] net/mlx5: fix VXLAN port registration race condition
2019-01-23 7:51 [dpdk-dev] [PATCH] net/mlx5: fix VXLAN port registration race condition Viacheslav Ovsiienko
@ 2019-01-23 13:19 ` Viacheslav Ovsiienko
2019-01-24 11:16 ` Shahaf Shuler
0 siblings, 1 reply; 3+ messages in thread
From: Viacheslav Ovsiienko @ 2019-01-23 13:19 UTC (permalink / raw)
To: dev; +Cc: shahafs
E-Switch VXLAN tunneling rules require virtual VXLAN network
devices be created. These devices are managed by MLX5 PMD and
created/deleted dynamically.
Kernel creates the VXLAN devices and registers VXLAN UDP ports
to be hardware offloaded within the NIC kernel drivers. The
registration process is being performed into context of working
kernel thread and the race conditions might happen.
The VXLAN device is created and success code is returned to calling
application, but the UDP port registration process is not completed
yet and the next applied rule might be rejected by the driver with
ENOSUP code. This patch adds some timeout for new created devices,
allowing port registration process to be completed. The waiting
is performed once after device been created and first rule is being
applied.
Fixes: 95a464cecc21 ("net/mlx5: add E-switch VXLAN tunnel devices management")
Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
---
v2:
Added <rte_cycles.h> inclusion, not was not included on some
building setups (ARMv8).
v1:
Original patch: http://patches.dpdk.org/patch/50015/
drivers/net/mlx5/mlx5_flow_tcf.c | 50 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 48 insertions(+), 2 deletions(-)
diff --git a/drivers/net/mlx5/mlx5_flow_tcf.c b/drivers/net/mlx5/mlx5_flow_tcf.c
index 859e99a..f4921e0 100644
--- a/drivers/net/mlx5/mlx5_flow_tcf.c
+++ b/drivers/net/mlx5/mlx5_flow_tcf.c
@@ -28,6 +28,7 @@
#include <rte_flow.h>
#include <rte_malloc.h>
#include <rte_common.h>
+#include <rte_cycles.h>
#include "mlx5.h"
#include "mlx5_flow.h"
@@ -354,6 +355,11 @@ struct tc_tunnel_key {
/** Parameters of VXLAN devices created by driver. */
#define MLX5_VXLAN_DEFAULT_VNI 1
#define MLX5_VXLAN_DEVICE_PFX "vmlx_"
+/**
+ * Timeout in milliseconds to wait VXLAN UDP offloaded port
+ * registration completed within the mlx5 driver.
+ */
+#define MLX5_VXLAN_WAIT_PORT_REG_MS 250
/** Tunnel action type, used for @p type in header structure. */
enum flow_tcf_tunact_type {
@@ -445,7 +451,8 @@ struct tcf_vtep {
uint32_t refcnt;
unsigned int ifindex; /**< Own interface index. */
uint16_t port;
- uint8_t created;
+ uint32_t created:1; /**< Actually created by PMD. */
+ uint32_t waitreg:1; /**< Wait for VXLAN UDP port registration. */
};
/** Tunnel descriptor header, common for all tunnel types. */
@@ -5167,6 +5174,7 @@ struct tcf_nlcb_context {
* when we do not need it anymore.
*/
vtep->created = 1;
+ vtep->waitreg = 1;
}
/* Try to get ifindex of created of pre-existing device. */
ret = if_nametoindex(name);
@@ -5648,6 +5656,8 @@ struct tcf_nlcb_query {
struct mlx5_flow *dev_flow;
struct nlmsghdr *nlh;
struct tcmsg *tcm;
+ uint64_t start = 0;
+ uint64_t twait = 0;
int ret;
dev_flow = LIST_FIRST(&flow->dev_flows);
@@ -5681,8 +5691,44 @@ struct tcf_nlcb_query {
dev_flow->tcf.tunnel->ifindex_org);
*dev_flow->tcf.tunnel->ifindex_ptr =
dev_flow->tcf.tunnel->vtep->ifindex;
+ if (dev_flow->tcf.tunnel->vtep->waitreg) {
+ /* Clear wait flag for VXLAN port registration. */
+ dev_flow->tcf.tunnel->vtep->waitreg = 0;
+ twait = rte_get_timer_hz();
+ assert(twait > MS_PER_S);
+ twait = twait * MLX5_VXLAN_WAIT_PORT_REG_MS;
+ twait = twait / MS_PER_S;
+ start = rte_get_timer_cycles();
+ }
}
- ret = flow_tcf_nl_ack(ctx, nlh, flow_tcf_collect_apply_cb, nlh);
+ /*
+ * Kernel creates the VXLAN devices and registers UDP ports to
+ * be hardware offloaded within the NIC kernel drivers. The
+ * registration process is being performed into context of
+ * working kernel thread and the race conditions might happen.
+ * The VXLAN device is created and success is returned to
+ * calling application, but the UDP port registration process
+ * is not completed yet. The next applied rule may be rejected
+ * by the driver with ENOSUP code. We are going to wait a bit,
+ * allowing registration process to be completed. The waiting
+ * is performed once after device been created.
+ */
+ do {
+ struct timespec onems;
+
+ ret = flow_tcf_nl_ack(ctx, nlh,
+ flow_tcf_collect_apply_cb, nlh);
+ if (!ret || ret != -ENOTSUP || !twait)
+ break;
+ /* Wait one millisecond and try again till timeout. */
+ onems.tv_sec = 0;
+ onems.tv_nsec = NS_PER_S / MS_PER_S;
+ nanosleep(&onems, 0);
+ if ((rte_get_timer_cycles() - start) > twait) {
+ /* Timeout elapsed, try once more and exit. */
+ twait = 0;
+ }
+ } while (true);
if (!ret) {
if (!tcm->tcm_handle) {
flow_tcf_remove(dev, flow);
--
1.8.3.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [dpdk-dev] [PATCH v2] net/mlx5: fix VXLAN port registration race condition
2019-01-23 13:19 ` [dpdk-dev] [PATCH v2] " Viacheslav Ovsiienko
@ 2019-01-24 11:16 ` Shahaf Shuler
0 siblings, 0 replies; 3+ messages in thread
From: Shahaf Shuler @ 2019-01-24 11:16 UTC (permalink / raw)
To: Slava Ovsiienko, dev
Wednesday, January 23, 2019 3:19 PM, Viacheslav Ovsiienko:
> Subject: [dpdk-dev] [PATCH v2] net/mlx5: fix VXLAN port registration race
> condition
>
> E-Switch VXLAN tunneling rules require virtual VXLAN network devices be
> created. These devices are managed by MLX5 PMD and created/deleted
> dynamically.
>
> Kernel creates the VXLAN devices and registers VXLAN UDP ports to be
> hardware offloaded within the NIC kernel drivers. The registration process is
> being performed into context of working kernel thread and the race
> conditions might happen.
>
> The VXLAN device is created and success code is returned to calling
> application, but the UDP port registration process is not completed yet and
> the next applied rule might be rejected by the driver with ENOSUP code. This
> patch adds some timeout for new created devices, allowing port registration
> process to be completed. The waiting is performed once after device been
> created and first rule is being applied.
>
> Fixes: 95a464cecc21 ("net/mlx5: add E-switch VXLAN tunnel devices
> management")
>
> Signed-off-by: Viacheslav Ovsiienko <viacheslavo@mellanox.com>
Applied to next-net-mlx, thanks.
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-01-24 11:16 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-23 7:51 [dpdk-dev] [PATCH] net/mlx5: fix VXLAN port registration race condition Viacheslav Ovsiienko
2019-01-23 13:19 ` [dpdk-dev] [PATCH v2] " Viacheslav Ovsiienko
2019-01-24 11:16 ` Shahaf Shuler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).