From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0076.outbound.protection.outlook.com [104.47.1.76]) by dpdk.org (Postfix) with ESMTP id BE298AAB9 for ; Tue, 5 Jun 2018 02:12:00 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wNRK80aWpSzalVZIU1CIa74EwUTQhVwIdHkyEgbiYsE=; b=LiPE9kQIkfI96cWNCideHRzom190YSu8aGbBMewO1mkGu0K+VaKVdpn/Lfy93YN3HDxZUP8lnAXtGfLpmps8CYZpcuIOzKmouIAIRogKD6agdtzRRT6g5xqjsAGMmOmMiV1LAu3KllPyjpE/CrGUkodMR1Dk6PuIv42ouRyUvP0= Received: from mellanox.com (209.116.155.178) by AM5PR0501MB2036.eurprd05.prod.outlook.com (2603:10a6:203:1a::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.820.11; Tue, 5 Jun 2018 00:11:58 +0000 From: Yongseok Koh To: yliu@fridaylinux.org Cc: stable@dpdk.org, shahafs@mellanox.com, adrien.mazarguil@6wind.com, nelio.laranjeiro@6wind.com, Olivier Matz Date: Mon, 4 Jun 2018 17:10:26 -0700 Message-Id: <20180605001129.13184-5-yskoh@mellanox.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180605001129.13184-1-yskoh@mellanox.com> References: <20180605001129.13184-1-yskoh@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [209.116.155.178] X-ClientProxiedBy: CO1PR15CA0060.namprd15.prod.outlook.com (2603:10b6:101:1f::28) To AM5PR0501MB2036.eurprd05.prod.outlook.com (2603:10a6:203:1a::22) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:AM5PR0501MB2036; X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 3:SB1xY7xwayQxp0VPQEUxXKxXLRMZaCMu+TSZvsDb7F055EnZzi3T6Dkp0W5wmC9QjFbm0Y2j2RFPwcLP+/wYApzaxeh5GFi6JzE9fhYXG0QwMUdI9ODCtoxoBZgplLz/Q74XHWDViPTw2CmEe4QEuUOHgChIC540VhTrdBgg+GhhV+Zn1thOfhXoyk80tNrif7St8W8UV7IjReVK0NPr397eH/yQ5fUVTv8sa+PxojPO+9KI6EbMPJJDJ3UiSXdp; 25:L4GS+GcqGGO85fIKl/oBwwLfnRQYvsS+oZXxpgiVAk27KS2HhVEQxBEb4tOmeWUvXLMuaNHq/op4RxW/bHpjyDA5H7Aclc7nqpKdbv2W6voJKQqx0TP8RdeKYxtKgImuKI2YfP71BlIF3vApBUxByMsN0RK8OET0ACcr+oAddQnzldRzZCDt7TQ+uQz1WT0IJsxpE8RGZa+yG8/g0h5zOlrbNRnwQcKsvvQ7EZGaNAAUXLEdrvIB3zcDnQmMw4EkbKOEYgJd0TMn8+AfnHXIvk/IRDqwUGTPi++MFzpR5gmUvR0dyH0DmXoTdxqTTnb7+l7PGApFrZTCrUdoz9Aa4w==; 31:/TPbfFifhlaCr0chgPe5ZCqNLeDKcI1aNTUGS3T2sFAgnFFlzFHymAzEV+03s2CMZbjDNyOHxyjCU7yMbelCx6FDmSAdl2K9I0GwIPFbFKaTXBFat7ucTpaBs8Ai0pB2AbEDG7M4pigDMUOrrBmAdDJkUbR/qUm1OX3IRG9RtykXwbq4SVDT1yr5SVtO/Zey+DMmt22fHnNXE4np2MkayLTGsBBJ+DXz92nN4VzxQCM= X-MS-TrafficTypeDiagnostic: AM5PR0501MB2036: Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 20:fAe+oSBfy2in7Vv0FuSBDPXvQdW+KeYJWQaxzVgL2UnyR1S/XkzjL92YrEZMzxrF1SnHiVLTi71RNls/FZmlJXln8WjR6DMwNb1IqSsVv23op6ApP6jsZAymiEOXSS79HG8jxz/AXBZlGlmJAuRHKNdQVd9H9pTpTaOUyy8QDG9h8ETomlynLZGpkLimL/QAw/PS9vOGf6Pwa809x89J6RgtG+fogpw49dqwDX3XY+CHeZAyLN+ULCVVCL+Zn7zBinu9O51Pu8S6ACHFX/XM4KiMBxdH7dwoNCCDDgM0xe0sxUtv+WQrmZGRdwKhSS6m6xg+yPIyi5DTsMJnq+DR9qPz97e6ka77o78Co7X2imRF4LEIfKWjjphmgtqRxV+lNTTNhytmbRkXM7Cynw/7Mf8MMyFusdz4hTN/mDfDSo9YSIvjKZJPQJ9uXnXXjRlpr+hKzgRwSfZAElayxgSVoc+HnqYti06ElL7uyRMDmkEBUJeBC0I4ELuI8ywJxCAl; 4:kbrTTkV6ljE0x9PHsDXrS1BIEYoAZUzyXCgNhjgCU0RVfbhD7Aulo2O6xvzsYLi2pvg5CA94JiCAyWOUr1Q+r1PA7RonBfu+Tw+4LYq2+GzvNVthqG3UBEPRd7brkrJrf4BtkivR70B4ZY8RBWTVcxtCDTKm2mq7rPg2TiUOVjkTRlQrYf+drUqIqQraN3MM5i1+zzPa1XciVVPbZ1yLYSLaZyR9Tw868qVcyCcIEucspMvVbex6tS8MvMcro/WlspcUZeEmvuAwVc+3W+qLxg== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-MS-Exchange-SenderADCheck: 1 X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(3231254)(944501410)(52105095)(10201501046)(3002001)(6055026)(149027)(150027)(6041310)(20161123560045)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(20161123558120)(6072148)(201708071742011)(7699016); SRVR:AM5PR0501MB2036; BCL:0; PCL:0; RULEID:; SRVR:AM5PR0501MB2036; X-Forefront-PRVS: 0694C54398 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(39380400002)(366004)(396003)(39860400002)(346002)(376002)(189003)(199004)(478600001)(81156014)(76176011)(51416003)(7696005)(956004)(2361001)(50226002)(305945005)(2616005)(50466002)(47776003)(486006)(97736004)(5660300001)(6666003)(2351001)(106356001)(8936002)(186003)(81166006)(52116002)(6116002)(476003)(6916009)(386003)(446003)(11346002)(59450400001)(66066001)(26005)(3846002)(8676002)(16526019)(1076002)(25786009)(21086003)(86362001)(575784001)(53936002)(4326008)(2906002)(316002)(55016002)(36756003)(48376002)(105586002)(69596002)(68736007)(7736002)(16586007); DIR:OUT; SFP:1101; SCL:1; SRVR:AM5PR0501MB2036; H:mellanox.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; AM5PR0501MB2036; 23:haqyC+Kt2fR83ZwHx8i6w+vZ1o4EOx58x6H0vTz?= =?us-ascii?Q?/RzY4oqL8iscqiaUi36ZJs+ilwlPWHc7MKLNodRjK7eKuQrzobtFre8dp/cW?= =?us-ascii?Q?rbeVPFaY+cXV8v4YgKiIdk0NpS5SFz0fp5iUfQFYD3ovyrjITiCeSaGDNw+Y?= =?us-ascii?Q?ZBSTXRi2atVvMnXlmu3D8/Qd6DOhn3DhO1DW+2SRDu3mD5+bxaadWZHqzCVP?= =?us-ascii?Q?U+9MYMzE8Dv1KB0+dnk/HQgAa8MebNmrnmkYrkjVr2sfbzqezn3FRl/nuDy9?= =?us-ascii?Q?UKRpqLotzB1i1hbUN2tApsLrHvfNVKoiU0a/bkEIZ30bZlmnVvTYpEvsxDqh?= =?us-ascii?Q?SQA2F5Mq1JmFTEhfCKS41J61iHo8rxOBpHr9SCIaaNga7lzaVXZx9+a5DAT8?= =?us-ascii?Q?CjQHrEssDDXUBmZJm8XEcm6d5c3oSr07O2JTdqv2FgCsh0jsCt6qp7KJHv/I?= =?us-ascii?Q?LRdFv0sqWF64ewC8e2V822UEap/R18SvFD8wSXi6RyV0YMKzkMY2xgg5rCqP?= =?us-ascii?Q?0BoJuRCl0qpJWf6sHnObMUsYFlgSfWHDvDB+WhPaBG6hduiETTOH+q4iM4De?= =?us-ascii?Q?l3m3EJB8z2uXWuWohc7zbhu5rIGv0OQP7lh358dRfyZNaelojki4B2FC9b70?= =?us-ascii?Q?cxTxiRyMJMORodM/a+XFdjiyrN7nt7inRcyEJTIJL9gGo4zcNiwA+Q0FBu9I?= =?us-ascii?Q?60oQg3P3wVoVQ4He06qePGj4/w+dscHmQ/PofGzhlh3iSk9rNX+YYYGeP6Td?= =?us-ascii?Q?cCx8ZPoJ8GsxOsFnxvU0h1bP46FdUyztenn1vvuVyY65Ppn509NEII1wpPyU?= =?us-ascii?Q?GElbia4mFw9bNDrF0dYWuQM/6wooRqmWrB61m5Y028p2K3XWZ0eiVTElf8qW?= =?us-ascii?Q?ukro/7rBXJ9RQ2/a+3J9NpDPcz1X33jIqN+LuE6+ychTZUamlLQdMlbbGASg?= =?us-ascii?Q?brZ4vzZykxiGDhPcX9+UvwBnjK0VjbuXXDAv6TARnfkLLlN8Ujzl9D4Q6CyS?= =?us-ascii?Q?Ytzt7BfTM2XEeIrZC66bAZ6TO5VTBsy1qFcFAOZABysjlqO6IzARSPmBDuwc?= =?us-ascii?Q?i6WqHgoYujH9BosmGjCZMOvY8A/BCG4Y251p3w9S+Cb9pHETuZzVg2XwH5I6?= =?us-ascii?Q?qBgbWLjNwqSSU/V44iCUpsUASc2nSNUt7C64zkDg5W5wQs20hdsxjV2OXw2I?= =?us-ascii?Q?ekpOOUKAaKMrfDRJjthbTfUr6lCVYhZqhVNBqY95QROUaR/d9GZR8H0RMsZg?= =?us-ascii?Q?xDveI6K6wgsFJQDOtQDBEBFpFcJqF0TvwwNVuXZcw43B2jeAAhfszoDN+G2J?= =?us-ascii?Q?VmH9euFyriKVpErKXqcWzLOg=3D?= X-Microsoft-Antispam-Message-Info: x0EChGxjqfFqg5N/EniwQOS36ZLXNuHh4V7Ks9qs5RPLnIxM6rs1xFeacHDOBbDJZpZISQM9UfrSqZo9CUfQdGwXpTv/jv2o5Cl7cF3zh39RaddUYKM3mokNTXgSIV99AK+kTzch/qtgZFzam6WqxBiF2V41Iz88mlbfO6iY+HRse880J3/UGqy7FfKSZfKT X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 6:V4lA3vRrVO+upolBfFTAPawzbpPbU6zJC0XKZjvfJO4dKUN5AzUanYuJXDiyq+cGUjo4r5a9YCbU5v14DMhzHBLcmtmwtz8s1k9774qJ9cQ3CVqbcc3npszw/cVi5jWpwOzwpMf48KcbHeg8vX4Ffnvl2Vv0xfR5QAZuqgIggYSmZ9YQ1akbfuLqAQsB96ciBJA/4D7WDY+/Ud+B/z2SB/Kl8YjIdGMltTZmRXHt6t4aF+0jXPi2MxOA2p7tsHFlBsVn9qdILZ9Qjx+o14qE56Fj73CFp9P2LiLRzqWJg4yica7y1zMYDIMJJQN66mjryMxq+5MAdkjWWtWtT5/Wgs0ppik1EhvTclU7VVFk4qpWUfr29sn1ibEC32GXWDt2LvK+b45eZtZ9NIcgIZZNkMqS2uQg45WBJOshDpxHfkcCqVc59ihLJVNasB7irNQAmRz4Xw6PEPTwyBSsllURnQ==; 5:U8E7xIof6OUHoUNEI7NyqyI8kAAxvkfqWp3G2bmo0Ky16uXh+rx21jrvKYGoMHXzePbKX+nN0d2fJYN59EduKTZ6ge0ds5MTCgtml9p6sWmDvG4NlSqdoed1OToJAGAkb0hfxRfEJwSbPxaHmPw66xDDr15wHi6uu/lmF20ot+0=; 24:qNm27XDhfSIvKTABb/jtuLvGcuBd6vlghmAvncd+5NSljTCt6ofgBMiSt/JCAzzQR0wfNGEGVAQU8nmdKmoRwbQmOoYtnu18vWUaL3+l22c= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; AM5PR0501MB2036; 7:X+W9TB9N/VcV8fnq7rpeGk2YnOol4LeUcdZh+dyl9cDRk6eOZUI07DO50Z/XQ9pAzyM+NrFvwnW3Z/kC1x+1RR4HQXUVCXkP/B8c/jCODRu8mLw0tA+We/k3eWerSicnME0bdWJFq9XdvA0c4mbw0jJMeAah1y+6HNfpVMzYO88tg++FoTUlo8njEclAB4TaPmKqpOeBebxNnbuAwFQsvstYu6Gl7RWqb33ls3BgZya26eG+Rhlw6eOuDTjHefh1 X-MS-Office365-Filtering-Correlation-Id: 1fe80fe8-224f-4092-d4ab-08d5ca78f4dd X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jun 2018 00:11:58.0388 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1fe80fe8-224f-4092-d4ab-08d5ca78f4dd X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5PR0501MB2036 Subject: [dpdk-stable] [PATCH 04/67] net/mlx5: fix allocation when no memory on device NUMA node X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jun 2018 00:12:01 -0000 From: Olivier Matz [ backported from upstream commit d10b09db0a455a27f1f1fa5e4a0fa7b1350706a5 ] When no memory is available on the same numa node than the device, the initialization of the device fails. However, the use case where the cores and memory are on a different socket than the device is valid, even if not optimal. To fix this issue, this commit introduces an infrastructure to select the socket on which to allocate the verbs objects based on the ethdev configuration and the object type, rather than the PCI numa node. Fixes: 1e3a39f72d5d ("net/mlx5: allocate verbs object into shared memory") Cc: stable@dpdk.org Signed-off-by: Olivier Matz Signed-off-by: Nelio Laranjeiro --- drivers/net/mlx5/mlx5.c | 14 ++++++++++++-- drivers/net/mlx5/mlx5.h | 20 ++++++++++++++++++++ drivers/net/mlx5/mlx5_rxq.c | 4 ++++ drivers/net/mlx5/mlx5_txq.c | 4 ++++ 4 files changed, 40 insertions(+), 2 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 7f44bb169..16ba5b724 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -156,10 +156,20 @@ mlx5_alloc_verbs_buf(size_t size, void *data) struct priv *priv = data; void *ret; size_t alignment = sysconf(_SC_PAGESIZE); + unsigned int socket = SOCKET_ID_ANY; + if (priv->verbs_alloc_ctx.type == MLX5_VERBS_ALLOC_TYPE_TX_QUEUE) { + const struct mlx5_txq_ctrl *ctrl = priv->verbs_alloc_ctx.obj; + + socket = ctrl->socket; + } else if (priv->verbs_alloc_ctx.type == + MLX5_VERBS_ALLOC_TYPE_RX_QUEUE) { + const struct mlx5_rxq_ctrl *ctrl = priv->verbs_alloc_ctx.obj; + + socket = ctrl->socket; + } assert(data != NULL); - ret = rte_malloc_socket(__func__, size, alignment, - priv->dev->device->numa_node); + ret = rte_malloc_socket(__func__, size, alignment, socket); DEBUG("Extern alloc size: %lu, align: %lu: %p", size, alignment, ret); return ret; } diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index d49595bc9..004e83670 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -90,6 +90,24 @@ struct mlx5_xstats_ctrl { /* Flow list . */ TAILQ_HEAD(mlx5_flows, rte_flow); +/** + * Type of objet being allocated. + */ +enum mlx5_verbs_alloc_type { + MLX5_VERBS_ALLOC_TYPE_NONE, + MLX5_VERBS_ALLOC_TYPE_TX_QUEUE, + MLX5_VERBS_ALLOC_TYPE_RX_QUEUE, +}; + +/** + * Verbs allocator needs a context to know in the callback which kind of + * resources it is allocating. + */ +struct mlx5_verbs_alloc_ctx { + enum mlx5_verbs_alloc_type type; /* Kind of object being allocated. */ + const void *obj; /* Pointer to the DPDK object. */ +}; + struct priv { struct rte_eth_dev *dev; /* Ethernet device of master process. */ struct ibv_context *ctx; /* Verbs context. */ @@ -149,6 +167,8 @@ struct priv { rte_spinlock_t lock; /* Lock for control functions. */ int primary_socket; /* Unix socket for primary process. */ struct rte_intr_handle intr_handle_socket; /* Interrupt handler. */ + struct mlx5_verbs_alloc_ctx verbs_alloc_ctx; + /* Context for Verbs allocator. */ }; /** diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index a4cdd374a..457f919b4 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -573,6 +573,8 @@ mlx5_priv_rxq_ibv_new(struct priv *priv, uint16_t idx) assert(rxq_data); assert(!rxq_ctrl->ibv); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_RX_QUEUE; + priv->verbs_alloc_ctx.obj = rxq_ctrl; tmpl = rte_calloc_socket(__func__, 1, sizeof(*tmpl), 0, rxq_ctrl->socket); if (!tmpl) { @@ -736,6 +738,7 @@ mlx5_priv_rxq_ibv_new(struct priv *priv, uint16_t idx) DEBUG("%p: Verbs Rx queue %p: refcnt %d", (void *)priv, (void *)tmpl, rte_atomic32_read(&tmpl->refcnt)); LIST_INSERT_HEAD(&priv->rxqsibv, tmpl, next); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; return tmpl; error: if (tmpl->wq) @@ -746,6 +749,7 @@ mlx5_priv_rxq_ibv_new(struct priv *priv, uint16_t idx) claim_zero(ibv_destroy_comp_channel(tmpl->channel)); if (tmpl->mr) priv_mr_release(priv, tmpl->mr); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; return NULL; } diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c index 7ca99f5aa..12f258bbb 100644 --- a/drivers/net/mlx5/mlx5_txq.c +++ b/drivers/net/mlx5/mlx5_txq.c @@ -307,6 +307,8 @@ mlx5_priv_txq_ibv_new(struct priv *priv, uint16_t idx) int ret = 0; assert(txq_data); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_TX_QUEUE; + priv->verbs_alloc_ctx.obj = txq_ctrl; if (mlx5_getenv_int("MLX5_ENABLE_CQE_COMPRESSION")) { ERROR("MLX5_ENABLE_CQE_COMPRESSION must never be set"); goto error; @@ -435,12 +437,14 @@ mlx5_priv_txq_ibv_new(struct priv *priv, uint16_t idx) DEBUG("%p: Verbs Tx queue %p: refcnt %d", (void *)priv, (void *)txq_ibv, rte_atomic32_read(&txq_ibv->refcnt)); LIST_INSERT_HEAD(&priv->txqsibv, txq_ibv, next); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; return txq_ibv; error: if (tmpl.cq) claim_zero(ibv_destroy_cq(tmpl.cq)); if (tmpl.qp) claim_zero(ibv_destroy_qp(tmpl.qp)); + priv->verbs_alloc_ctx.type = MLX5_VERBS_ALLOC_TYPE_NONE; return NULL; } -- 2.11.0