From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id CC631A0093 for ; Fri, 11 Mar 2022 09:06:39 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id BF10140042; Fri, 11 Mar 2022 09:06:39 +0100 (CET) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2054.outbound.protection.outlook.com [40.107.93.54]) by mails.dpdk.org (Postfix) with ESMTP id 544254113F for ; Fri, 11 Mar 2022 09:06:38 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=b39MB0u06ywiaHTpSrU5VWTwgVwrempwPFyark/6gTdFlwshqri/27YYeI7XxMvE/c+9MSDKnXFySEAiDBIRfgD+Wuax+Rcq6zXANgmc/tpvLgDEqRyVZ7HX5CV+GNNX25sRxc+yuQAHnNdr9psIt9ekZH1tBn4xQaWd9PVg4Th7LV0sQ+qiGH37FfZjteaS7fL9tMJb36DUQ6W4jwxOOcim12RoFz3LNwe5lgGSb1rp8/KPrw5NNlEj3gptPBhVirrOwYcRweVUI4u6/1hYU9gXtcgUqJDyTqaKVMNQTnF1MEC6JYS1K7XLNEPJXJdOQDRPRiEJz2DWU+CVMZuzIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pyWHxAz5e9mZukOuc59b/+s1BjQ2wpjfcJ1PKt6ZkZU=; b=K64L0qFiEmd/4a74tmti5lFtDJKS9SZuM97zHHvKZ94oV62uzzlOIY0iQd/xzL75M6+9vIj9y7nWQtCH3Ko0+lH5T7x5pZFYqXKexz5tmMjiYwiWctgRDhpMrhuYHjzizRMWVAYsvsw9WoveF+4hSnNFBwZ+YYxH/3VbIvFW7lksLSdUOK4qgaVLo/3jc1OpZ8S/E9JlsWGrVBCOIDovNd6OXZmJkZUe0q47iRLvMuSWh4z5Mt+KK6W7FgDgQUYYJqwOr8Oph9n8VBZmAck4iQE0d34VjsTW+FCwDedH8uCv2B4ZzC2O7eoDT3ecpdOGGGF9rLqK4gln461Zd53iRQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.238) smtp.rcpttodomain=canonical.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pyWHxAz5e9mZukOuc59b/+s1BjQ2wpjfcJ1PKt6ZkZU=; b=cltwwwLBtOpfE6QnxlKKw6K5/iVxMDDpjQxv5iB1FKk8rg0Nr9AmslGM3yeOKVHU+wfYNX4jFutl1OcixO1Kz5VZVzMR4sAPgSD0oObIm6LYY8ubX2H/9pytwja8YKfT4fwLdDfIix/DVHhPON5lF47VzQtJZ77aFDwWpwDi/e2WpsVIrdQtHrXfDIEvuvd2SZRJSXRzZmOghEDw8PJeSXWgx4capQEp97bk5rSE5lr8U2ekpB5kPhfGbNmgT0X+cJzKsi2RuMFECl3Tb2yNxvylpCb+xPjP8bttZrtyZrFhuhv8uL5aACWDq6Q92lbivBe+JdKiu2/0CT/1mToSEg== Received: from MWHPR14CA0039.namprd14.prod.outlook.com (2603:10b6:300:12b::25) by MW2PR12MB2346.namprd12.prod.outlook.com (2603:10b6:907:4::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5061.22; Fri, 11 Mar 2022 08:06:35 +0000 Received: from CO1NAM11FT044.eop-nam11.prod.protection.outlook.com (2603:10b6:300:12b:cafe::92) by MWHPR14CA0039.outlook.office365.com (2603:10b6:300:12b::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5061.23 via Frontend Transport; Fri, 11 Mar 2022 08:06:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.238) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.238 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.238; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.238) by CO1NAM11FT044.mail.protection.outlook.com (10.13.175.188) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5061.22 via Frontend Transport; Fri, 11 Mar 2022 08:06:35 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by DRHQMAIL105.nvidia.com (10.27.9.14) with Microsoft SMTP Server (TLS) id 15.0.1497.32; Fri, 11 Mar 2022 08:06:34 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.22; Fri, 11 Mar 2022 00:06:33 -0800 Received: from nvidia.com (10.127.8.11) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server id 15.2.986.22 via Frontend Transport; Fri, 11 Mar 2022 00:06:32 -0800 From: Dmitry Kozlyuk To: CC: Christian Ehrhardt , "Viacheslav Ovsiienko" Subject: [PATCH 19.11 2/3] net/mlx5: fix link status change detection Date: Fri, 11 Mar 2022 10:06:26 +0200 Message-ID: <20220311080627.1195847-3-dkozlyuk@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220311080627.1195847-1-dkozlyuk@nvidia.com> References: <20220311080627.1195847-1-dkozlyuk@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: faf00fb4-1c6e-4657-c3bd-08da03360fbd X-MS-TrafficTypeDiagnostic: MW2PR12MB2346:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: syR+w4IZAsXCSwtB4ddxBHOK4HD2lzTRfszdedkf9zUbOOvpIvVIulysKXI7Jq9gT+s2Tr0hvCxnQc2kF7HzavCXVYVrEoiF+Zo/qj5J9m4R1jjsT/uYfI5CsJCM3Psepf7vnKoFte+JLhwITiIUHR8MmiJFOsr9UMmzxCl2xMLiKOPsY7mBYaVsG/evYqdvvDUf2XMZGxAjRMIL5MDmF/E0qs89e56QSiMMFnrm6vUqNNO3PGhAmmMa083z+XZbvZp7i6NgkkasPZafbhQbqMhBesvPvfRzzephtVVS/WUcy3xKMOvnmsUj31WdYTSFmDHplvYu05facwWYda44MliLbBogi2dxpEEtqZpl8n7HdZoEthrTfrE2CRtXeoGSSAxdslnQhY6oGgkm0oZ8tmuyElSIgyxjjaEkIeG4Grgb952E70hn9caQM53vr48hh+gv6LtgXbZ/zDLrUj/1YO9qvDxBf2GYPYSo7BIfHmgY8Z3U1Eg6nKsYIfxU0HRfqsLRyIQoK0IBa7AxGM/bfUcG6PdgWxBKO3sajaZZy/BFwTaIWwGzcgL6PrZ5QqE8LF0L5IBOca9qJXgLL2P6r0ifsijMmyxZIIWNCeHDJfQozDgoIgwHcpUlkV0arXWxJkTfZjp9ENrHA/KiCf9SJ/DWh1XsQ8npmV6TVk8LhD8/nkQf/tDp0tc3+WxjuWyj2OvqWNK/iHSqjwMqNdvOaQ== X-Forefront-Antispam-Report: CIP:12.22.5.238; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(46966006)(36840700001)(40470700004)(8676002)(70586007)(4326008)(70206006)(82310400004)(6666004)(86362001)(508600001)(47076005)(316002)(83380400001)(40460700003)(8936002)(5660300002)(36860700001)(2616005)(426003)(336012)(356005)(81166007)(186003)(7696005)(1076003)(26005)(54906003)(6916009)(55016003)(6286002)(2906002)(107886003)(36756003)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Mar 2022 08:06:35.2464 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: faf00fb4-1c6e-4657-c3bd-08da03360fbd X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.238]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT044.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW2PR12MB2346 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org [ upstream commit 17f95513adca1f4ee5bba305b154ac984ee50cee ] Sometimes net/mlx5 devices did not detect link status change to "up". Each shared device was monitoring IBV_EVENT_PORT_{ACTIVE,ERR} and queried the link status upon receiving the event. IBV_EVENT_PORT_ACTIVE is delivered when the logical link status (UP flag) is set, but the physical link status (RUNNING flag) may be down at that time, in which case the new link status would be errornously considered down. IBV interface is insufficient for the task. Monitor interface events using Netlink. Fixes: 198a3c339a8f ("mlx5: handle link status interrupts") Signed-off-by: Dmitry Kozlyuk Reviewed-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.c | 40 +++++++++++++++++++++ drivers/net/mlx5/mlx5.h | 3 ++ drivers/net/mlx5/mlx5_ethdev.c | 63 ++++++++++++++++++++++++++------- drivers/net/mlx5/mlx5_trigger.c | 12 +++++-- 4 files changed, 104 insertions(+), 14 deletions(-) diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index 27e04b5d9e..068b0ef981 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -470,6 +470,40 @@ mlx5_restore_doorbell_mapping_env(int value) setenv(MLX5_SHUT_UP_BF, value ? "1" : "0", 1); } +static int +mlx5_os_dev_shared_handler_install_lsc(struct mlx5_ibv_shared *sh) +{ + int nlsk_fd, flags, ret; + + nlsk_fd = mlx5_nl_init(NETLINK_ROUTE, RTMGRP_LINK); + if (nlsk_fd < 0) { + DRV_LOG(ERR, "Failed to create a socket for Netlink events: %s", + rte_strerror(rte_errno)); + return -1; + } + flags = fcntl(nlsk_fd, F_GETFL); + ret = fcntl(nlsk_fd, F_SETFL, flags | O_NONBLOCK); + if (ret != 0) { + DRV_LOG(ERR, "Failed to make Netlink event socket non-blocking: %s", + strerror(errno)); + rte_errno = errno; + goto error; + } + sh->intr_handle_nl.type = RTE_INTR_HANDLE_EXT; + sh->intr_handle_nl.fd = nlsk_fd; + if (rte_intr_callback_register(&sh->intr_handle_nl, + mlx5_dev_interrupt_handler_nl, + sh) != 0) { + DRV_LOG(ERR, "Failed to register Netlink events interrupt"); + sh->intr_handle_nl.fd = -1; + goto error; + } + return 0; +error: + close(nlsk_fd); + return -1; +} + /** * Install shared asynchronous device events handler. * This function is implemented to support event sharing @@ -499,6 +533,11 @@ mlx5_dev_shared_handler_install(struct mlx5_ibv_shared *sh) sh->intr_handle.fd = -1; } } + sh->intr_handle_nl.fd = -1; + if (mlx5_os_dev_shared_handler_install_lsc(sh) < 0) { + DRV_LOG(INFO, "Fail to install the shared Netlink event handler."); + sh->intr_handle_nl.fd = -1; + } if (sh->devx) { #ifdef HAVE_IBV_DEVX_ASYNC sh->intr_handle_devx.fd = -1; @@ -651,6 +690,7 @@ mlx5_alloc_shared_ibctx(const struct mlx5_dev_spawn_data *spawn, for (i = 0; i < sh->max_port; i++) { sh->port[i].ih_port_id = RTE_MAX_ETHPORTS; sh->port[i].devx_ih_port_id = RTE_MAX_ETHPORTS; + sh->port[i].nl_ih_port_id = RTE_MAX_ETHPORTS; } sh->pd = mlx5_glue->alloc_pd(sh->ctx); if (sh->pd == NULL) { diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index b74f1bbc83..1cf5f41689 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -575,6 +575,7 @@ struct mlx5_flow_counter_mng { struct mlx5_ibv_shared_port { uint32_t ih_port_id; uint32_t devx_ih_port_id; + uint32_t nl_ih_port_id; /* * Interrupt handler port_id. Used by shared interrupt * handler to find the corresponding rte_eth device @@ -693,6 +694,7 @@ struct mlx5_ibv_shared { /* Shared interrupt handler section. */ struct rte_intr_handle intr_handle; /* Interrupt handler for device. */ struct rte_intr_handle intr_handle_devx; /* DEVX interrupt handler. */ + struct rte_intr_handle intr_handle_nl; /* Netlink interrupt handler. */ struct mlx5dv_devx_cmd_comp *devx_comp; /* DEVX async comp obj. */ struct mlx5_devx_obj *tis; /* TIS object. */ struct mlx5_devx_obj *td; /* Transport domain. */ @@ -841,6 +843,7 @@ int mlx5_dev_to_pci_addr(const char *dev_path, void mlx5_dev_link_status_handler(void *arg); void mlx5_dev_interrupt_handler(void *arg); void mlx5_dev_interrupt_handler_devx(void *arg); +void mlx5_dev_interrupt_handler_nl(void *arg); void mlx5_dev_interrupt_handler_uninstall(struct rte_eth_dev *dev); void mlx5_dev_interrupt_handler_install(struct rte_eth_dev *dev); int mlx5_set_link_down(struct rte_eth_dev *dev); diff --git a/drivers/net/mlx5/mlx5_ethdev.c b/drivers/net/mlx5/mlx5_ethdev.c index 7af7fb0f9f..5334377c10 100644 --- a/drivers/net/mlx5/mlx5_ethdev.c +++ b/drivers/net/mlx5/mlx5_ethdev.c @@ -1301,6 +1301,57 @@ mlx5_dev_interrupt_device_fatal(struct mlx5_ibv_shared *sh) } } +static void +mlx5_dev_interrupt_nl_cb(struct nlmsghdr *hdr, void *cb_arg) +{ + struct mlx5_ibv_shared *sh = cb_arg; + uint32_t i; + uint32_t if_index; + + if (mlx5_nl_parse_link_status_update(hdr, &if_index) < 0) + return; + for (i = 0; i < sh->max_port; i++) { + struct mlx5_ibv_shared_port *port = &sh->port[i]; + struct rte_eth_dev *dev; + struct mlx5_priv *priv; + bool configured; + + if (port->nl_ih_port_id >= RTE_MAX_ETHPORTS) + continue; + dev = &rte_eth_devices[port->nl_ih_port_id]; + configured = dev->process_private != NULL; + /* Probing may initiate an LSC before configuration is done. */ + if (configured && !dev->data->dev_conf.intr_conf.lsc) + break; + priv = dev->data->dev_private; + if (priv->if_index == if_index) { + /* Block logical LSC events. */ + uint16_t prev_status = dev->data->dev_link.link_status; + + if (mlx5_link_update(dev, 0) < 0) + DRV_LOG(ERR, "Failed to update link status: %s", + rte_strerror(rte_errno)); + else if (prev_status != dev->data->dev_link.link_status) + _rte_eth_dev_callback_process + (dev, RTE_ETH_EVENT_INTR_LSC, NULL); + break; + } + } +} + +void +mlx5_dev_interrupt_handler_nl(void *arg) +{ + struct mlx5_ibv_shared *sh = arg; + int nlsk_fd = sh->intr_handle_nl.fd; + + if (nlsk_fd < 0) + return; + if (mlx5_nl_read_events(nlsk_fd, mlx5_dev_interrupt_nl_cb, sh) < 0) + DRV_LOG(ERR, "Failed to process Netlink events: %s", + rte_strerror(rte_errno)); +} + /** * Handle shared asynchronous events the NIC (removal event * and link status change). Supports multiport IB device. @@ -1364,18 +1415,6 @@ mlx5_dev_interrupt_handler(void *cb_arg) tmp = sh->port[tmp - 1].ih_port_id; dev = &rte_eth_devices[tmp]; assert(dev); - if ((event.event_type == IBV_EVENT_PORT_ACTIVE || - event.event_type == IBV_EVENT_PORT_ERR) && - dev->data->dev_conf.intr_conf.lsc) { - mlx5_glue->ack_async_event(&event); - if (mlx5_link_update(dev, 0) == -EAGAIN) { - usleep(0); - continue; - } - _rte_eth_dev_callback_process - (dev, RTE_ETH_EVENT_INTR_LSC, NULL); - continue; - } DRV_LOG(DEBUG, "port %u cannot handle an unknown event (type %d)", dev->data->port_id, event.event_type); diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index 04b06e11d7..e91970ddae 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -331,11 +331,18 @@ mlx5_dev_start(struct rte_eth_dev *dev) priv->sh->port[priv->ibv_port - 1].ih_port_id = (uint32_t)dev->data->port_id; } else { - DRV_LOG(INFO, "port %u starts without LSC and RMV interrupts.", + DRV_LOG(INFO, "port %u starts without RMV interrupts.", dev->data->port_id); - dev->data->dev_conf.intr_conf.lsc = 0; dev->data->dev_conf.intr_conf.rmv = 0; } + if (priv->sh->intr_handle_nl.fd >= 0) { + priv->sh->port[priv->ibv_port - 1].nl_ih_port_id = + (uint32_t)dev->data->port_id; + } else { + DRV_LOG(INFO, "port %u starts without LSC interrupts.", + dev->data->port_id); + dev->data->dev_conf.intr_conf.lsc = 0; + } if (priv->sh->intr_handle_devx.fd >= 0) priv->sh->port[priv->ibv_port - 1].devx_ih_port_id = (uint32_t)dev->data->port_id; @@ -379,6 +386,7 @@ mlx5_dev_stop(struct rte_eth_dev *dev) mlx5_rx_intr_vec_disable(dev); priv->sh->port[priv->ibv_port - 1].ih_port_id = RTE_MAX_ETHPORTS; priv->sh->port[priv->ibv_port - 1].devx_ih_port_id = RTE_MAX_ETHPORTS; + priv->sh->port[priv->ibv_port - 1].nl_ih_port_id = RTE_MAX_ETHPORTS; mlx5_txq_stop(dev); mlx5_rxq_stop(dev); } -- 2.25.1