From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DDF20A034F; Tue, 1 Mar 2022 21:26:33 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id B2FAB426E0; Tue, 1 Mar 2022 21:26:30 +0100 (CET) Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2057.outbound.protection.outlook.com [40.107.93.57]) by mails.dpdk.org (Postfix) with ESMTP id 30CC1426DE for ; Tue, 1 Mar 2022 21:26:29 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BfvJ+KXKA8j9lRovFcBCJsbhgku9gkdlYcZn00Bvtq8VlIYFJThBCrfgz5HXcqWumUHP4Ib7ujFdlq8HL0DNeZDSuC8brjpWNx2qT2uGbWKqaWdxKrFzzv/rvZcJFUH4s9B/xnYQeTysjbJCN6wZqG7m4wSovTgfp1RPqvOeCEVbmv6txdM6u5L+hZaM+uaNoZwVKvzhC/aNnsi0t4uVaR/nymvFG+u9adUYTfUmU6w8X6ICIzBouNNFCnmv1pJ/VFjZG2fWAyXHDFdos4775KxseQxkCVG5i/g+53LjkZCaPG/HdZGNBYPM/yk+n89t9DF/zX0PZlyX6SnMZKRY3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ij778+IJtyVZQ4t9U1zPQFk8NKwlbOpL6QSERGqFafA=; b=eumvRZgjfjjo3zD70EF6L3gP2P6sWETHiPuzYrZ65nJIIPE6vN85AtoXgufCrPw38lS/lQ95yN8wuIi+N7SplyF+rqocc+NaDYvBx0eAwPcDSoZ54H9JMW1qdkD/3rmrMvGiZwTFbAWMJgyvZ4HyhtgGbbjPLJSMwdO0clukPV8qH0arI/Et6OuQYsFckWNrtNbmBsmHphpXzASD6BPDruQFX0w9VCJnOQsUhsDrT1op17P5Y0HHrhbj7bPru2MkTiH7Ll0P8H9C+Z9nzmc8++rJurvvkUjDuQ0wREGQx3CsXwjEz6T8Ug7Rkwx6BfZFDShn7PqFUYxHHT4GcuZJxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.236) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ij778+IJtyVZQ4t9U1zPQFk8NKwlbOpL6QSERGqFafA=; b=tuJ8/b7jHLIbq66Q9eNf9IhgF+JQyd8LeiKFsSIDetuOnQSQyKRM/Wr5QbbVWYSmNtPSmmKfyB0P1fvDFDiNrNrpSI+o39w3okR3lb703vgmEbbqYPN40L+DSZUGvdbPOgS4vezhze7reLzrIETR8ILcgsMvoimrKSTZ6b34jrdIiwMAXck38jVRToMXaNb79lGZvfSCcInVFLwuzGH7KoCmnnXxicvAPFLOEZzZ6A4DNGlwIbmvuulYWoVYPN+g4oa9Lgr5+TBYn1WUJgdujgw0EqqGEt0c3y8Vpy51SpT84kuzgRuItp+/0OBTBnHnbasaTnUdmt+WmvgCZe8vuA== Received: from BN9PR03CA0232.namprd03.prod.outlook.com (2603:10b6:408:f8::27) by BN6PR1201MB0036.namprd12.prod.outlook.com (2603:10b6:405:4e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Tue, 1 Mar 2022 20:26:26 +0000 Received: from BN8NAM11FT058.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f8:cafe::b0) by BN9PR03CA0232.outlook.office365.com (2603:10b6:408:f8::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.23 via Frontend Transport; Tue, 1 Mar 2022 20:26:26 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.236) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.236 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.236; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.236) by BN8NAM11FT058.mail.protection.outlook.com (10.13.177.58) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.5038.14 via Frontend Transport; Tue, 1 Mar 2022 20:26:25 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by DRHQMAIL109.nvidia.com (10.27.9.19) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 1 Mar 2022 20:26:24 +0000 Received: from rnnvmail203.nvidia.com (10.129.68.9) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Tue, 1 Mar 2022 12:26:23 -0800 Received: from nvidia.com (10.127.8.13) by mail.nvidia.com (10.129.68.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9 via Frontend Transport; Tue, 1 Mar 2022 12:26:22 -0800 From: Michael Baum To: CC: Xiaoyun Li , Aman Singh , Yuying Zhang , Matan Azrad Subject: [PATCH 1/2] app/testpmd: add test for remote PD and CTX Date: Tue, 1 Mar 2022 22:26:14 +0200 Message-ID: <20220301202615.4103972-2-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220301202615.4103972-1-michaelba@nvidia.com> References: <20220301202615.4103972-1-michaelba@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3246a214-fdcb-4fa8-0227-08d9fbc1c265 X-MS-TrafficTypeDiagnostic: BN6PR1201MB0036:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0Mt5RNxUl4iG/G0eVKlIJwWuKAOFhX1OuNmw+FSXUlw/e1hoorPkCzg2miVZYuhOGdKeXxq15/yLnBotTAmErm/5SpNwwlIYMMvFoYPb7xiHfSwiRHChRWht61f6JBx/dSa9p02OD+hToqKCI3v316NBuFlvYQIKURsK9WYyWSajrWg+PU3xq2moNzjAJ1ZUrY2jQszha+Al/rt0IHrylKOv9caMcZgUE7lj6u6BIm//M0Igkwmffh94vr+orZHU5rb9X7G/9NFCnrNS0AVkobGkxTQhlVmZNlHkrR//0cNVxzNN+O/czAOBLN9nQc94dmZ0qTWhp0wxTWTMU3SxpAE8W5+p+yfpgJ6mE/lE1IlxbwAHNTmsF/L7lqwzC6qjUcC9jnX7rMqeYpQECVctBCVQ2oT39JA2If5NPvow3MZoCLbqjwHBi3flhYStUEQvEXmCu9PIbvnjJBMNwPnblpEmXwJZRXzf1J9WQt0piyuGxP653AoUzkO7ZBZUgPYKJyD/ct4tYvG5suTp19jt6Boh8hjCgam40cNwetonhqdJOHs+a/weU0DodlbK5ZnhTIerF5Ru4K4e9s7A1oKuQ+m1eA0DagHSOOU9vLeL3hjIWoc+96Ms3ErJI9khCokad90CUzbQy/mrSuxfyB7sLfKaLqowbFNzfl6sT6TB6iXjd872CRs29VYNFDZrHOXwTSoYOh9GK2WyPGZnOb0WYA== X-Forefront-Antispam-Report: CIP:12.22.5.236; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(83380400001)(4326008)(40460700003)(47076005)(70206006)(70586007)(8676002)(6916009)(356005)(54906003)(36756003)(82310400004)(508600001)(316002)(55016003)(81166007)(107886003)(26005)(186003)(336012)(2906002)(6666004)(7696005)(426003)(2616005)(86362001)(5660300002)(6286002)(1076003)(8936002)(36860700001)(36900700001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Mar 2022 20:26:25.5795 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3246a214-fdcb-4fa8-0227-08d9fbc1c265 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.236]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT058.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR1201MB0036 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add mlx5 internal option in testpmd run-time function "port attach" to add another parameter named "mlx5_socket" for attaching port and add 2 devargs before. The arguments are "cmd_fd" and "pd_handle" using to import device created out of PMD. Testpmd application import it using IPC, and updates the devargs list before attaching. The syntax is: testpmd > port attach (identifier) mlx5_socket=(path) Where "path" is the IPC socket path agreed on the remote process. Signed-off-by: Michael Baum Acked-by: Matan Azrad --- app/test-pmd/cmdline.c | 14 +- app/test-pmd/meson.build | 3 + app/test-pmd/testpmd.c | 153 ++++++++++++++++++++ doc/guides/testpmd_app_ug/testpmd_funcs.rst | 40 +++++ 4 files changed, 208 insertions(+), 2 deletions(-) diff --git a/app/test-pmd/cmdline.c b/app/test-pmd/cmdline.c index 7ab0575e64..479e0290c4 100644 --- a/app/test-pmd/cmdline.c +++ b/app/test-pmd/cmdline.c @@ -773,6 +773,12 @@ static void cmd_help_long_parsed(void *parsed_result, "port attach (ident)\n" " Attach physical or virtual dev by pci address or virtual device name\n\n" +#ifdef RTE_NET_MLX5 + "port attach (ident) mlx5_socket=(path)\n" + " Attach physical or virtual dev by pci address or virtual device name " + "and add \"cmd_fd\" and \"pd_handle\" devargs before attaching\n\n" +#endif + "port detach (port_id)\n" " Detach physical or virtual dev by port_id\n\n" @@ -1379,8 +1385,12 @@ cmdline_parse_token_string_t cmd_operate_attach_port_identifier = cmdline_parse_inst_t cmd_operate_attach_port = { .f = cmd_operate_attach_port_parsed, .data = NULL, - .help_str = "port attach : " - "(identifier: pci address or virtual dev name)", + .help_str = "port attach mlx5_socket=: " + "(identifier: pci address or virtual dev name" +#ifdef RTE_NET_MLX5 + ", path (optional): socket path to get cmd FD and PD handle" +#endif + ")", .tokens = { (void *)&cmd_operate_attach_port_port, (void *)&cmd_operate_attach_port_keyword, diff --git a/app/test-pmd/meson.build b/app/test-pmd/meson.build index 43130c8856..c4fd379e67 100644 --- a/app/test-pmd/meson.build +++ b/app/test-pmd/meson.build @@ -73,3 +73,6 @@ endif if dpdk_conf.has('RTE_NET_DPAA') deps += ['bus_dpaa', 'mempool_dpaa', 'net_dpaa'] endif +if dpdk_conf.has('RTE_NET_MLX5') + deps += 'net_mlx5' +endif diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index fe2ce19f99..7ec95e5ae4 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -11,6 +11,10 @@ #include #ifndef RTE_EXEC_ENV_WINDOWS #include +#ifdef RTE_NET_MLX5 +#include +#include +#endif #endif #include #include @@ -3200,11 +3204,150 @@ reset_port(portid_t pid) printf("Done\n"); } +#if defined(RTE_NET_MLX5) && !defined(RTE_EXEC_ENV_WINDOWS) +static const char* +get_socket_path(char *extend) +{ + if (strstr(extend, "mlx5_socket=") == extend) { + const char *socket_path = strchr(extend, '=') + 1; + + TESTPMD_LOG(DEBUG, "MLX5 socket path is %s\n", socket_path); + return socket_path; + } + + TESTPMD_LOG(ERR, "Failed to extract a valid socket path from %s\n", + extend); + return NULL; +} + +static int +attach_port_extend_devargs(char *identifier, char *extend) +{ + struct sockaddr_un un = { + .sun_family = AF_UNIX, + }; + struct sockaddr_un dst = { + .sun_family = AF_UNIX, + }; + int cmd_fd; + int pd_handle; + struct iovec iov = { + .iov_base = &pd_handle, + .iov_len = sizeof(int), + }; + union { + char buf[CMSG_SPACE(sizeof(int))]; + struct cmsghdr align; + } control; + struct msghdr msgh = { + .msg_name = &dst, + .msg_namelen = sizeof(dst), + .msg_iov = NULL, + .msg_iovlen = 0, + }; + struct cmsghdr *cmsg; + const char *path = get_socket_path(extend + 1); + size_t length = 1; + int socket_fd; + int ret; + + if (path == NULL) { + TESTPMD_LOG(ERR, "Invalid devargs extension is specified\n"); + return -1; + } + + /* Initialize IPC channel. */ + socket_fd = socket(AF_UNIX, SOCK_DGRAM, 0); + if (socket_fd < 0) { + TESTPMD_LOG(ERR, "Failed to create unix socket: %s\n", + strerror(errno)); + return -1; + } + snprintf(un.sun_path, sizeof(un.sun_path), "%s_%d", path, getpid()); + unlink(un.sun_path); /* May still exist since last run */ + if (bind(socket_fd, (struct sockaddr *)&un, sizeof(un)) < 0) { + TESTPMD_LOG(ERR, "Failed to bind %s: %s\n", un.sun_path, + strerror(errno)); + close(socket_fd); + return -1; + } + + strlcpy(dst.sun_path, path, sizeof(dst.sun_path)); + /* Send the request message. */ + do { + ret = sendmsg(socket_fd, &msgh, 0); + } while (ret < 0 && errno == EINTR); + if (ret < 0) { + TESTPMD_LOG(ERR, "Failed to send request to (%s): %s\n", path, + strerror(errno)); + close(socket_fd); + unlink(un.sun_path); + return -1; + } + + msgh.msg_iov = &iov; + msgh.msg_iovlen = 1; + msgh.msg_control = control.buf; + msgh.msg_controllen = sizeof(control.buf); + do { + ret = recvmsg(socket_fd, &msgh, 0); + } while (ret < 0); + if (ret != sizeof(int) || (msgh.msg_flags & (MSG_TRUNC | MSG_CTRUNC))) { + TESTPMD_LOG(ERR, "truncated msg"); + close(socket_fd); + unlink(un.sun_path); + return -1; + } + + /* Translate the FD. */ + cmsg = CMSG_FIRSTHDR(&msgh); + if (cmsg == NULL || cmsg->cmsg_len != CMSG_LEN(sizeof(int)) || + cmsg->cmsg_level != SOL_SOCKET || cmsg->cmsg_type != SCM_RIGHTS) { + TESTPMD_LOG(ERR, "Fail to get FD using SCM_RIGHTS mechanism\n"); + close(socket_fd); + unlink(un.sun_path); + return -1; + } + memcpy(&cmd_fd, CMSG_DATA(cmsg), sizeof(int)); + + TESTPMD_LOG(DEBUG, "Command FD (%d) and PD handle (%d) " + "are successfully imported from remote process\n", + cmd_fd, pd_handle); + + /* Cleanup IPC channel. */ + close(socket_fd); + unlink(un.sun_path); + + /* Calculate the new length of devargs string. */ + length += snprintf(NULL, 0, ",cmd_fd=%d,pd_handle=%d", + cmd_fd, pd_handle); + /* Extend the devargs string. */ + snprintf(extend, length, ",cmd_fd=%d,pd_handle=%d", cmd_fd, pd_handle); + + TESTPMD_LOG(DEBUG, "Attach port with extra devargs %s\n", identifier); + return 0; +} + +static bool +is_delimiter_path_spaces(char *extend) +{ + while (*extend != '\0') { + if (*extend != ' ') + return true; + extend++; + } + return false; +} +#endif + void attach_port(char *identifier) { portid_t pi; struct rte_dev_iterator iterator; +#if defined(RTE_NET_MLX5) && !defined(RTE_EXEC_ENV_WINDOWS) + char *extend; +#endif printf("Attaching a new port...\n"); @@ -3213,6 +3356,16 @@ attach_port(char *identifier) return; } +#if defined(RTE_NET_MLX5) && !defined(RTE_EXEC_ENV_WINDOWS) + extend = strchr(identifier, ' '); + if (extend != NULL && is_delimiter_path_spaces(extend) && + attach_port_extend_devargs(identifier, extend) < 0) { + TESTPMD_LOG(ERR, "Failed to extend devargs for port %s\n", + identifier); + return; + } +#endif + if (rte_dev_probe(identifier) < 0) { TESTPMD_LOG(ERR, "Failed to attach port %s\n", identifier); return; diff --git a/doc/guides/testpmd_app_ug/testpmd_funcs.rst b/doc/guides/testpmd_app_ug/testpmd_funcs.rst index 1083c6d538..d6490947c4 100644 --- a/doc/guides/testpmd_app_ug/testpmd_funcs.rst +++ b/doc/guides/testpmd_app_ug/testpmd_funcs.rst @@ -2127,6 +2127,46 @@ the mode and slave parameters must be given. Done +port attach with mlx5 socket path +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +MLX5 internal option to attach a port specified by pci address or virtual device +args and add extra devargs to it, which is imported from external process:: + + testpmd> port attach (identifier) mlx5_socket=(path) + +where: + +* ``identifier``: pci address or virtual device args. +* ``path``: socket path to import arguments agreed by the external process. + +The mlx5 PMD enables to import CTX and PD created outside the PMD. +It gets as devargs the device's ``cmd_fd`` and ``pd_handle``, +then using those arguments to import objects. +See :ref:`mlx5 driver options ` for more information. + +When ``cmd_fd`` and ``pd_handle`` arguments are coming from another process, +the FD must be dup'd before being passed. +In this function, testpmd initializes IPC socket to get FD using SCM_RIGHTS. +It gets the external process socket path, then import the ``cmd_fd`` and +``pd_handle`` arguments and add them to devargs list. +After updating this, it calls the regular ``port attach`` function +with extended idevtifier. + +For example, to attach a port whose pci address is ``0000:0a:00.0`` and its +socket path is ``/var/run/import_ipc_socket``. + +.. code-block:: console + + testpmd> port attach 0000:0a:00.0 mlx5_socket=/var/run/import_ipc_socket + Attaching a new port... + testpmd: MLX5 socket path is /var/run/import_ipc_socket + testpmd: Attach port with extra devargs 0000:0a:00.0,cmd_fd=40,pd_handle=1 + EAL: Probe PCI driver: mlx5_pci (15b3:101d) device: 0000:03:00.0 (socket 0) + Port 0 is attached. Now total ports is 1 + Done + + port detach ~~~~~~~~~~~ -- 2.25.1