From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7181EA034E; Wed, 9 Feb 2022 18:55:40 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 40D1E41101; Wed, 9 Feb 2022 18:55:40 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 2BDD940140 for ; Wed, 9 Feb 2022 18:55:38 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1644429338; x=1675965338; h=message-id:date:to:cc:references:from:subject: in-reply-to:content-transfer-encoding:mime-version; bh=xy0rGH6+GP6C0JAQUDq0odMRUiUOOcArDTedR3Y/VI8=; b=e1Q1ip1dszXlQnCJquiUwrYQigjUHy8qWkw+BW95p/x5M8D+yCa1FL+u yS2sDVFNNiqw5H8TXNzSn9yM1r2GiRsfaNhWn0dUtgYFX+BeYgDbUO4cF LEqBty915Prdrf9Ko2mnODqzsVwnM2TR4Wptw7ZbvfnnyeLjO7Uy8JTzO PaJndg/pRYn0TbJaFl8WT/phi2JsuY60WBs6d4OWPdqc7ByZMqWr9x8C9 /f6weyo3k2sjotH8ekm5qfVMwpi8ibyeLVAberGPo1KAKLS2uRygJk8Wt y9cyPgGuETkyDCfY0T9n3+ofNyNfdv9iANfwpAZXpaYeFHmgvQLS57/lC Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10252"; a="232844140" X-IronPort-AV: E=Sophos;i="5.88,356,1635231600"; d="scan'208";a="232844140" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Feb 2022 09:55:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,356,1635231600"; d="scan'208";a="585654235" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by fmsmga008.fm.intel.com with ESMTP; 09 Feb 2022 09:55:36 -0800 Received: from fmsmsx608.amr.corp.intel.com (10.18.126.88) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Wed, 9 Feb 2022 09:55:36 -0800 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx608.amr.corp.intel.com (10.18.126.88) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Wed, 9 Feb 2022 09:55:35 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20 via Frontend Transport; Wed, 9 Feb 2022 09:55:35 -0800 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.176) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.20; Wed, 9 Feb 2022 09:55:35 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dMG6n+QV46UV+KAHEOwK3yUD0L1OZyxYCGFFlEDgohVSp5UPPwRC7PizuAOvlhsSb7V2Ps5T5/geCD+xY+WoSKIH4x15ERySG0XeCWU4TFuvG1eeg8SL1tW8HkheyhLL6I4j1DTteKaZlq/m0S/IUO9LZZv01cAxegdbG1klk+HQBpLDX6TxByRLhoGFuWIn9d0qcOJj/KuzeibrCDmnVwZyyBou/Iz3M3aWF4zUyFG4tRsBVhmtnkS5UXKIXYct9xt/mhYBtoy5qF9poVEWzUQXc3jvFmVEk+fENqnvB7y9k9GumMUdioE9l2pAPQgKl9Ovle4WHLXRUUW9ii0lcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QYTt8O36NI5643E8xwQCa+wa1GKeyX4xqRHjT1GwPcU=; b=kUFopRnNkWilaBRU9HBInO45QoJU7yo64NtQmCfOlc/32+p7wfWwLQ0mo2FzagOX8bmSAGayOow9+KwuXWf/HfAj7P2Lb3YqqvKxGG5dvO7qpZVgWZW3xrGZcOvjiy1z+r1Rrgx1l7vOX1MTNUSS2fIx6xBfAna9wNeuII5xlzt+KcF22YDuBMFY3OVYax5BcCSSxi3YIRJYFREW8raShtEW1VArSHpQVr7FhtGlE5EuV6qx6OW9qJ3lsJEm2a8ZWdO4dzT2MJZz8MTB5cm6m28RHXT8oOfftJeAM7macqBlcRC17XaFhpy53dYFLahsDhVYt4NAJ5ShRXMePwxiOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) by MN2PR11MB3824.namprd11.prod.outlook.com (2603:10b6:208:f4::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4975.11; Wed, 9 Feb 2022 17:55:33 +0000 Received: from PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::98be:5506:5020:28a2]) by PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::98be:5506:5020:28a2%4]) with mapi id 15.20.4975.011; Wed, 9 Feb 2022 17:55:32 +0000 Message-ID: <713f6d3a-51dc-a9e8-d0d2-2623fd44adb6@intel.com> Date: Wed, 9 Feb 2022 17:55:27 +0000 Content-Language: en-US To: Ciara Loftus , CC: , References: <20220208134800.52142-1-ciara.loftus@intel.com> <20220209094808.41464-1-ciara.loftus@intel.com> From: Ferruh Yigit Subject: Re: [PATCH v4] net/af_xdp: re-enable secondary process support X-User: ferruhy In-Reply-To: <20220209094808.41464-1-ciara.loftus@intel.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P123CA0327.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18c::8) To PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c2129347-1e4f-467a-caa8-08d9ebf55d7f X-MS-TrafficTypeDiagnostic: MN2PR11MB3824:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3383; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gf+CL3fT9WQbzXh6Y8AIWUTmjlvx7ckFCXBoFp/OKHU+gK97AI67Ma22lChNZGzPQL80PJBcfF4MTEj5Q+ArFzBYlc0ziXb7dFTea1Qify7xWsfMv+n2o0oKhfM18CSu4dMi6S3ATtAbsYgsL8d13w80FWWY8QXb7XDTcfvo5eLkSSs9uUjmjGML2JX7Tu8fyZG6hPD8zQYhHdusPuYkapEqm16J4nefqVxA7gsfNo2K9EJ6RegvaPHn6m/pgXg5mEjg7rWGeUn4DqaPMD8ZuA82w8k4/L7WA28BYwm7t6PY/Bjz2El+vnV342JqsjbhGxSyxbRi/H/N1S8hftkv7sx5QDsUVZU3zl/0xkcJvq3HHHZLjCx3jrMN4I0eHyysoZOklu8izh7r3FiMRuWjcyaC0bgIJiVfSQvvcYE/+D0h67x84VKNzrAKrRQxAMtx05FSby0wCScGNWXN6rp7/M1BYTnVScxlCTyEmyuJqu/HDG0nwX6fB7gVTk/Bnt+tNjGmAtSBkmprTFGMbMqvPMcjQtr76iM2hldrBLeDJ4uJyvthGNdguPjNhegHezZ84Zj46ctAfGksFgI5hA12gKMQNgxHVwjhGtKlEBI89Y6YNbPOZVVGPc3Q56bgq92eSBJ1HgrcaaiZiAx4sS9RCoBpouLp5YjJQADoJDdseS5PywLyg8SdgjvtqcfhOoKd0j2wLOnQUVCqBKsd/W/oDH4swPtWZXya/rSuGAasRn43U7D65q55FUxw5tImiw72vPKC8RvmcjJGtdJ6gGk1hSt36RhtrCpWxgQ3Gah2fd5Z8dzwGjynOnx31Gqk2L2Y3ooC7ie0HpykDpTQdFpQEQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB5000.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(31696002)(86362001)(38100700002)(82960400001)(6666004)(44832011)(8936002)(5660300002)(6512007)(508600001)(316002)(966005)(6486002)(26005)(4326008)(66476007)(66556008)(8676002)(66946007)(107886003)(83380400001)(31686004)(186003)(53546011)(55236004)(36756003)(2906002)(30864003)(6506007)(2616005)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?T2NVb041S3Y5Tzd0TUVGdlQzTEpobDBlMERtN1lZZ29iMm5IVzVsSTlQRlB5?= =?utf-8?B?SnBFMEloVlI1UnFBUGtqbzRubnpYWjAwbGdqbzhkcVVlb3U3M0ZhNlBLWitr?= =?utf-8?B?Ym8yQXhwK3QvQlY1Y1EzdmhZM3hRSWhNL2J0ZktaRjQvc2ptUmZ1bFFPbzdV?= =?utf-8?B?Qm1LTXlwTDFnUlRlSzdzZ3QzTFBRejJxU2QxangzbVR3TzRDcjQrak1IaE1I?= =?utf-8?B?OFNHZEdZL3FJdnYzN2syQTVJN3lNMVFJTnpGUGZ4MzVUSG13TEVVZVd2VVd2?= =?utf-8?B?QkJDWXVTamtiRzBxZDhTRTBwVkVHVkpobGRmbmYwQUhEM0VvM3Z5cTBzZ1F4?= =?utf-8?B?L3JOR2IzTEhsbXM3WlRJbG56L2ovcWNRWWYvM2E3L0Nra1phcDdxVGZ4bzN1?= =?utf-8?B?YWVMa3hvWENEU09OZkkvN2YrYzNsVTZSWHBNV2Y1a2xjSlNwRC9FeWR1RGhj?= =?utf-8?B?SE8yd1hxV0M1YnY2bW5nRUE0azh0aHlGYkt1KzA0aHIza0tSUllXbERzaGxm?= =?utf-8?B?V0tOMm9rSnlwNmN0VWpQS0RTZlhlQ0tBSGtsV2V0M2tpL3VHTEUvSUpScGQx?= =?utf-8?B?MmxlNC9XYjlCOFRnNWp3dDNjdUxYTk43RnBDVE5QK1pnbWI4S2xSVzQyblps?= =?utf-8?B?NkJzYVVqMTlmbHArVFJKUTBodFJod3RkZVNQbVFFRU5tZ05adGtJYUttd1BT?= =?utf-8?B?Q0ZUemtxQ0tVN09UcTJqblB3V0p4WGpEZ0dZY2FnanY3MUx1Q2QxTWVpeWxm?= =?utf-8?B?SmdIektmL0pVbUtSbG05NmZZUnhLNG80elpEblpDZW5xSG1wNS9qMHFrNHFD?= =?utf-8?B?NTluYUl5VWVVN1duaDhwQ1BFNFpGUk85L0RKRmZkZTBJb0RUQUJpUWNDT3V5?= =?utf-8?B?Q1BVVjJEdXVWYkZOY0VnVUNBd1loaUdMMUFpYThnaFpRc2VWTWNVcVkvNktF?= =?utf-8?B?azVEK2E2STBrSndhN2ViYnY5dGFlZG1sN1dNY05vUjdHc2ZFNTM3UEhkaHdm?= =?utf-8?B?U3RvOUI2Z0llSlNqLzVZR0REdXByYmoxWE9ldXM1OU9jUE1kRHArcWZCS2N2?= =?utf-8?B?a21DcjJoVHJVeTRTaGJXd25zdnhST2ZkYTVYcHc1L2FCUjU0YUNldHdWeGVv?= =?utf-8?B?Z3ZNSmMvbE1hR01lZjhPM3FHUU9hRjZBT2E2N0RJb1VlaUhVRi9zTjVldkU2?= =?utf-8?B?MGlJUUlsVi9ZM2tTZm5kQVFJVGNPTEp4RWxWZk10b1ZYZEJqdTg1SzhuTjIv?= =?utf-8?B?MWphN0xPRHFGTkJLSTZFNTFFTzRTMWgycytaWVpvbVhSeVBRT1Y4VjlCdVFM?= =?utf-8?B?c2QxSWxCamxrNlVFeW05clIwcjErcXd6K3hKL2Fiek1rN2dXbkRmeXV2eHVm?= =?utf-8?B?RU9TRWJRcnpiOUxSSzkzOGZ0akVwMmtZVGp1Vyt1T1d5dnA3bjlWa2plNkl4?= =?utf-8?B?b3A1dzZYLy8xbGVNcTNiRjRBS1lYMU1OWUlUOWVwb2J5S0Frc3dSRHdNSEF0?= =?utf-8?B?ak9wQUJUUzdBNlEwNDhOUzdQRXppUTIxZHh6ZXU1Z1lYNHUzLzBhZUVSQ2RR?= =?utf-8?B?SEpvZ2J3YUtJRzFyWjJPN2lndlRSMVlkd2ZXUzN6bW5WRzl6UEVwODExSEVB?= =?utf-8?B?cG5yL3MrZC9rUU5rUVJ3WVpGU1FaV1pQakZWU2Yya2JCVHdWaGM1eGszNjQw?= =?utf-8?B?ZElTSjVSWlBQREo0M3VodHEwOGo2SDNQTjFEUVBUejI4cm13SEhPRDRJYk1p?= =?utf-8?B?a1pieEZaU0xzU1VvUmJhTitGN0FwY0xwb29sdzNWVjZ6OGdBNkI4ZnhSZk1a?= =?utf-8?B?U0x0Nm9NVjVGd3hSVDRGMWEybnZWdjFFeUYvUkNqMlcvcEYvMGRTbHJTbjQv?= =?utf-8?B?cThnN081Smk5SDlqZHJuQW04TGxyOHhnTHFxYmR3aG5qWGZBS1h5ZXJIcUIr?= =?utf-8?B?MThoV2pidFd4VDEzS2xHTmszNk5sdjlYVVE2am5tTEdJVTVEanJwOTYzZ21t?= =?utf-8?B?dSt2MDZpZVBpd3ZBZ21BZ1VBSTNTcDN3T3BxOThSeGpqWmd6REhCVzV6VWRM?= =?utf-8?B?NlhSLzdub3hZS1RjTUs4b0pJdlFkY2lkazJQNkxMdEZhejhNL3ZyMVgzMjMx?= =?utf-8?B?L2FraUhQbk9TSndEaUJueWhRRjl2YXZRMWpuR1ZvTnNMZHY0YkFGVHFPOG5D?= =?utf-8?Q?1H76BiiEdl2ihylqcX1GlgY=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: c2129347-1e4f-467a-caa8-08d9ebf55d7f X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5000.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Feb 2022 17:55:32.0306 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4c5AsgJKX5hbYIrXVhShByg0O42xRK/oiBk6FGuJDmZsGabJllNJiFTyxS6d5Ir2GA/RFSdaG81vpu+B3PIF1w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB3824 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2/9/2022 9:48 AM, Ciara Loftus wrote: > Secondary process support had been disabled for the AF_XDP PMD because > there was no logic in place to share the AF_XDP socket file descriptors > between the processes. This commit introduces this logic using the IPC > APIs. > > Rx and Tx are disabled in the secondary process due to memory mapping of > the AF_XDP rings being assigned by the kernel in the primary process only. > However other operations including retrieval of stats are permitted. > > Signed-off-by: Ciara Loftus > > --- > v3 -> v4: > * Rebase to next-net > * Reword limitation which requires Rx and Tx to be disabled. > > v2 -> v3: > * Rebase to next-net > * Updated logs to be more specific about multi-process IPC > * Free process_private in _close and _remove functions > * Use rte_eth_dev_get_by_name instead of global array > > v1 -> v2: > * Rebase to next-net > > RFC -> v1: > * Added newline to af_xdp.rst > * Fixed spelling errors > * Fixed potential NULL dereference in init_internals > * Fixed potential free of address-of expression in afxdp_mp_request_fds > --- > doc/guides/nics/af_xdp.rst | 9 ++ > doc/guides/nics/features/af_xdp.ini | 1 + > doc/guides/rel_notes/release_22_03.rst | 1 + > drivers/net/af_xdp/rte_eth_af_xdp.c | 215 +++++++++++++++++++++++-- > 4 files changed, 211 insertions(+), 15 deletions(-) > > diff --git a/doc/guides/nics/af_xdp.rst b/doc/guides/nics/af_xdp.rst > index db02ea1984..3d8b70e3f8 100644 > --- a/doc/guides/nics/af_xdp.rst > +++ b/doc/guides/nics/af_xdp.rst > @@ -141,4 +141,13 @@ Limitations > NAPI context from a watchdog timer instead of from softirqs. More information > on this feature can be found at [1]. > > +- **Secondary Processes** > + > + Rx and Tx are not supported for secondary processes due to memory mapping of > + the AF_XDP rings being assigned by the kernel in the primary process only. > + However other operations including statistics retrieval are permitted. > + The maximum number of queues permitted for PMDs operating in this model is 8 > + as this is the maximum number of fds that can be sent through the IPC APIs as > + defined by RTE_MP_MAX_FD_NUM. > + > [1] https://lwn.net/Articles/837010/ > diff --git a/doc/guides/nics/features/af_xdp.ini b/doc/guides/nics/features/af_xdp.ini > index 54b738e616..8e7e075aaf 100644 > --- a/doc/guides/nics/features/af_xdp.ini > +++ b/doc/guides/nics/features/af_xdp.ini > @@ -9,4 +9,5 @@ Power mgmt address monitor = Y > MTU update = Y > Promiscuous mode = Y > Stats per queue = Y > +Multiprocess aware = Y > x86-64 = Y > diff --git a/doc/guides/rel_notes/release_22_03.rst b/doc/guides/rel_notes/release_22_03.rst > index f03183ee86..a9bf3cc7a6 100644 > --- a/doc/guides/rel_notes/release_22_03.rst > +++ b/doc/guides/rel_notes/release_22_03.rst > @@ -72,6 +72,7 @@ New Features > * **Updated AF_XDP PMD** > > * Added support for libxdp >=v1.2.2. > + * Re-enabled secondary process support. RX/TX is not supported. > > * **Updated Cisco enic driver.** > > diff --git a/drivers/net/af_xdp/rte_eth_af_xdp.c b/drivers/net/af_xdp/rte_eth_af_xdp.c > index 1b6192fa44..bfa5d2a038 100644 > --- a/drivers/net/af_xdp/rte_eth_af_xdp.c > +++ b/drivers/net/af_xdp/rte_eth_af_xdp.c > @@ -80,6 +80,18 @@ RTE_LOG_REGISTER_DEFAULT(af_xdp_logtype, NOTICE); > > #define ETH_AF_XDP_ETH_OVERHEAD (RTE_ETHER_HDR_LEN + RTE_ETHER_CRC_LEN) > > +#define ETH_AF_XDP_MP_KEY "afxdp_mp_send_fds" > + > +static int afxdp_dev_count; > + > +/* Message header to synchronize fds via IPC */ > +struct ipc_hdr { > + char port_name[RTE_DEV_NAME_MAX_LEN]; > + /* The file descriptors are in the dedicated part > + * of the Unix message to be translated by the kernel. > + */ > +}; > + > struct xsk_umem_info { > struct xsk_umem *umem; > struct rte_ring *buf_ring; > @@ -147,6 +159,10 @@ struct pmd_internals { > struct pkt_tx_queue *tx_queues; > }; > > +struct pmd_process_private { > + int rxq_xsk_fds[RTE_MAX_QUEUES_PER_PORT]; > +}; > + > #define ETH_AF_XDP_IFACE_ARG "iface" > #define ETH_AF_XDP_START_QUEUE_ARG "start_queue" > #define ETH_AF_XDP_QUEUE_COUNT_ARG "queue_count" > @@ -795,11 +811,12 @@ static int > eth_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) > { > struct pmd_internals *internals = dev->data->dev_private; > + struct pmd_process_private *process_private = dev->process_private; > struct xdp_statistics xdp_stats; > struct pkt_rx_queue *rxq; > struct pkt_tx_queue *txq; > socklen_t optlen; > - int i, ret; > + int i, ret, fd; > > for (i = 0; i < dev->data->nb_rx_queues; i++) { > optlen = sizeof(struct xdp_statistics); > @@ -815,8 +832,9 @@ eth_stats_get(struct rte_eth_dev *dev, struct rte_eth_stats *stats) > stats->ibytes += stats->q_ibytes[i]; > stats->imissed += rxq->stats.rx_dropped; > stats->oerrors += txq->stats.tx_dropped; > - ret = getsockopt(xsk_socket__fd(rxq->xsk), SOL_XDP, > - XDP_STATISTICS, &xdp_stats, &optlen); > + fd = process_private->rxq_xsk_fds[i]; > + ret = fd >= 0 ? getsockopt(fd, SOL_XDP, XDP_STATISTICS, > + &xdp_stats, &optlen) : -1; > if (ret != 0) { > AF_XDP_LOG(ERR, "getsockopt() failed for XDP_STATISTICS.\n"); > return -1; > @@ -884,7 +902,7 @@ eth_dev_close(struct rte_eth_dev *dev) > int i; > > if (rte_eal_process_type() != RTE_PROC_PRIMARY) > - return 0; > + goto out; > > AF_XDP_LOG(INFO, "Closing AF_XDP ethdev on numa socket %u\n", > rte_socket_id()); > @@ -927,6 +945,9 @@ eth_dev_close(struct rte_eth_dev *dev) > } > } > > +out: > + rte_free(dev->process_private); > + > return 0; > } > > @@ -1349,6 +1370,7 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, > struct rte_mempool *mb_pool) > { > struct pmd_internals *internals = dev->data->dev_private; > + struct pmd_process_private *process_private = dev->process_private; > struct pkt_rx_queue *rxq; > int ret; > > @@ -1387,6 +1409,8 @@ eth_rx_queue_setup(struct rte_eth_dev *dev, > rxq->fds[0].fd = xsk_socket__fd(rxq->xsk); > rxq->fds[0].events = POLLIN; > > + process_private->rxq_xsk_fds[rx_queue_id] = rxq->fds[0].fd; > + > dev->data->rx_queues[rx_queue_id] = rxq; > return 0; > > @@ -1688,6 +1712,7 @@ init_internals(struct rte_vdev_device *dev, const char *if_name, > { > const char *name = rte_vdev_device_name(dev); > const unsigned int numa_node = dev->device.numa_node; > + struct pmd_process_private *process_private; > struct pmd_internals *internals; > struct rte_eth_dev *eth_dev; > int ret; > @@ -1753,9 +1778,17 @@ init_internals(struct rte_vdev_device *dev, const char *if_name, > if (ret) > goto err_free_tx; > > + process_private = (struct pmd_process_private *) > + rte_zmalloc_socket(name, sizeof(struct pmd_process_private), > + RTE_CACHE_LINE_SIZE, numa_node); > + if (process_private == NULL) { > + AF_XDP_LOG(ERR, "Failed to alloc memory for process private\n"); > + goto err_free_tx; > + } > + > eth_dev = rte_eth_vdev_allocate(dev, 0); > if (eth_dev == NULL) > - goto err_free_tx; > + goto err_free_pp; > > eth_dev->data->dev_private = internals; > eth_dev->data->dev_link = pmd_link; > @@ -1764,6 +1797,10 @@ init_internals(struct rte_vdev_device *dev, const char *if_name, > eth_dev->dev_ops = &ops; > eth_dev->rx_pkt_burst = eth_af_xdp_rx; > eth_dev->tx_pkt_burst = eth_af_xdp_tx; > + eth_dev->process_private = process_private; > + > + for (i = 0; i < queue_cnt; i++) > + process_private->rxq_xsk_fds[i] = -1; > > #if defined(XDP_UMEM_UNALIGNED_CHUNK_FLAG) > AF_XDP_LOG(INFO, "Zero copy between umem and mbuf enabled.\n"); > @@ -1771,6 +1808,8 @@ init_internals(struct rte_vdev_device *dev, const char *if_name, > > return eth_dev; > > +err_free_pp: > + rte_free(process_private); > err_free_tx: > rte_free(internals->tx_queues); > err_free_rx: > @@ -1780,6 +1819,119 @@ init_internals(struct rte_vdev_device *dev, const char *if_name, > return NULL; > } > > +/* Secondary process requests rxq fds from primary. */ > +static int > +afxdp_mp_request_fds(const char *name, struct rte_eth_dev *dev) > +{ > + struct pmd_process_private *process_private = dev->process_private; > + struct timespec timeout = {.tv_sec = 1, .tv_nsec = 0}; > + struct rte_mp_msg request, *reply; > + struct rte_mp_reply replies; > + struct ipc_hdr *request_param = (struct ipc_hdr *)request.param; > + int i, ret; > + > + /* Prepare the request */ > + memset(&request, 0, sizeof(request)); > + strlcpy(request.name, ETH_AF_XDP_MP_KEY, sizeof(request.name)); > + strlcpy(request_param->port_name, name, > + sizeof(request_param->port_name)); > + request.len_param = sizeof(*request_param); > + > + /* Send the request and receive the reply */ > + AF_XDP_LOG(DEBUG, "Sending multi-process IPC request for %s\n", name); > + ret = rte_mp_request_sync(&request, &replies, &timeout); > + if (ret < 0 || replies.nb_received != 1) { > + AF_XDP_LOG(ERR, "Failed to request fds from primary: %d", > + rte_errno); > + return -1; > + } > + reply = replies.msgs; > + AF_XDP_LOG(DEBUG, "Received multi-process IPC reply for %s\n", name); > + if (dev->data->nb_rx_queues != reply->num_fds) { > + AF_XDP_LOG(ERR, "Incorrect number of fds received: %d != %d\n", > + reply->num_fds, dev->data->nb_rx_queues); > + return -EINVAL; > + } > + > + for (i = 0; i < reply->num_fds; i++) > + process_private->rxq_xsk_fds[i] = reply->fds[i]; > + > + free(reply); > + return 0; > +} > + > +/* Primary process sends rxq fds to secondary. */ > +static int > +afxdp_mp_send_fds(const struct rte_mp_msg *request, const void *peer) > +{ > + struct rte_eth_dev *dev; > + struct pmd_process_private *process_private; > + struct rte_mp_msg reply; > + const struct ipc_hdr *request_param = > + (const struct ipc_hdr *)request->param; > + struct ipc_hdr *reply_param = > + (struct ipc_hdr *)reply.param; > + const char *request_name = request_param->port_name; > + int i; > + > + AF_XDP_LOG(DEBUG, "Received multi-process IPC request for %s\n", > + request_name); > + > + /* Find the requested port */ > + dev = rte_eth_dev_get_by_name(request_name); > + if (!dev) { > + AF_XDP_LOG(ERR, "Failed to get port id for %s\n", request_name); > + return -1; > + } > + process_private = dev->process_private; > + > + /* Populate the reply with the xsk fd for each queue */ > + reply.num_fds = 0; > + if (dev->data->nb_rx_queues > RTE_MP_MAX_FD_NUM) { > + AF_XDP_LOG(ERR, "Number of rx queues (%d) exceeds max number of fds (%d)\n", > + dev->data->nb_rx_queues, RTE_MP_MAX_FD_NUM); > + return -EINVAL; > + } > + > + for (i = 0; i < dev->data->nb_rx_queues; i++) > + reply.fds[reply.num_fds++] = process_private->rxq_xsk_fds[i]; > + > + /* Send the reply */ > + strlcpy(reply.name, request->name, sizeof(reply.name)); > + strlcpy(reply_param->port_name, request_name, > + sizeof(reply_param->port_name)); > + reply.len_param = sizeof(*reply_param); > + AF_XDP_LOG(DEBUG, "Sending multi-process IPC reply for %s\n", > + reply_param->port_name); > + if (rte_mp_reply(&reply, peer) < 0) { > + AF_XDP_LOG(ERR, "Failed to reply to multi-process IPC request\n"); > + return -1; > + } > + return 0; > +} > + > +/* Secondary process rx function. RX is disabled because memory mapping of the > + * rings being assigned by the kernel in the primary process only. > + */ > +static uint16_t > +eth_af_xdp_rx_noop(void *queue __rte_unused, > + struct rte_mbuf **bufs __rte_unused, > + uint16_t nb_pkts __rte_unused) > +{ > + return 0; > +} > + > +/* Secondary process tx function. TX is disabled because memory mapping of the > + * rings being assigned by the kernel in the primary process only. > + */ > +static uint16_t > +eth_af_xdp_tx_noop(void *queue __rte_unused, > + struct rte_mbuf **bufs __rte_unused, > + uint16_t nb_pkts __rte_unused) > +{ > + return 0; > +} > + Hi Ciara, Can you please review following patch, if it is good we can merge it first and rebase this patch on top of it to use generic dummy burst functions. "ethdev: introduce generic dummy packet burst function" https://patches.dpdk.org/project/dpdk/patch/20220208194437.426143-1-ferruh.yigit@intel.com/ > static int > rte_pmd_af_xdp_probe(struct rte_vdev_device *dev) > { > @@ -1789,19 +1941,39 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev) > int xsk_queue_cnt = ETH_AF_XDP_DFLT_QUEUE_COUNT; > int shared_umem = 0; > char prog_path[PATH_MAX] = {'\0'}; > - int busy_budget = -1; > + int busy_budget = -1, ret; > struct rte_eth_dev *eth_dev = NULL; > - const char *name; > + const char *name = rte_vdev_device_name(dev); > > - AF_XDP_LOG(INFO, "Initializing pmd_af_xdp for %s\n", > - rte_vdev_device_name(dev)); > + AF_XDP_LOG(INFO, "Initializing pmd_af_xdp for %s\n", name); > > - name = rte_vdev_device_name(dev); > if (rte_eal_process_type() == RTE_PROC_SECONDARY) { > - AF_XDP_LOG(ERR, "Failed to probe %s. " > - "AF_XDP PMD does not support secondary processes.\n", > - name); > - return -ENOTSUP; > + eth_dev = rte_eth_dev_attach_secondary(name); > + if (eth_dev == NULL) { > + AF_XDP_LOG(ERR, "Failed to probe %s\n", name); > + return -EINVAL; > + } > + eth_dev->dev_ops = &ops; > + eth_dev->device = &dev->device; > + eth_dev->rx_pkt_burst = eth_af_xdp_rx_noop; > + eth_dev->tx_pkt_burst = eth_af_xdp_tx_noop; > + eth_dev->process_private = (struct pmd_process_private *) > + rte_zmalloc_socket(name, > + sizeof(struct pmd_process_private), > + RTE_CACHE_LINE_SIZE, > + eth_dev->device->numa_node); > + if (eth_dev->process_private == NULL) { > + AF_XDP_LOG(ERR, > + "Failed to alloc memory for process private\n"); > + return -ENOMEM; > + } > + > + /* Obtain the xsk fds from the primary process. */ > + if (afxdp_mp_request_fds(name, eth_dev)) > + return -1; > + > + rte_eth_dev_probing_finish(eth_dev); > + return 0; > } > > kvlist = rte_kvargs_parse(rte_vdev_device_args(dev), valid_arguments); > @@ -1836,6 +2008,17 @@ rte_pmd_af_xdp_probe(struct rte_vdev_device *dev) > return -1; > } > > + /* Register IPC callback which shares xsk fds from primary to secondary */ > + if (!afxdp_dev_count) { > + ret = rte_mp_action_register(ETH_AF_XDP_MP_KEY, afxdp_mp_send_fds); > + if (ret < 0) { > + AF_XDP_LOG(ERR, "%s: Failed to register multi-process IPC callback: %s", > + name, strerror(rte_errno)); > + return -1; > + } > + } > + afxdp_dev_count++; > + > rte_eth_dev_probing_finish(eth_dev); > > return 0; > @@ -1858,9 +2041,11 @@ rte_pmd_af_xdp_remove(struct rte_vdev_device *dev) > return 0; > > eth_dev_close(eth_dev); > + if (afxdp_dev_count == 1) > + rte_mp_action_unregister(ETH_AF_XDP_MP_KEY); > + afxdp_dev_count--; > rte_eth_dev_release_port(eth_dev); > > - > return 0; > } >