From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id AD502A04AE; Tue, 8 Feb 2022 11:58:59 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9BCF341144; Tue, 8 Feb 2022 11:58:59 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mails.dpdk.org (Postfix) with ESMTP id CFC5D410FC for ; Tue, 8 Feb 2022 11:58:57 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1644317938; x=1675853938; h=message-id:date:to:cc:references:from:subject: in-reply-to:content-transfer-encoding:mime-version; bh=W9BaZ3jBUjPDb6eNRb2+JLjSM2DVExWLuZTY2szY/xk=; b=U5pHCzB5RvwR045Axo8MaEV5EqCWr2mwGc2evypqPonhWhMnb2FAQeUM Q/D7XAb/EoG8GVxQiee1ZLZOD5a7Fgt6lMqXbgPpeVZXcKT7XOtxjT4Lq dMD4N0zDPtnnQBzlj4QLEWEoPwk8p+o0TgxRgAFHYlaSAhHnf51oGf+t6 I3+K3wpIcz7+aSsn93wAzxA1Ga8trcD8bwHR83AWnZC7MpF+1d+ynJOKd d4NqO3+FaU+ryGyIG0bVZ1F6lnNdEiKJbb15rboWg7z1RHaalPOZSOf5O UOSmN1kEwC2zjkUVseq4mE6dAaQtNeUB2fJiR+JKyF07DpMhxqjVy6tWo Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10251"; a="312221704" X-IronPort-AV: E=Sophos;i="5.88,352,1635231600"; d="scan'208";a="312221704" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Feb 2022 02:58:56 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,352,1635231600"; d="scan'208";a="628854230" Received: from fmsmsx605.amr.corp.intel.com ([10.18.126.85]) by fmsmga002.fm.intel.com with ESMTP; 08 Feb 2022 02:58:56 -0800 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx605.amr.corp.intel.com (10.18.126.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Tue, 8 Feb 2022 02:58:56 -0800 Received: from fmsmsx604.amr.corp.intel.com (10.18.126.84) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20; Tue, 8 Feb 2022 02:58:55 -0800 Received: from fmsedg602.ED.cps.intel.com (10.1.192.136) by fmsmsx604.amr.corp.intel.com (10.18.126.84) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.20 via Frontend Transport; Tue, 8 Feb 2022 02:58:55 -0800 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.104) by edgegateway.intel.com (192.55.55.71) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2308.20; Tue, 8 Feb 2022 02:58:54 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OJAXU9IIqaOzoUVgH0G2wwh0U3Oiratmc89zwLOt6/ZoUU+MogfuskXhVBYezep9NkMesYKx0nE6aOg9Z0aq0Ef3OWJt7jFetsa340H/Q1dlR0uAD7CqTS6NUiUKf5i3jpUMXz666e2n1fcrFp+nwpYYDJ+iQDRHQoeFdtP9gdJ11O7c+A8WIbqQugNVDiQoH777We7IdMAFQBBRMT0XzaDRRd23Lu0DgT/hAFk9RuZLEsCpF/WcfZIZF63rkGI0Vj1RgjjrYZCbgV+8RtuNK8n3Jg9PEjK5+Vzwv4L2Be7mBZ5TYlyQ+QK8OKQ3tHFDV0a13x5PsH4w0+CpBopYEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=pZbexIB/ycCein+HfFxB80en8cvbHJOt2seSiWwBWIo=; b=XS3GJzpIEB0ep9hgyFXvKpUNz2G6mIinkrHSEdom0VPvMDdd01YkBK0KjK6VYssC7qnw7AfMqnig3pXs18q/BSaupXdeDKHZ/uW77pE+bPd61YvSMpTT6fSnr+CijHYcXPTjbdFmd+PCqs2tjW9Vq+WlIAH0lN+viBm9rQHBxv29nrQy+mWXSlSCSFmtYanOLI5Ax+2MjWXSAs78AuEqUHD02TQEqR5FSrZ5KTJYs+Wq6OEqftrR5WvWgtLq3cFvkmAMSKPxuREBAwubRmaFuya1ITU2sSomfcCD6rPSpAe3XnU0kbFwCZc4s1StVyyT1PXsuYuL2eLcafRzbsiggA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; Received: from PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) by MN2PR11MB4301.namprd11.prod.outlook.com (2603:10b6:208:188::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4951.18; Tue, 8 Feb 2022 10:58:53 +0000 Received: from PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::5046:8550:928d:850e]) by PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::5046:8550:928d:850e%7]) with mapi id 15.20.4951.019; Tue, 8 Feb 2022 10:58:53 +0000 Message-ID: Date: Tue, 8 Feb 2022 10:58:47 +0000 Content-Language: en-US To: "Loftus, Ciara" , "Burakov, Anatoly" CC: "stephen@networkplumber.org" , "dev@dpdk.org" References: <20220112075406.54121-1-ciara.loftus@intel.com> <20220204125436.30397-1-ciara.loftus@intel.com> <927b094f-2d84-1dbb-0ad5-37dcf1e1c98e@intel.com> <0fa260e7-d5c5-7b92-abd1-b125eb62c6f9@intel.com> From: Ferruh Yigit Subject: Re: [PATCH v2] net/af_xdp: re-enable secondary process support X-User: ferruhy In-Reply-To: Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P123CA0368.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:18e::13) To PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 67fa71b0-fc27-49e0-a7ce-08d9eaf1fe7f X-MS-TrafficTypeDiagnostic: MN2PR11MB4301:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3826; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4BqSPapIPXukKkCFeEDhUGXP8WHYYEPYVsG7Kf6t8vXLsU6Pw7OUbsPnQzBB/B5bxwUEEsj3xrLOXTpdLVc4hkwRP+TRuODG1GjHmTmUKuD1krVSL3CmflbaA8lSUTOtqUq7CWHDy2gq34SxAOmF8rGg80PeT+3CXY9eVkibnK7B2xNIYif2l80SWv31T4lL7CNrv4GlD5U+k7/gTMe7Tlc76Q2ZR1jRLkMwFuAaGnwO6xt02JiSzcDHH3m88M3gWhunhB0QrU3vVYq/pk+qTfH3P/jbaFUcII/y4aTO7VP/X5OE+O7KMZQ+Rs5B3cPr14h9kaAb+C9RpisPHu2bqgTf90nmXH3jmCbumC7rye2buHMJBp92VZDaMLVn4ZLGvO8ns7vdaB1kFZBxyugHPg2ayK6/wItnG7e7dIvUkSRXHx39Eh+qHygNc4xTJCaXO19fmCETpeJuV0y5RRxKBSiv7OM2cX2nbw++2sPApOrM0hnheUzfAbBMYP390t6mnlpei2n/sooeY0HClwpwZgR+fjIFm3ZJ8OsZuQn+dpUdqHGH5lFuLPYjEYCsEkOSfvlJ4+lOG1SOTGZzWHtmtxR8eKQYEBKHOXr0ReHyOoaJUguaJAvi8P65SrpP0PI+rfRo3Zlscmw7HNbcrJB1jFpH4IDKCnD8Ze8wHiXfmo1S/wn7soQpJ03LmJFwHonYfiMPKyDiPBUThsB+ItXvhM4KwHg9JlMqwYTaaWixe1yxIkoWjirHouD+nnCJcxNz8vMIY+zKbamq74kgklmSOmrWWlMp+9b5dSZbzXmK7iE= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB5000.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(186003)(31696002)(54906003)(5660300002)(316002)(86362001)(83380400001)(55236004)(53546011)(36756003)(6506007)(6666004)(6512007)(66476007)(26005)(38100700002)(8676002)(4326008)(66556008)(66946007)(8936002)(6636002)(6486002)(31686004)(508600001)(44832011)(2616005)(966005)(2906002)(82960400001)(110136005)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?UlZ6QjZlUk9BOTVtUXFhd3ZpT2xrSzZJWjVXT290WlhzOWlCc1d1eU5XS0Yz?= =?utf-8?B?ZmFiYlRGdWw1cDRYYVZhTXQ0V01GeGNQMFBzTFpyNzUvQTQ5a0g4VkFhaDl1?= =?utf-8?B?bmxWS3B0S3h1UlYrbWdwUzN5VS9VRWZ2SjhwcjFrazVBUHNoZ2VHUWpMUzJU?= =?utf-8?B?RFRZYzhraW9pb2pENm9xcGluSWpXSllrbExQQkVLTW0xWkZaam9FQ0FzZlpH?= =?utf-8?B?WUMvbllRNmc1Zi9TUjg2NzJ1NDRrTW5QVVovTDZTK1hHK1crN1E0MzR3c2hJ?= =?utf-8?B?UFpJRWM5UVBkeWNVYTVoRWJ1NkQzRE1rYVdEWUoydVNPTVVFUzNpem1YaGp2?= =?utf-8?B?UGpQTmhXWmdoSCs1QUl0aU1CQWNhR1RiYUlYd0tCUG4rTUY2K3BPM3F5U0g3?= =?utf-8?B?WXdMWkJEZEtWSVVtNTY1MlJqUzBpdDhCejJPVXNoek82WVA0RU1vSFlaREF5?= =?utf-8?B?UEREZWNuc2ZyMGRvcDJMaGVPWGNpUUtuTkJRV2NYMmZnZ3FTd2ltby9JNUk4?= =?utf-8?B?em9RWmhyUHFrci9TVHRsTWlsMjhBVSt1SGNiR2dqMUJGSlY0QVRtc3c2bmJC?= =?utf-8?B?RWVYYnVUeDQ4M2ljN1paMnU2MElJbEs0SmtnU1hpd3dqOTBlZ2JIWkFBMTVJ?= =?utf-8?B?TnZNc0lkaWs3L2QzOC9ITGpmczBTVVdnMkVmbHc4eEdBWkZHMVdDVG1zQXE1?= =?utf-8?B?TldDZ0p1MCs5TzZSWWxGVzFZTkdpTll5MzNCK25sbm84dVIyWmxMdVM0bWpw?= =?utf-8?B?WmIxWDFRVjM5c3QrVkt0U1NlVUtjdGRNNEIwMWJmMlhhV1E4YW5TbFh4aXBq?= =?utf-8?B?UXdvSGNySjNBalRUZHBZa29zWmVCeklWMXhLNWc5dnNMN2lZSjlYdWtEY1JV?= =?utf-8?B?NEJYZlBYMTRYM1FzVm52bzQ2U3dRM1RhaWtPbnp6ZnlpQmF1SGV4T2xFVWZO?= =?utf-8?B?UlF6THJGSDN0WkNpM1VsRXVsZy9qTlJWL2ZWcU9XVkp0MWdBV0MwMThoSGp4?= =?utf-8?B?Q2ZaNlpmNjhwdmNjUVd0N0JpUmtFZmh1SENQWmtLb2l6cERXUWRJV2E2TnN4?= =?utf-8?B?UEx5MUJGYURvNFdXSm9jbC9ibS9FYWZQSzBRQ0NDNDR4UUxVM3djWG5CUCtB?= =?utf-8?B?enNOWjFBSXVxMHlTWGx0aDJJTTlzUXY1eTRpU1ljVUxuR1NvN21aeXR6ZnIy?= =?utf-8?B?WFZ2TDI0NHhoZStKallBTXE0UEp5SW8zNzduZlFPZVM0TzB2eFdaM2hNQUNp?= =?utf-8?B?bEs4MnlTYjZVY291eW55SnBQWXJadmdQOHUzRGF4SnZ2UWVuUDZ5amFoWm9z?= =?utf-8?B?MnJXTUVUVjZVeUsvTlg2aHk1R080M3lnQWFGa0tZS1NxY0huYUt4L0svNjla?= =?utf-8?B?UENESC84NzNVM2tRcVphak0yZ2t6Z2RmamIxMk95VmxRdUExQVV5dDRLcTVn?= =?utf-8?B?WkV5RW1NRDh5Vm9LRVQ4K2lVRkpYZkFkenF0TWtWZ2VSVUdSRmFIVllwRTdM?= =?utf-8?B?K1dyRWNkZGxzTnZ4SjZTZGpIbTNrdlh4dmF0SVFKR2JWZVZ5anVDblZyNW9t?= =?utf-8?B?RVFSd2l0V2ZxU3FwOU5nRUFMYWlDZkRMSGpId2JPMmtXSDJsNnF1RzhHYWVJ?= =?utf-8?B?RXFUaklsMzV2ZVhuOEZoTWpoNTNUWnViWmtRcDd2dFR4b3FJL0NxWFRnTWcw?= =?utf-8?B?OUd1V21KT2ZBNFpNT2taZDI3Si9CT0NubEJ1cnR2VDdWN3FqU1hWYzF2SUhM?= =?utf-8?B?WEpUT2hNZUQ4cHlqWkRlNmZQeE9qOVZ0OS8wM0I5cWdVeXl6OGtFWVJYWllL?= =?utf-8?B?N0FlYWkxRUpNaXhyMWhkYXIvV2JMalBFZmxUSEJBYkYxQTk3bXVURGUxNHBx?= =?utf-8?B?TGRKQTJyY0ZySHZ4ZmF6QWdIMldONjB5TFMwK1BLSzZpUTlncTFKaFJrbjBt?= =?utf-8?B?Mkh3KzRMdU1majl4TU5iY0FwWUhUdlhZZVJPdHlyYVRBWk9nL3B0d0tCNTV4?= =?utf-8?B?SkN0U0MyZkM3T1VuQzVTcEVQYnJLQUVRMllIdzNhN3lrdGJSanZZVk9WcFRB?= =?utf-8?B?SHFWYzNSazJ6eUJxWjQzdHNsR0lrRzh5ZjBYejUzbFFNcit6L1BoNC9mK1M2?= =?utf-8?B?S1Z6MDhWb0tYL0YzTmlYL0hVZWY2Vm5xekhKTld6YXRyQjVFa25HdW5CUW9n?= =?utf-8?Q?gh8rzq8Y91bb3sMPHxaCNWw=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 67fa71b0-fc27-49e0-a7ce-08d9eaf1fe7f X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5000.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2022 10:58:52.9930 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6550nAPhL+noYXlGB9Fp/bDE2lnVeGMTvR9aVpstOwHY2sP0quvdeFLmCli37KUTeUWyMTfeVR/91AfB1Gj35w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR11MB4301 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2/7/2022 11:39 AM, Loftus, Ciara wrote: >>>> >>>> On 2/4/2022 12:54 PM, Ciara Loftus wrote: >>>>> Secondary process support had been disabled for the AF_XDP PMD >>>>> because there was no logic in place to share the AF_XDP socket >>>>> file descriptors between the processes. This commit introduces >>>>> this logic using the IPC APIs. >>>>> >>>>> Since AF_XDP rings are single-producer single-consumer, rx/tx >>>>> in the secondary process is disabled. However other operations >>>>> including retrieval of stats are permitted. >>>>> >>>>> Signed-off-by: Ciara Loftus >>>>> >>>>> --- >>>>> v1 -> v2: >>>>> * Rebase to next-net >>>>> >>>>> RFC -> v1: >>>>> * Added newline to af_xdp.rst >>>>> * Fixed spelling errors >>>>> * Fixed potential NULL dereference in init_internals >>>>> * Fixed potential free of address-of expression in >> afxdp_mp_request_fds >>>>> --- >>>>> doc/guides/nics/af_xdp.rst | 9 ++ >>>>> doc/guides/nics/features/af_xdp.ini | 1 + >>>>> doc/guides/rel_notes/release_22_03.rst | 1 + >>>>> drivers/net/af_xdp/rte_eth_af_xdp.c | 210 >>>> +++++++++++++++++++++++-- >>>>> 4 files changed, 207 insertions(+), 14 deletions(-) >>>>> >>>>> diff --git a/doc/guides/nics/af_xdp.rst b/doc/guides/nics/af_xdp.rst >>>>> index db02ea1984..eb4eab28a8 100644 >>>>> --- a/doc/guides/nics/af_xdp.rst >>>>> +++ b/doc/guides/nics/af_xdp.rst >>>>> @@ -141,4 +141,13 @@ Limitations >>>>> NAPI context from a watchdog timer instead of from softirqs. More >>>> information >>>>> on this feature can be found at [1]. >>>>> >>>>> +- **Secondary Processes** >>>>> + >>>>> + Rx and Tx are not supported for secondary processes due to the >> single- >>>> producer >>>>> + single-consumer nature of the AF_XDP rings. However other >> operations >>>> including >>>>> + statistics retrieval are permitted. >>>> >>>> Hi Ciara, >>>> >>>> Isn't this limitation same for all PMDs, like not both primary & secondary >> can >>>> Rx/Tx >>>> from same queue at the same time. >>>> But primary can initiallize the PMD and secondary can do the datapath, >>>> or isn't af_xdp supports multiple queue, if so some queues can be used >> by >>>> primary and some by secondary for datapath. >>>> >>>> Is there anyhing special for af_xdp that prevents it? >>> >>> Hi Ferruh, >>> >>> Thanks for the review. >>> Each queue of the PMD corresponds to a new AF_XDP socket. >>> Each socket has an RX and TX ring that is mmapped from the kernel to >> userspace and this mapping is only valid for the primary process. >>> I did not figure out a way to share that mapping with the secondary process >> successfully. Can you think of anything that might work? >>> >> >> Does the application knows the buffer address for the Rx/Tx, or is >> abstracted to the 'fd'? > > The application knows the buffer address of the Rx/Tx rings. > We pass a pointer to these rings to the libbpf xsk_socket__create API, which sets up the mappings: > http://code.dpdk.org/dpdk/v21.11/source/drivers/net/af_xdp/rte_eth_af_xdp.c#L1291 > Then later on in the datapath we operate directly on those rings: > http://code.dpdk.org/dpdk/v21.11/source/drivers/net/af_xdp/rte_eth_af_xdp.c#L268 > The fd is used in the datapath, but just for the syscalls (recvfrom/poll/send). > Got it, if the buffer address is explicitly required for datapath, fd conversion is not enough. Primary/secondary process works by mapping memory to same virtual address on two different process. The same method can be used for af_xdp multi process support, again @Anatoly can comment better. But this method is fragile, not sure if we should implement it in more places... Anyway, agree to continue this patch without datapath support in secondary. >> If only 'fd' is used, this patch already converts 'fd' between >> processes. >> cc'ed Anatoly, but what I understand is after MP fd conversion: >> Primary process: FD=x >> Secondary process: FD=y >> And both x & y points to exact same socket in the kernel side. >> >> At least this is how it works for the 'tap' interface, and that is >> why 'fs' are in the process_private area and converted between primary >> and secondary, I thought it will be same for the xdp socket. >> >> Did you test the secondary Rx/Tx in the secondary after this patch?