From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D912641C61; Fri, 10 Feb 2023 21:50:40 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 8558F410D3; Fri, 10 Feb 2023 21:50:40 +0100 (CET) Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2050.outbound.protection.outlook.com [40.107.92.50]) by mails.dpdk.org (Postfix) with ESMTP id DB22840687 for ; Fri, 10 Feb 2023 21:50:38 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RgVSLwevVwVjXq6rJnYzK0+YKJXE8YyZKIWjZGdd4YPvQI+9dx+GkmxHZbrYvrX2UmhAL5AVtHSaDCI8Uoiiv7/zkvssciTaIVPkgjnE2HApL4fjZCTnUj+yyKmw6nXB0kbBY0ir/IB5DsyjoAwPUdc6zzkJ/E4FphgAR3XgBifsE/QZI88JDhhG/88oqIGKRp77xFbdnF8IPgd26hhoF9TmcT4oskNoFA1aPiif2p3Tt/OtKl7ttugekvtTSSdzNvFRXsn8BdwTsY7gtp2Dv3aY67L0Ci6M4cKmZS9++eYY1o5w2ivGwKM3ZEJSKNUEk+4egzfa/1ehbFKb+h9E3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GH2MvY9/9r+9UgFxrNyUtpp4VDzof17bvbdox9qTYRU=; b=ASxp/FgQrWn4/7DF5Z2cfP1sw8pNv0Hi0m2p5oDtq21kQl3KzR7Ehzlp/4Qnj8nhoPS9fIHY4NGLwhcc7kaQsFlebSwyzwrLXZ+KE75Knd/thrF8xGDKgwbFz1xLccnoV2Sc/Rc1K3+bpBvUtbJkdZ1soY53LuEiEY/2lgRrKps9mQbZYUagepcKJJpx84kKFx/1a2txMIWDA+CAkBrUKUbYiI5maFOosDYnaIMi0S6hXwre3SNxp/op5CS5GAXIBJAMXvdfZjui2StScSkBLITwEwzmIknIo1hmRm+2gnwCthOiBR/tLgqfGA1Ut7xv2nVYTBs0L4KJTbQ7I+fDAw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=GH2MvY9/9r+9UgFxrNyUtpp4VDzof17bvbdox9qTYRU=; b=RYk0ETnr/JMNuUUGmMi0mH07z1OJe4x/qG/FOakzSwxxbOOBWYhumplT/ftmyiiFAV3osXraDCZogHUQHL1v1ZTQk+fu2i58QOhR+3JqQYlEKDDkAx3IJfVCfXlhZKXbIxqAIyxoCNBLUDP+o3EV3+pFBMvAGUE6dKXbmRwerhQ= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by SA0PR12MB4415.namprd12.prod.outlook.com (2603:10b6:806:70::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6086.21; Fri, 10 Feb 2023 20:50:36 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::3614:22ed:ed5:5b48]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::3614:22ed:ed5:5b48%8]) with mapi id 15.20.6086.021; Fri, 10 Feb 2023 20:50:36 +0000 Message-ID: <8de21cfd-458f-17ed-5b32-013bde7636ca@amd.com> Date: Fri, 10 Feb 2023 20:50:29 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.7.1 Content-Language: en-US To: "Koikkara Reeny, Shibin" , "dev@dpdk.org" , "Zhang, Qi Z" , "Burakov, Anatoly" , "Richardson, Bruce" , "Mcnamara, John" Cc: "Loftus, Ciara" References: <20230202165513.31012-1-shibin.koikkara.reeny@intel.com> <20230209120549.388318-1-shibin.koikkara.reeny@intel.com> <651bec6d-2d33-347e-3938-06c8118c6c0b@amd.com> From: Ferruh Yigit Subject: Re: [PATCH v4] net/af_xdp: AF_XDP PMD CNI Integration In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P265CA0214.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:33a::9) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|SA0PR12MB4415:EE_ X-MS-Office365-Filtering-Correlation-Id: f53344bd-db71-40cc-56e7-08db0ba87591 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4G3gQhu9nu+z4PYuxZgHVRtMBsEvDaEQT9lLc/S3nR/lCSRq+QwVNAPlP/loSNIt+vs8buG2L6dccPZbQQFatK/mdfb+Vu3WrJo507rpSLzNoPOpEK0DnNCImZPsnhmgNVaZr+Yvz9OuRzD9jbgmxSpMJmr2J+zeH/V8Ek9xIE2IQIbPIDFR2Gt4MRRndK/MEAMLXJFvADjWEwtxY38Lw46hJTGNzOsid/gwEcepLXZ8LM0+Ds/XYWO664v0O07MRQUtod/46xl0S4MWPvGTQaUzWqMJnPvvqJTq7BVM6codKHGH5AYhXPfWMpXFFGUox3mRPqIILFW7qmgWE+itbbYdqLkGpJZssclabXYQQCrYhEj1OSlL2/HkfPm1UvbED7cUM4aszQ1LMIKaFvItzIoGq/mNgfp6plK1/ZGue3TttaG23StXCB/K6LOI1nolT8ke/gFmSNyTllv3ISIj0UTK5sT6B0CRLR+8i81T/qqHmOFg1Ivh9eQSmaINR9cMtrxLSYuXaw1T4Z6IDmQkXI3CQSHVIJTGe2Jz/wPJjcgb8lsvw+oKGWdB2LDcQ7yUWCvrB5goOM3CIPhA5inEeI6bG7ywy5Vq3lwEfHNVtmUAhFx9iEjHDa09Xtzu8x3EVGihkWU2MFTsgMQ6cKnmb4FAkqJ8Rx6BUzTvhS56IFfdEvI7t8QMclZPRTWa7+E9VlAj3e1rhSUCzw4ukT536VcBt792pD/zsnJyFi7D2rL8D/9zPmkH6Xwv/r7ptlRbwtEkxTZGp7mGW4dIg8LD8eEpzIu2KiGQSE/9EoR5DmU= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(4636009)(376002)(346002)(396003)(136003)(366004)(39860400002)(451199018)(8676002)(110136005)(4326008)(8936002)(316002)(41300700001)(66556008)(66476007)(66946007)(31696002)(921005)(36756003)(86362001)(38100700002)(6512007)(186003)(26005)(53546011)(6506007)(31686004)(6666004)(6486002)(2616005)(478600001)(5660300002)(44832011)(2906002)(966005)(83380400001)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OXJ6QWgvQjlGVHR6RVBSQjF1QitjbHVBQjBaRWxmZE5kT0pZS3pjUGVaZXg5?= =?utf-8?B?Q1Nid216ZE5GOEd4RjNJblptMjBLVFJCcVFOakZUTnRJSXNhRE1LdzZxMmZG?= =?utf-8?B?WllCMUNoV1ZyMi9udW13a1hTMGY5NFJkanpoVzYwNUs3K0pVQU9MaXE0YmR6?= =?utf-8?B?U2RleE9lOHNLdW9qbXRJREppWDJkNGF4T1V4WU0rQ2syOTBmNzh5NFljSDdY?= =?utf-8?B?VXNKcnFPbzBDNW5CdjJMOE5mUlVqSXFsK2VQVzV0cW5MT0pZbTlEbTFOVzdD?= =?utf-8?B?TmtMMjY5RlFndmNpdjN4U0dYUFZBYjliVW9ZTGVXbWp6Y25WVU0xTzJjSXBy?= =?utf-8?B?dmp3NHBNRGIyV25lQlA1a0dsNkZ4RTFiZFRLU1BMdlpTcW1Wc2tJOTJZZzlv?= =?utf-8?B?Z0VOd0d3amFDVVdYRSs0SEV3QmMrZVVEdjZaYkI3elZlQW5hOXkvZTNsWVJK?= =?utf-8?B?OEhOSUdhRENNdmpqZVRQaEJHQm9rZXp2Zk1WUHZoU2NweStxVDZnc2ZwNWZr?= =?utf-8?B?SDBkdW9MclVtcnQ1cXVzN0krMSs4bkc0S0JGZ1lsQTF5bnBvUitnUWlvem9G?= =?utf-8?B?WG1NdzBWaFA5SDVTV2oxUklqc2ttdHNCc1NFMjIwaXJaejA0SDZzbDBpeTcr?= =?utf-8?B?V3R6eFg5YlNVbktzWWZ0NzYvTElRcW9wQS91QW5sbzRwYlJhV0RwVXI4WlVo?= =?utf-8?B?MWZMeDkwZWxTbDhpWHk4VTgyeENGVnVabVE3S0JYR0VYVjBmOFN1ZTdZOWhu?= =?utf-8?B?Y2ZmZjVjZjdSdjY0aTNVVTZhcHFHeFg2SW5oZlZKRjRaQ0Q3bHh0SmtpeXV4?= =?utf-8?B?MzNhek1sWXp0UTY3d0htM1JBOXM1a294SmZpTElWazFCYjQ1dnJoOGlPS2Er?= =?utf-8?B?V1hOWlIzNFdmSXJmc0JHYzdRZkV0ejZRTFlJT25wUGx5MG5tWDlwY0ZkWmpq?= =?utf-8?B?bnFIdnpZWFFNQVB6OEgvK2dFRDBsT3h4Tk5lclVIRVR5YUFLdmV2SjB6ZEJz?= =?utf-8?B?Zkl6Yk9nMWtqNnU2Vm4xZXZFMlhpRTh5enUrbG5CV0dNWHBWKzhaaGNHcnQx?= =?utf-8?B?emdTS3EzbFFTS3ZuaWNoWWlITHdJRDhDVk93SWg1MFFyN3JZRUFacU9pYXJj?= =?utf-8?B?bGFiV2JZSVIvWW1yeWN0a1pqZkdYYy9wUmt0UzNjd0tLbHJIcUZYY0xOeHZR?= =?utf-8?B?SGgzWkUzSXg3WU5xdVVQaTBDakluNldrcDN6d1RyMnVCMjhrZDVmbUd5NWN2?= =?utf-8?B?ZW55Y2hpVFhKQ3VMbm5WeFoyTUlCWVMwTkJCWDV5bGppVTNzdnZLZ0pVYXJE?= =?utf-8?B?cnlqbndmMHdPM05rSWNPU2hzV1VHa0Z2Z2htMFp6Z0p1VVl1MStlTnF0Ryts?= =?utf-8?B?QmNFUVREYWRIR0xsdVVZelJtRmZwaDgyaVVyYmJZMGloL2VFb1RFaDlIZXFL?= =?utf-8?B?bXFOYVpxb3RxaFVKcFVYY2ZEYjEwajBza3JwOWw2Y093akp3cldZdXBmOEZC?= =?utf-8?B?WXNDWUowNC9WbDlxNUZrTFZHVFhsbFgrK1UwUzBuYi9UQWdKS29kVGlRL3Zo?= =?utf-8?B?R2xCaFdndmZlU1BKRkh6WkZoVlJvYy9HbU5NeCtmV2lXSm5Eb1RSeFJtdjhV?= =?utf-8?B?OWNWNTBDallDSjA5N3VtdkdZYi9Gbm1wZWhoUUZSVlVJVnYySXJiTEdoK2FF?= =?utf-8?B?dGEwaGNCeTdrQVhSakNYZVFWZ29ZKythVFFLTEhibXhibVZqR3h3K2c2QkpW?= =?utf-8?B?THVEcWVEeTJXWHZlaExFWTNKRThOSWR3OEgxbkc5Q1ZtM0NnaDBDMC9OZHhT?= =?utf-8?B?V29EZkE4UVVhTDg0RTRxVk1VZ01iVEttTnk4Z294eWZmWlVyYVNLNUpuNHdW?= =?utf-8?B?bFBPUWtLcSthb2Y4ZEUzUFh3aGNWa1VhMFNWdGZkVXI0TVJZSlhiUWErckxx?= =?utf-8?B?Yk9DS3puN2t6cXlrbWI1cVBLeElOUkpKOHA5aHFIZ1NhSzdnSkZwZStYeEda?= =?utf-8?B?MWFZeERYdm1JcFZEV05qVDRSQnhpOEthKzJ6T1N0VSs0ODNmdjA4Uis3ZUVB?= =?utf-8?B?ZGZKd2Z6THFVQW11UndkM3hOTFdBSjdQU3YvajdBQmhXVUlyU0lrK29VQSto?= =?utf-8?Q?HhPzys5gBKocQVnWYbqZnGvdM?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: f53344bd-db71-40cc-56e7-08db0ba87591 X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Feb 2023 20:50:36.3251 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7yqNxB8Q0xTUDW5Ri84NxUR3p9CRWEr4+Qscs83tBFKVsRcVN4tnqaPw99FyEizc X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4415 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 2/10/2023 3:38 PM, Koikkara Reeny, Shibin wrote: > > >> -----Original Message----- >> From: Ferruh Yigit >> Sent: Friday, February 10, 2023 1:04 PM >> To: Koikkara Reeny, Shibin ; >> dev@dpdk.org; Zhang, Qi Z ; Burakov, Anatoly >> ; Richardson, Bruce >> ; Mcnamara, John >> >> Cc: Loftus, Ciara >> Subject: Re: [PATCH v4] net/af_xdp: AF_XDP PMD CNI Integration >> >> On 2/9/2023 12:05 PM, Shibin Koikkara Reeny wrote: >>> Integrate support for the AF_XDP CNI and device plugin [1] so that the >>> DPDK AF_XDP PMD can work in an unprivileged container environment. >>> Part of the AF_XDP PMD initialization process involves loading an eBPF >>> program onto the given netdev. This operation requires privileges, >>> which prevents the PMD from being able to work in an unprivileged >>> container (without root access). The plugin CNI handles the program >>> loading. CNI open Unix Domain Socket (UDS) and waits listening for a >>> client to make requests over that UDS. The client(DPDK) connects and a >>> "handshake" occurs, then the File Descriptor which points to the >>> XSKMAP associated with the loaded eBPF program is handed over to the >>> client. The client can then proceed with creating an AF_XDP socket and >>> inserting the socket into the XSKMAP pointed to by the FD received on >>> the UDS. >>> >>> A new vdev arg "use_cni" is created to indicate user wishes to run the >>> PMD in unprivileged mode and to receive the XSKMAP FD from the CNI. >>> When this flag is set, the XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf >>> flag should be used when creating the socket, which tells libbpf not >>> to load the default libbpf program on the netdev. We tell libbpf not >>> to do this because the loading is handled by the CNI in this scenario. >>> >>> Patch include howto doc explain how to configure AF_XDP CNI to working >>> with DPDK. >>> >>> [1]: https://github.com/intel/afxdp-plugins-for-kubernetes >>> >>> Signed-off-by: Shibin Koikkara Reeny >> >> >> Is Anatoly's tested-by tag still valid with this version? > > Yes it is still valid. > >> >> <...> >> >>> @@ -1413,7 +1678,23 @@ xsk_configure(struct pmd_internals *internals, >> struct pkt_rx_queue *rxq, >>> } >>> } >>> >>> - if (rxq->busy_budget) { >>> + if (internals->use_cni) { >>> + int err, fd, map_fd; >>> + >>> + /* get socket fd from CNI plugin */ >>> + map_fd = get_cni_fd(internals->if_name); >>> + if (map_fd < 0) { >>> + AF_XDP_LOG(ERR, "Failed to receive CNI plugin >> fd\n"); >>> + goto out_xsk; >>> + } >>> + /* get socket fd */ >>> + fd = xsk_socket__fd(rxq->xsk); >>> + err = bpf_map_update_elem(map_fd, &rxq- >>> xsk_queue_idx, &fd, 0); >>> + if (err) { >>> + AF_XDP_LOG(ERR, "Failed to insert unprivileged xsk >> in map.\n"); >>> + goto out_xsk; >>> + } >>> + } else if (rxq->busy_budget) { >> >> >> 'use_cni' argument is added as if-else, this result 'use_cni' parameter >> automatically makes 'busy_budget' argument ineffective, is this intentional? >> If so can you please describe why? >> And can you please document this in the driver documentation that 'use_cni' >> and 'busy_budget' paramters are mutually exclusive. >> May be this condition can be checked and an error message sent in runtime, >> not sure. >> > > When we use "use_cni" option inorder to configure the busy_budget we need to send the request to the CNI plugin > and CNI plugin will configure the busy_poll. As the dpdk is running inside a container with limited permissions. > >> >> Similarly, another parameter check above this (not visible in this patch), >> xdp_prog (custom_prog_configured) is calling same APIs >> (bpf_map_update_elem()), if both paramters are provided, 'use_cni' will >> overwrite previous one, is this intentional? >> Are 'use_cni' & 'xdp_prog' paramters mutually exclusive? > > When we use "use_cni" we don't have the permission to load the xdp_prog. As our privileges are limited inside the container. > CNI plugin handle the loading of the program. Yes, but what happens if user provides 'xdp_prog' parameter? >> >> >> Overall is the combination of 'use_cni' paramter with other parameters >> tested? > > We have tested the communication with CNI plugin which load the program and traffic flow. > I got that, but is the combination of 'use_cni' parameter with other parameters tested? Like what happens if user provides both 'xdp_prog' & 'use_cni'? There is no documentation for this condition or there is no check in the code that can provide some log message to user. >> >> >>> ret = configure_preferred_busy_poll(rxq); >>> if (ret) { >>> AF_XDP_LOG(ERR, "Failed configure busy >> polling.\n"); @@ -1584,6 >>> +1865,27 @@ static const struct eth_dev_ops ops = { >>> .get_monitor_addr = eth_get_monitor_addr, }; >>> >>> +/* CNI option works in unprivileged container environment >>> + * and ethernet device functionality will be reduced. So >>> + * additional customiszed eth_dev_ops struct is needed >>> + * for cni. Promiscuous enable and disable functionality >>> + * is removed. >> >> >> Why promiscuous enable and disable functionality can't be used with >> 'use_cni'? > > When we use "use_cni" we are running dpdk_testpmd inside a docker and inside the docker we have only > limited permissions only ie the reason I have written it as "unprivileged container environment" > it the comment. >> >> Can you please document the limitation in the driver document, also if >> possible briefly mention reason of the limitation? > > In the documentation as prerequisites we have added : > +* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and ``CAP_BPF`` > + for AF_XDP along with support for hugepages. > > In the Background: > +The standard `AF_XDP PMD`_ initialization process involves loading an eBPF program > +onto the kernel netdev to be used by the PMD. This operation requires root or > +escalated Linux privileges and thus prevents the PMD from working in an > +unprivileged container. The AF_XDP CNI plugin handles this situation by > +providing a device plugin that performs the program loading. > > If you think we need to add more please let me know. > Hi Shibin, Thanks for the update. I think it would be good to update driver documentation, 'doc/guides/nics/af_xdp.rst', and update where 'use_cni' parameter documented with following additional information: - When 'use_cni' parameter is used, 'busy_budget' parameter is not valid and has no impact - When 'use_cni' parameter is used, 'xdp_prog' parameter is not valid and ? (what happens when provided) - enable and disable promiscuous mode is not supported, and describe briefly why (I know code has comment for it but less put it in documentation too).