From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 19A9CA00C2; Fri, 9 Dec 2022 17:16:01 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 04F25410FB; Fri, 9 Dec 2022 17:16:01 +0100 (CET) Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2082.outbound.protection.outlook.com [40.107.95.82]) by mails.dpdk.org (Postfix) with ESMTP id 9F4A640A8B; Fri, 9 Dec 2022 17:15:59 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EyIVZeYgrUzw5Zm5YH2GHR7ffROsp0Zr2EJSnfXpS8bVOc1/2712f6z/3B0KooOmzVyb9HK7/WNkH8nM86XE4n+bqeWT6nCTyZNCmalUf/Ybp+QHQeGbrP2CWLAEtE1x+dVWdLGRoVkdZj+3kC9gGWWl58GXqZpgQMTsyvnMmfI6ygN6SwOLptsWHXwAJvllqius70VLbIS5b5MbdxoWNtVObZyDPAPL7FvAtowOd8Fox6ZPuQOpstVhtFMzSolVdmb93TavfAxk/N9N3TWVo4K2M4ON01Typ7EMQubf+eLGZKSrvp5KB3RHX+8KRsHGRyfG5iIFBAxWFJBVh3N1JA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QibGo4MW/YN1/Ym8QD+6gNLBohzygdhTPn189Gjm9CQ=; b=YBCe4DvCZT5Rd1YRHYZntjAn5LcKQWv6LEIFVEles6x3XbBqxxZ4jXqqTTYF40TDqrxv4WCq5MaGat46KOhBpaNjHiV+E9hmPTqhsKJXwDIItYIMPz3cYbbV8jNo/gwRdR/nb2nJXtt/s2qrmqWJdhkod9UGMrAU5P8fvXgqUdeKLYHaFXCGcK1B1z3A/2A8t3FO46GLmLy6Ynj1HB25OWozI9tY7c38hy4V1BFdwejKK9IRwSNd8udLXnHxYUSh7GmHJrB0aEtsojIyUHvNrOE5uStt854bd4V9/wjo7OZNSnPV/uc3dm7zPAGny7qWZlloNOv8eockCgCccyLAnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QibGo4MW/YN1/Ym8QD+6gNLBohzygdhTPn189Gjm9CQ=; b=ihalWmSiNSvhmGE/FOVrRsUso4GMBdHxnUBXblr9CBgzevLPjmAFcuWsUY1gr9fo6INJn0jib1M6Ify0sUTtUxPCLkgyRlZm8wgHMFzgf4vgOOCy3Z7PuGFpu9YUN+RsM42ZYmjWk8xwdn5APGqEIOXGr/WGsSJsN4UAgKUrdeA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by CH0PR12MB5284.namprd12.prod.outlook.com (2603:10b6:610:d7::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5880.16; Fri, 9 Dec 2022 16:15:57 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::b482:d5bd:c7d0:3842]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::b482:d5bd:c7d0:3842%8]) with mapi id 15.20.5880.014; Fri, 9 Dec 2022 16:15:57 +0000 Message-ID: Date: Fri, 9 Dec 2022 16:15:51 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.5.1 Content-Language: en-US To: Matt , Stephen Hemminger Cc: dev@dpdk.org, Hemant@freescale.com, stable@dpdk.org References: <20221109051339.1998037-1-zhouyates@gmail.com> <20221109060434.2012064-1-zhouyates@gmail.com> <20221109083909.6536bce8@hermes.local> From: Ferruh Yigit Subject: Re: [PATCH v2] kni: fix possible alloc_q starvation when mbufs are exhausted In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO2P123CA0065.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1::29) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|CH0PR12MB5284:EE_ X-MS-Office365-Filtering-Correlation-Id: 87fadfa4-bb54-4853-b449-08dada00a772 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Un4PdBs3C+pdypO0l8PQ3nkT740AufjJHeX0wnWFB/bVU5d91Rla8wmyx74ofL1hzOt/C9ej8rOv8AhDrtNrLll8Ufvb8PmvWvVqY2dKq9zXkxk9UpAYlCWJXS1H92ziYQ5PXfcp5m3b9UG1tIgNhbZvT/MdXX9UT1qsV1K/KzpUQqM5356SsCdmL/tQOqDlK6Kg6oeq5eP3HRtMRhoO0zDPlRYJhIqjcNbQ0ugO6xKL7ZyId7KkmXrZoPqVkPJJOH4k2Aph/Bowyeb1cliWHcGsqUtC+08CuFmvO7qkjGyOyr0bpmWm5RU5+L/mI7IpbYALp3Ku4jDPtYr65PqXBDHiiv1c7kac6Ady9Lfq2C5y3XYFj5yyxV3Ejis27Kt8UX8bo6kvZ8K6Fi7J6XsDKAq18Lt2w8t3gYxJgD2iUcWcM0Ruob0izN/f//2HaJendSdUkIk0Dv0vLKyE5XrMG/SRbqpWbQHwoxyq/PAXQ1iHr648emzs6z6INpa95rrH36T2h2OIcPuHhnyTg4rxFigF/UPWqj1E31OkKfc0Opit2Ib5enTo8ofAnN9its9N6VB4fjOpQAf5+K8K8lCEZk6t+ec1lwz0g5XYr3YSlI+L9oNRWAUSTvC6wvBdscK66d2+kOOjEm6E2W4Zg2mq51p/9k77ODGkqOSb3cCOLtpQ0Ca/kE3Gozeadq0hbzmN8FTgYeOWwNCAX8I73nBkJ3KcxpDs13kEhYrENLdR/jM= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(346002)(376002)(366004)(396003)(39860400002)(136003)(451199015)(2616005)(4326008)(66946007)(66556008)(8676002)(66476007)(110136005)(316002)(8936002)(83380400001)(44832011)(36756003)(2906002)(5660300002)(186003)(38100700002)(41300700001)(6506007)(6666004)(53546011)(478600001)(66899015)(31686004)(6486002)(26005)(6512007)(86362001)(31696002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?NVZsaEcxbDVRNHlIa3piNDlFRVVpNjBJWVhQZnAzQjRaYzRHU0FUYndGd0dE?= =?utf-8?B?ZVpJYXU0dVNsMEpVb2w3U3h5WWhRaW5Qcng2VVg2NElLdHB4NEgwakl4Snph?= =?utf-8?B?NEgyaUFnbGU5YytPWDdZV0FxemxqS0xvQ0svbC82M2FBTmVEaUJob0JBTGNx?= =?utf-8?B?ZFNtdGZPQ3pLU3ZnVTBkaDQ3ZDJvWldDQlpjb1NQaVF6ZG5KeW8rY1hkeXRz?= =?utf-8?B?eFFndVNVT2lSKzdzYlpPR2YwbDBUUko1Yzd4bEUrVnRDQ0prSVQyYXZVZXFD?= =?utf-8?B?eURseGpzcno0UnFtM3liWXdBekVRaTZUS0ZCd3BwaUIxZkRhN0gvczUwWkpU?= =?utf-8?B?SjZSS1BhdGt6OVVLcnVUQ2htRHlyQXpmdGdSQ3J3UCtWNW9XVkxzNjh0dzJE?= =?utf-8?B?VDNUTDNEek1EbngwK2pOQ1ZrVEVZcE5TdXhYWk5kendrUDZoayt5RnZqOEdq?= =?utf-8?B?TTVPYUFPMm80cThBeTI2dWdVekNJMWtISkQ2MjRUcFpMZjYrMHlZaTlRYzlj?= =?utf-8?B?QUtSS3dEcjFoaXZJKyszalRqbThHYUNhU0tVZHFQWi9WZkNKSWJDaTE5Mk9u?= =?utf-8?B?U0ZDSTJqMjJxZzRVTzA5cjNLUVkvOGN6OUhqMzVWUW9vdFRQUFVyV1lVY0Rn?= =?utf-8?B?VzBTTmRyTEJ1bEVuSnZPaWN4SkZyanRUUUtuQnJDazNtNFZlK1VnRFpiQWU5?= =?utf-8?B?aWE4ZUNOQnRldjJtTENzNFR6UjROTmQ5MDN5OW9aU0ZGSG1BbkQ0aE5XaUpZ?= =?utf-8?B?TWF3WjBhc092bmFWMW5jaUpvbzNxL3NZaDJXcjVDL1BvU3lVSktpaC9DQms4?= =?utf-8?B?OXQreE10ODcvcGxxMnhRYmhMRnd4dEM1bnQzNER4Mi9BWWJwT013bkJLbkxj?= =?utf-8?B?YjNHWlBTU3MzY3dwZ2NzOTNOQzBRc2RGbDR5VHdsOG5rU3BLMFRTSmpJbjgr?= =?utf-8?B?K1hBUkZOem1wbW9FekJvKzUrWVA4a2xwL1hCRHcwbmpDeURBL2RMTGx2Mkdi?= =?utf-8?B?elRaQnJNd1hndmgzaWRWdU93c0NUL3FKN2pQMDBBNGtUbmdpUmhTa0ZIOG9w?= =?utf-8?B?UEJieEl1SENiNnF5NFBHeml5bHZ5UzJkS1BFM1gvd1JJbkowTU01ZzY5S004?= =?utf-8?B?c1NuN00zZ2FCWTNqRVFYNFU3NW11L05oeHpYM3RKVGRXZEhZM1dWNzVheHlu?= =?utf-8?B?dyt0SndaMEkwbmpla0d2djdkWW9tTnRzb3VrVmhxeXYwNFFWT3dRVWFJbkU4?= =?utf-8?B?S2pHbXNQQUhQaEhzdHlaRUhZZW1sVlJRVHVZbldpUm1temh1UmYzVHQrc1BY?= =?utf-8?B?d0tjazFUelR0cU1PNnJHTVd0QTJxU2o2UElaRit5dkZYYzJ2S2dpVElXdWxa?= =?utf-8?B?T3JDSm94YnBWVkRiK2szeXQxaWUxd2RhSEI1UlRNTnJVRncvMHBXYTMxWUFl?= =?utf-8?B?TmdOYW8xV0Y2eVJuSk00VFo3QnIxeHQxM1Z6MTZRNGZRTjBrdE1UWGxGUy9C?= =?utf-8?B?NW1VOHM4VlJjemlxa0NGRzJBb3NBb1JwU3ptWEQ0R01aMFhXZ0JUMS9CWmt1?= =?utf-8?B?M2lUMW9PWkxxS2Q3SENtRjFRMUVpV3MxK1hoZ2FFVVE0RU1xN240Rlh6VDhI?= =?utf-8?B?M2JObktJZDBpZytlMFliZ0M2R3dIMS8vMm54cGRmWmhXdVM0RHZHbGFkVVZQ?= =?utf-8?B?a053QmIxR3o1WUxCL3JaK1dTdnhKVis1d1lOMHNCUzB5SUZUSXBCWmhPaUl3?= =?utf-8?B?aVpCRGMvZUtBTHhlb04rTWlUV1pqdWNaY3ZrUG1OVlNQN1ZjYVFKUTJSVUZL?= =?utf-8?B?cDVzcU5RMXVBcHd2L3ZGaExWQzJ3ekdJS3RpcDEwdnRLRi8zQ2J1MkZCU1Mx?= =?utf-8?B?b0FoUTFnVW0vVE9YTkFHalZaRk5iMlp5VUxFNG56NlZkcVBJbnU1Q3FMTEJK?= =?utf-8?B?QVNXMGxFSlI5L1JaSDc2NGlqZm5nYlUzbFJVZGJLeFNrTiszU0JyMGhSR1B0?= =?utf-8?B?blJvZ3V6UlU4dFBJMmxJUTZDWTNpdnViUVdMcHV1cDFJVmR3aXZQdkI2RnZr?= =?utf-8?B?Y2hqODdaU0pYRDNkdDl6UnpxSFZTdFFWQ1FPenZuSFFuQVdXaG5vUytkc2Jp?= =?utf-8?Q?vga/HyCYcNuLMuT4rxMz+NE9E?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 87fadfa4-bb54-4853-b449-08dada00a772 X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Dec 2022 16:15:57.3794 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ti4D/bJc4iHEPj6MWYMihKir8uz1oLYb56Oj4U2/GwdPyT/8TuaJvq70PetDhcG5 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5284 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 11/11/2022 9:12 AM, Matt wrote: > > > On Thu, Nov 10, 2022 at 12:39 AM Stephen Hemminger > > wrote: > > On Wed,  9 Nov 2022 14:04:34 +0800 > Yangchao Zhou > wrote: > > > In some scenarios, mbufs returned by rte_kni_rx_burst are not freed > > immediately. So kni_allocate_mbufs may be failed, but we don't know. > > > > Even worse, when alloc_q is completely exhausted, kni_net_tx in > > rte_kni.ko will drop all tx packets. kni_allocate_mbufs is never > > called again, even if the mbufs are eventually freed. > > > > In this patch, we always try to allocate mbufs for alloc_q. > > > > Don't worry about alloc_q being allocated too many mbufs, in fact, > > the old logic will gradually fill up alloc_q. > > Also, the cost of more calls to kni_allocate_mbufs should be > acceptable. > > > > Fixes: 3e12a98fe397 ("kni: optimize Rx burst") > > Cc: Hemant@freescale.com > > Cc: stable@dpdk.org > > > > Signed-off-by: Yangchao Zhou > > > Since fifo_get returning 0 (no buffers) is very common would this > change impact performance. > > It does add a little cost, but there is no extra mbuf allocation > and deallocation. > It is common that 'rte_kni_rx_burst()' called in continuous loop, I expect to try to fill alloc_q in each call impacts performance, even no mbuf allocated at all. > > If the problem is pool draining might be better to make the pool > bigger. > > Yes, using a larger pool can avoid this problem. But this may lead to > resource wastage and full resource calculation is a challenge for developers > as it involves to mempool caching mechanism, IP fragment cache, > ARP cache, NIC txq, other transit queue, etc. > > The mbuf allocation failure may also occur on many NIC drivers, > but if the mbuf allocation fails, the mbuf is not taken out so that > it can be recovered after a retry later. > KNI currently does not have such a takedown and recovery mechanism. > It is also possible to consider implementing something similar to > the NIC driver, but with more changes and other overheads. I agree this can cause starvation and recovery mechanism is missing, Following may be an option, but still concerned about performance impact and not sure about upstreaming it. Other solution can be to use larger pool as Stephen suggested. rte_kni_rx_burst() ret = kni_fifo_get() if (ret & kni_fifo_count(alloc_q) == 0) kni_allocate_mbufs()