From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7898742AF1 for ; Thu, 18 May 2023 16:56:49 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3F9EA40E25; Thu, 18 May 2023 16:56:49 +0200 (CEST) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2071.outbound.protection.outlook.com [40.107.244.71]) by mails.dpdk.org (Postfix) with ESMTP id 0CCE34014F for ; Thu, 18 May 2023 16:56:48 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZnfrGyT73SBuU13/ITyC6MVoaIgF63KFUnF1vaLODEbSxPwYu6xYdEQEQGix+42hHxI2UHrFzR63MnNqe/FMesVekWpUOlBGnlYjdwusYXcQ00UwbE/kQWVtMOrEEezhwM+5GksSWy6jdgJwJoNrzNcqLa1FzUcFI6Eb4E3lw1myBOXShSKvRFL3xS9apx1rAkSHoPq9kqMUrH9ByQPTQ7MJPsuCdMFkBEvTFelW1X9u3IvcRb6+rEInw5ZEgaCiyTCQ+z6/m/jyYuasEpR86Bfm7EkjrtACH6XpCpHNxKOat69zV9rULiIJ+g84ftfQGbA9wKGkITYKaQDabIhEFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=m7QxXP39KlkH+7D8ZylwlZjQ/z/RuialNhpjBYZ0PFs=; b=IZNBKOi9EBPBK52f8GJOUqu7/OMTZfvtaJ5YW7X9MQ987Y/kyPEJssHVlEg9yxlCasvtWmCD7y9HrIbBtsTvV4XzooBEQij+e1ENrRQcUun6hHT5sFqOS/fe4PQ0O95T7ndnVhqzDDovsnPY0djHHQ95j7EiIBuVFZKDKkTq9RSEB0c3vPF/k8fTmNVsJBU0Z/rdWWg5f3MgXRog/0Pfu0TqW1w61JIBpddvBqlp9oPtV5lupM65QZZcvDXgHP+bje1Pv3GCl16GhawLpNJy+/+PobuCrcsW+18vUFbuO7KN/8klrCMC8CdgpYoaIPJcjokMfw5OBaPRMLQcyYUgZw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=m7QxXP39KlkH+7D8ZylwlZjQ/z/RuialNhpjBYZ0PFs=; b=auhYO/UrxNfVN267iktlwC3T/AdLqE+jYr08THsYY9QWp5XJ4NnrA9+ljiA2ByfM92My+ogqup2imYOs/wb55RhLE1oU5lr8SHPjQYsfLrcRPD5L2BiNGzkPmAesFxtZRYXVdBXhoK9XVrYc5YvikvEa4SgLcpvIRrWOl7KiYmw= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by CH2PR12MB5516.namprd12.prod.outlook.com (2603:10b6:610:6b::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6411.19; Thu, 18 May 2023 14:56:45 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::7957:641d:6aba:3f9a]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::7957:641d:6aba:3f9a%4]) with mapi id 15.20.6411.019; Thu, 18 May 2023 14:56:45 +0000 Message-ID: <4f53f3be-0bae-e204-5737-7735b4a2ba5b@amd.com> Date: Thu, 18 May 2023 15:56:39 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: DPDK 22.11 - How to fix memory leak for KNI - How to debug Content-Language: en-US To: Yasin CANER Cc: Stephen Hemminger , users@dpdk.org References: <20230508091845.646caf17@hermes.local> From: Ferruh Yigit In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO4P123CA0526.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:2c5::12) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|CH2PR12MB5516:EE_ X-MS-Office365-Filtering-Correlation-Id: d2ada686-d895-400e-a721-08db57b01920 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rxTtzv+nmMAbN68LDw2SHaLg2s/RApJIoMDi8kfi7zC0TIFOrF8rztrMovJhcmD5FRSGO1+VLot9iIQyC2OtIE4ohvxNaPlR14Bt5w75o51T08P5MIv+/C9JvCiOzVZUXwBJtbvJES6ThdgeEPEXx73jiUmr3lb4tmyecCQfZa9P6v0LozyD4v5IE85L5GFrHQmAV18RN90Fg950RoshDLD54O0ohq8g1q2wPXi8WOtrbGcKLSqZu8Fx8IsOTDc0QIg8OmesOUoZBC64+vpOxpr5Gt30hucmuXLOt/fZFPnNAyNS97Lutpdja0EPap5IcPwf2M5kcVblOpfAhTUT/nWKpI2SXPcLZMJji1GDTJVDPfJvw2fDMCtLv8ccihd6e7juE+fT6hgMlcUTBZ1tcbMuG10Gzs30eoNvoEIlV1xM9GK9mdbloGO1XdszryacI2kl8KSMBnPuOuoQvuypgwv68xd50RJVB2oO5YeDswfbUshZqNCSzlFHbeejI3DpTJTO8VnCS496rfaHJzhHBwDTefnpe0G4NL6gLzG7VXF9gjsGe/KPjEolPVO1j1RysrYuOV/3JKhqMzWRj3xwXzTera2/DWF4lfyoX/Sklr0rURK2KhftHcSy/wtb8yPmDA24KhDtg7GST5afPwe4QMwl0GH1JIecU3WaWVm/ZYA= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(366004)(396003)(136003)(346002)(39860400002)(376002)(451199021)(66476007)(66556008)(6666004)(66946007)(31686004)(53546011)(6486002)(966005)(316002)(4326008)(6916009)(478600001)(41300700001)(5660300002)(186003)(44832011)(26005)(8936002)(6512007)(6506007)(8676002)(2616005)(66574015)(36756003)(2906002)(83380400001)(38100700002)(31696002)(86362001)(45980500001)(43740500002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?MytNdm1mK2tkNVgyeWdtUy80U3dSUTZvaHhTMFdCa1lHQTlVdm9zWER4bVlU?= =?utf-8?B?L0lubHNBTUNvazhmS2RudTg5L0U4dFR6alRocnJqMUhtQnlYYXNZOG15Mmlk?= =?utf-8?B?VXFCelN1dmt2TW5FbW1LU2FjK3NCNEt2YlBaK3Y3dXdFdHBIbmwyNXA5UmhN?= =?utf-8?B?R1RwVG1zaVJ4akFqRHVlUnhKemZJRFhML0NtSGdXWHpNY1c3eEFia0FHb1dM?= =?utf-8?B?Q1AyODd6STYxUW1ocnUyZFY4aFhOY25kbGFpUHkrTXdtWW8xcEFpV3FxM1Nz?= =?utf-8?B?QXAxM0p3c3d2YjUzL0NhUnlQZnBGMU15VFg5U3hzZmJFSFZxQ0RxUU1BN2J6?= =?utf-8?B?cXR3dzhyTlRzOUJPMy93Mk9xSjBvUDVHeXM0MFdYa0l1SERJMTUyYm01REFz?= =?utf-8?B?Vm1LS1dCQTVBdDJibWhmeU9YTDBrMXFVemp3NllOK3lyeEhmcjduaTZjNm8y?= =?utf-8?B?SEVGeXVvVzF1d0V6LzQ5cGhNVEIva2lSUkVtc3hKVlJyK0RVRWFlMzBwQTN6?= =?utf-8?B?Nms5cWhOY0ErNjBOZ2pYYWpuRnhLbjZjcEowUFF0b1Y1d29ZS1ZNd0k2T0U0?= =?utf-8?B?QVVJL1cxRzdKWEpoMTYrb2R6REU4b1d0QVFVeGFsMDYwU2tUSzh2blRiSDRT?= =?utf-8?B?SWNDNmF2bmdabnVXZWNFcEUvUXVJY3RURDhHbmp2OUJIelB0OGd3V0EwMGVK?= =?utf-8?B?UmZqOVZqQXFVTDUxeGVnV0sxMndiVnJhSU96ekZ3c0kxdklqT1pyMDR3dER6?= =?utf-8?B?eW5XMlgvSitUaW9lZzBWUytBMG9ZblZuOWxwRDNQakxsL1hmK1hZS2lXcmVs?= =?utf-8?B?ZktHNlo5T2tHWXlSZEZqbWRLQkEvZWVTK1lIRTJKN2JGQ3ZWWkQ2azB5d3Jz?= =?utf-8?B?RERqWWZiWjkwTTVENnY4VG9YTjFXQjVlWlNubjNxWTMvK0ptUk42b3lwSVVI?= =?utf-8?B?T3E5TTI1ZnVCZkFUL2lSazZMK2l1NC9YUjhFam9wbjdVaXNIQStIMVlXYWNz?= =?utf-8?B?a1VDc2owMXk4OE9Rb0NneXBTbUhMV2lDakozTENGWHh0VlBwT00rbFBxNmYy?= =?utf-8?B?N01hbnhKT0dLNHJ4TWFjSDZoVTU1SnVVS05obm1ZWVN6NmdqVlRvMzREZitT?= =?utf-8?B?aFlpamxBcm9LWE9QN2lYR1NBQ2tPZjduUEw0alhDdjlVOUZZV25KUzhkdDNl?= =?utf-8?B?bmg1UWU2RjdzWENkQ3VndVBIb2pkSmI3UGU0bDFWMGUvellnUkNJV3dHUHAw?= =?utf-8?B?NERaMlJISklOcVNiL3RwZ3N5TVptNnRQMU44SmxNZ3ppa3RIblZvMSt4K2cw?= =?utf-8?B?R1o1Y3phT21OanFDOHdkZ3V0Vi9WVC8zTFkvTmlweGxHSk9GRG95eEF0bXlS?= =?utf-8?B?VUlqTTF4czhDSm1OME93ZWd3dDBrNE90K01lRUtCQnEzRWZFeEZwWkpyZ1Fm?= =?utf-8?B?bXZDcWE3UUVoT2h3UXMwWTYvTEVreFFWSEgwUTFhSEI4NXdaRktRMy8zdGxr?= =?utf-8?B?Q281T25sWFJYd00xU3QxNk1EMjF5NXhWNHRlWkdNamIrQ1BCdDRsUXY0SkF1?= =?utf-8?B?NEhJWE1XS2IyckFYQlBRMXlhL2FSaFBMNTdpbUVNWWpDYW5icnRnRDdrU3Y0?= =?utf-8?B?VmtSNHBGeEhqc3VoRzRkaWoyeVk2MVRRdCtJUk0yclBNR3pIUVU3ZUpzUFRj?= =?utf-8?B?cXVTSUZGRDFPWXNwSkZBcDVsT0lnMjJWU3NKdU5qZjE2amJtYXQxMjUrZC9y?= =?utf-8?B?SDJkL2xQcEdEODlKem1sZkZSUVVBNXJRaWYzODdPWWdsVUZiZ1dHNWtxbWQ3?= =?utf-8?B?WEVMQUhpOTlkVStCemRUYnh2MGJGdm9wN0czUnVEV1VsVmNJZS9ocU8zUm5Y?= =?utf-8?B?YkYreERINGZJVUhhYWY0Z0hoOGwyMUtYQ1IzNldCdlBmN1Jsc0RqQUtzSnN2?= =?utf-8?B?MlRjVW8zWXk1NUxTamNyZmZFMUtWSktqeUUzR1hHaTBlVURaRE1ZdEIrVUZo?= =?utf-8?B?Z2E3T3RsU2NnRFZWY2R6M1llZ2JUYjhTVWJnOXlVK0thYXgrUERDYmVsNFF4?= =?utf-8?B?ZmRCcldhNlZlZUpSWEZWcHVLTi95WU05QTJBUWFBYmVoK0FCZ3grNVo3akxZ?= =?utf-8?Q?oSkIZ8RwWQlUALwSaQfnKAcge?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: d2ada686-d895-400e-a721-08db57b01920 X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 18 May 2023 14:56:45.2981 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aXlXh80K4R+gveHJAnbSgdoDLAMUBirdAklB1AbaebPIJ74z4WGKrRU3EJlnXHWy X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH2PR12MB5516 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On 5/18/2023 9:14 AM, Yasin CANER wrote: > Hello Ferruh, > > Thanks for your kind response. Also thanks to Stephen. > > Even if 1 packet is consumed from the kernel , each time rx_kni > allocates another 32 units. After a while all mempool is used in alloc_q > from kni. there is not any room for it. > What you described continues until 'alloc_q' is full, by default fifo length is 1024 (KNI_FIFO_COUNT_MAX), do you allocate less mbuf in your mempool? You can consider either increasing mempool size, or decreasing 'alloc_q' fifo length, but reducing fifo size may cause performance issues so you need to evaluate that option. > Do you think my mistake is using one and common mempool usage both kni > and eth? > Using same mempool for both is fine. > If it needs a separate mempool , i'd like to note in docs. > > Best regards. > > Ferruh Yigit >, 17 > May 2023 Çar, 20:53 tarihinde şunu yazdı: > > On 5/9/2023 12:13 PM, Yasin CANER wrote: > > Hello, > > > > I draw a flow via asciiflow to explain myself better. Problem is after > > transmitting packets(mbufs) , it never puts in the kni->free_q to back > > to the original pool. Each cycle, it allocates another 32 units that > > cause leaks. Or I am missing something. > > > > I already tried the rte_eth_tx_done_cleanup() function but it > didn't fix > > anything. > > > > I am working on a patch to fix this issue but I am not sure if there > > is another way. > > > > Best regards. > > > > https://pastebin.ubuntu.com/p/s4h5psqtgZ/ > > > > > > > > > > unsigned > > rte_kni_rx_burst(struct rte_kni *kni, struct rte_mbuf **mbufs, > unsigned > > int num) > > { > > unsigned int ret = kni_fifo_get(kni->tx_q, (void **)mbufs, num); > > > > /* If buffers removed, allocate mbufs and then put them into > alloc_q */ > > /* Question, how to test buffers is removed or not?*/ > > if (ret) > >     kni_allocate_mbufs(kni); > > > > return ret; > > } > > > > Selam Yasin, > > > You can expect 'kni->alloc_q' fifo to be full, this is not a memory > leak. > > As you pointed out, number of mbufs consumed by kernel from 'alloc_q' > and number of mbufs added to 'alloc_q' is not equal and this is > expected. > > Target here is to prevent buffer underflow from kernel perspective, so > it will always have available mbufs for new packets. > That is why new mbufs are added to 'alloc_q' at worst same or sometimes > higher rate than it is consumed. > > You should calculate your mbuf requirement with the assumption that > 'kni->alloc_q' will be full of mbufs. > > > 'kni->alloc_q' is freed when kni is removed. > Since 'alloc_q' holds physical address of the mbufs, it is a little > challenging to free them in the userspace, that is why first kernel > tries to move mbufs to 'kni->free_q' fifo, please check > 'kni_net_release_fifo_phy()' for it. > > If all moved to 'free_q' fifo, nothing left to in 'alloc_q', but if not, > userspace frees 'alloc_q' in 'rte_kni_release()', with following call: > `kni_free_fifo_phy(kni->pktmbuf_pool, kni->alloc_q);` > > > I can see you have submitted fixes for this issue, although as I > explained above I don't think a defect exist, I will review them > today/tomorrow. > > Regards, > Ferruh > > > > Stephen Hemminger > > >>, 8 May 2023 Pzt, 19:18 tarihinde > > şunu yazdı: > > > >     On Mon, 8 May 2023 09:01:41 +0300 > >     Yasin CANER >> > >     wrote: > > > >     > Hello Stephen, > >     > > >     > Thank you for response, it helps me a lot. I understand problem > >     better. > >     > > >     > After reading mbuf library ( > >     > https://doc.dpdk.org/guides/prog_guide/mempool_lib.html > > >      >)  i > >     realized that > >     > 31 units allocation memory slot doesn't return to pool! > > > >     If receive burst returns 1 mbuf, the other 31 pointers in the > array > >     are not valid. They do not point to mbufs. > > > >     > 1 unit mbuf can be freed via rte_pktmbuf_free so it can back > to pool. > >     > > >     > Main problem is that allocation doesn't return to original pool, > >     act as > >     > used. So, after following rte_pktmbuf_free > >     > > >    >   >> > >     > function, > >     > i realized that there is 2 function to helps to mbufs back > to pool. > >     > > >     > These are rte_mbuf_raw_free > >     > > >    >   >> > >     >  and rte_pktmbuf_free_seg > >     > > >    >   >>. > >     > I will focus on them. > >     > > >     > If there is another suggestion, I will be very pleased. > >     > > >     > Best regards. > >     > > >     > Yasin CANER > >     > Ulak > > >