From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id D5A64A051C for ; Fri, 26 Jun 2020 16:17:00 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BC7C21C012; Fri, 26 Jun 2020 16:16:59 +0200 (CEST) Received: from EUR04-VI1-obe.outbound.protection.outlook.com (mail-eopbgr80092.outbound.protection.outlook.com [40.107.8.92]) by dpdk.org (Postfix) with ESMTP id DDD811BFA6 for ; Fri, 26 Jun 2020 15:55:25 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HsOk9hcQ36Vb4CSRfhPJzCkfIB3mNRZsPrvXaEldGUNd35gfDI+UXLe0k87xuLnYtlCL23cyIpuuO2XBludmo46KJP+rZHW0yFc4lA6GQDqhrqIuvQO3o2AbXn/wagbatyi6SBNb12OT5utNYP2RJAaAymdJ+TSsanBva4jKpWjD+fDbPOMnx6782+F1a6MUTB4pGb1objbGdSv1YUgLDdqaahok1GAYMCFjFx/09CGT8gOoWb9nacYS1i7NrcBg6ZvOM0fRDAtabKbGGHo/1CbkWNDp5TQeuVPQUQTPckqjRj+/eumDZ9C2cJRkIfRgZNgPOrIiLSLOXAm417EhIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4pXPvun85p0dEn8KOXYpxrO+RSTdgGZpDSFH8rwpuQs=; b=O5drF5NGgAIOXWgtoWlnvgCT9HZIrAfLGaBcj0hc+4F5I25KDt0VLcUV4fQ3ury7vPTP4e48XF6PktCZ0+Z0+aqxRHruhsh7w045hyPH11T+XI1KMvKEG0dmk/C3IQPdzqQ//T7HvCrIv7jJ3AX7ZLtB68R6guH+fy2F2ntwf1ZDLkDeWG8QLISJnTsPQdLDMJTUxyC/M0W9qmsl3JnD4CdkrLBvrkGRtf+Oryg8mMOKn03D1uSnozJ/SQ1CEKRuh5UnFC4VKWPPEVZ9oNE2cTBJ3tyLxSRAxSwdkKcZJ7r96aSosTPEXkfngkDSYR2XnoZpQyJICQNQCewMXp/B+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nokia.com; dmarc=pass action=none header.from=nokia.com; dkim=pass header.d=nokia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nokia.onmicrosoft.com; s=selector1-nokia-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4pXPvun85p0dEn8KOXYpxrO+RSTdgGZpDSFH8rwpuQs=; b=Ffr9vu/qeQT8YCWTaLVVbAJ2pUi+KpS+5w5GNVMfrNiWGHU8brnPTjNulqqfEBw3fhzkjnFthmkdt18u0c/7UMiP1xvpJftQL523ZESKs9pnb0ZYd2omJ22H7Csy1qm9Ji9E8wf4aOV450CS9AnOOPP5g1+Kq4kpcLHeaGknyc8= Authentication-Results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=nokia.com; Received: from AM6PR07MB5605.eurprd07.prod.outlook.com (2603:10a6:20b:8c::18) by AM7PR07MB6545.eurprd07.prod.outlook.com (2603:10a6:20b:1a5::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.10; Fri, 26 Jun 2020 13:55:25 +0000 Received: from AM6PR07MB5605.eurprd07.prod.outlook.com ([fe80::5d48:4287:214b:f172]) by AM6PR07MB5605.eurprd07.prod.outlook.com ([fe80::5d48:4287:214b:f172%3]) with mapi id 15.20.3153.013; Fri, 26 Jun 2020 13:55:25 +0000 From: Julien Meunier To: users@dpdk.org Message-ID: <9120fdb0-a10b-e0e0-cfa9-daa104c31359@nokia.com> Date: Fri, 26 Jun 2020 15:54:53 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-ClientProxiedBy: MA1PR0101CA0056.INDPRD01.PROD.OUTLOOK.COM (2603:1096:a00:20::18) To AM6PR07MB5605.eurprd07.prod.outlook.com (2603:10a6:20b:8c::18) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [172.30.9.6] (131.228.32.166) by MA1PR0101CA0056.INDPRD01.PROD.OUTLOOK.COM (2603:1096:a00:20::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3131.20 via Frontend Transport; Fri, 26 Jun 2020 13:55:23 +0000 X-Originating-IP: [131.228.32.166] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 12fb9152-1ca6-4557-722c-08d819d89367 X-MS-TrafficTypeDiagnostic: AM7PR07MB6545: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-Forefront-PRVS: 0446F0FCE1 X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: O3ehbqAj86HKy7senIQfRnHB8xqFSrMJcjmFvnNLE2KueM0Wb1hudGGO58iOtQ/qofcYlqQKotaJfNRWPbFrFBqp5Qx7VaDSivcKL2p8AKsIX25BwcCZogRf7LFujla6bhJZAzlg7Fqnv+6xc/dkq9J9uhWNOhnOdqgFIyhUUWbRpRvVtjn2dg1SIi9at6JKslK+lO+jPczGFdI0iLdgEcDULpNEnWtiHR7Hkdx0Y+H4BGIfHr+IiaR1vYX6WqaN8dlrh8pxwDtwAoPPlydWrfoaUaJ1pTQG4Ow3wjq+AM4MzUC7mh1zfa1YMrqIt7t3DYlwuMonBsUxjlC8/1HOhBUKfRo3f42CEhiV0qIUGNsbdN7b1QQu7OQPg424SRCiwLecxLE84Ear+zKOeroCtWBTD8sUZyCJC/01one3AK18L6nPtSFgsREcp2EF5amBkhsesnWmxtYKwWyPnuyBTw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM6PR07MB5605.eurprd07.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(39860400002)(376002)(396003)(366004)(136003)(346002)(6486002)(31686004)(52116002)(966005)(478600001)(8936002)(6666004)(6916009)(2616005)(8676002)(956004)(44832011)(31696002)(83380400001)(316002)(36756003)(16576012)(66556008)(86362001)(16526019)(66476007)(2906002)(186003)(66946007)(26005)(5660300002)(43740500002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData: KlLxnR9JD2J7Ogg4JVc4RAwmwX3p36nweXxqsgT5FVZ3+GlA8H+4ae/nnXpdAmxfmvhiqsA7ZtQEkkqjr4MxjSThYXUKrzGBmuCJioABSPUlV+EUaj8AGBmbMRmSAaQy0xJif0UFy2RLpTaBCLIUmw04WrNvHgNA/OAv2nsNTvmouATZ5Eqnq38j7/w8oeb1KHn8eheT1SagQHIGOZHW44dNEOkJ9kZ9/U2rCsmYNZZuAlfoOD2z3t7IsKR6RkF6/13RPQ+3IQlAv/Hv+xlR9KuJgd/+9R8yTfykxjwx3edy+qtLZEwLmzyvv2IfIi/sc/B1TcOrlOVwBmQCrYoU/y1ew5UikMHQcwiQEanBoPKvxQSLOX/YI+U0OkdhyFCtHxOVT+uC9EAHlAs+L4J7iyATKbXjpoeJa3oXWDrECREsFinFz540JI4rRdV69gF1g2i4QaNC4hBqUax8fVOLCU/53NWcg54WqK8YGY6FR1S+YX7BKBLx2G7ZHf5xINuY X-OriginatorOrg: nokia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 12fb9152-1ca6-4557-722c-08d819d89367 X-MS-Exchange-CrossTenant-AuthSource: AM6PR07MB5605.eurprd07.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Jun 2020 13:55:25.0575 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 5d471751-9675-428d-917b-70f44f9630b0 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: q19VxwDt9saWsTOvZ9g7soh7em3pesHywLDp+JjGIiFe4bBk0wLXx0E6M9Vd2/jomZFLkZ+nhRGjBgIfz5XEHNBTR7qF5I3DvXHVUQfNtmY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM7PR07MB6545 X-Mailman-Approved-At: Fri, 26 Jun 2020 16:16:57 +0200 Subject: [dpdk-users] ethdev: issues with tx_free_thresh + ixgbe X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hello DPDK community, I have a DPDK application running on a system with a high memory constraint: each applications have a memory budget to respect. So, the number of mbuf in the mempool is calculated in order to match this constraint. However, I saw a strange behavior regarding the way that the PMDs free the mbuf sent in the network. To simplify my explanation, I built a small DPDK application which only sends packets (but testpmd can be used I think). I used that on top of one VF (fm10kvf or ixgbevf). The VF has 1 RxQ / 1024 RxD (not used) / 1 TxQ / 128 TxD Also, let's say : 1 mbuf = 1 descriptor. =========================== First point: tx_free_thresh =========================== According to the DPDK programmer guide in section 11.4.5. ``Configuration of Transmit Queues``:: The minimum transmit packets to free threshold (tx_free_thresh). When the number of descriptors used to transmit packets exceeds this threshold, the network adaptor should be checked to see if it has written back descriptors. The default value for tx_free_thresh is 32. This ensures that the PMD does not search for completed descriptors **until at least 32 have been processed by the NIC for this queue.** However, in the DPDK headers, in ``rte_eth_txconf`` struct:: uint16_t tx_free_thresh; /**< Start freeing TX buffers if there are less free descriptors than this value. */ And in the docstring of ``rte_eth_tx_queue_setup``:: - The *tx_free_thresh* value indicates the [minimum] number of network buffers that must be pending in the transmit ring to trigger their [implicit] freeing by the driver transmit function. After a code review and tests on target (fm10kvf), my understanding is: * tx_free_thresh is set to 32 - if I sent 32 packets, all mbufs are locked in the TxQ. - if I sent 33 packets, all mbufs are locked in the TxQ. - if I sent 96 packets, all mbufs are locked in the TxQ. - if I sent 97 packets, PMD tries to clean the TxQ. * tx_free_thresh is set to 128-32=96 - if I sent 32 packets, all mbufs are locked in the TxQ. - if I sent 33 packets, PMD tries to clean the TxQ. Is there any misunderstanding with the DPDK Programmer Guide or in the doctring ? Should ``tx_free_thresh`` be defined as the following one ? This ensures that the PMD does not search for completed descriptors until at least 32 descriptors **are still available** by the NIC for this queue. ================================ Second: tx_free_thresh and ixgbe ================================ My application is running on two platforms. One with fm10kvf, one with ixgbevf. I did the following tests: - fm10kvf / no offload / TxD = 128 / tx_free_thresh = 96 => after 33 packets, PMD tries to cleanup mbuf, as expected. - fm10kvf / offload (TX multiseg ON) / TxD = 128 / tx_free_thresh = 96 => after 33 packets, PMD tries to cleanup mbuf, as expected. - ixgbevf / no offload / TxD = 128 / tx_free_thresh = 96 => after 33 packets, PMD tries to cleanup mbuf, as expected. - ixgbevf / offload (TX multiseg ON) / TxD = 128 / tx_free_thresh = 96 => after 33 packets, all mbufs are locked in the TxQ. => after 97 packets, all mbufs are locked in the TxQ. => after 128 packets, only the first mbuf sent is freed. I did some analysis in this PMD. The TX function is not the same when offload is enabled or not (ixgbe_set_tx_function) - when no offload is used: * TX function is ixgbe_xmit_pkts_vec * ixgbe_xmit_pkts_vec correctly manages the tx_free_thresh and calls ixgbe_tx_free_bufs in order to free the mbufs. - when offload is enabled: * TX function is ixgbe_xmit_pkts * ixgbe_xmit_pkts calls ixgbe_xmit_cleanup (instead of free), and seems to only manage internal pointers. * Free is done only when ixgbe detects if descriptor was previously used: http://git.dpdk.org/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n890 To sum-up: - when offload is disabled, TX ring can be cleanup quickly, and it enhances the mbuf circulation inside the application. - when offload is enabled, TX ring cannot be cleanup quickly, and at the end, all mbufs are locked in the TxQ. I had a usecase with 6 VFs / 4 TxQ / 1024 descriptors. After few seconds of test, 6 * 4 * 1024 = 24 576 mbufs were unavailable, because locked in the TxQ. As our mempool is quite low, this behavior cannot be handled, only because the PMD implementation is different. Is it the expected and wanted behavior for ixgbe with full TX features enabled ? Did I miss something in my configuration ? Thanks in advance ! Best regards, -- Julien Meunier