From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0A0B8A00C2; Thu, 6 Oct 2022 11:38:11 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9F6BE42BD5; Thu, 6 Oct 2022 11:38:10 +0200 (CEST) Received: from NAM02-SN1-obe.outbound.protection.outlook.com (mail-sn1anam02on2064.outbound.protection.outlook.com [40.107.96.64]) by mails.dpdk.org (Postfix) with ESMTP id 6BE0D42BCC for ; Thu, 6 Oct 2022 11:38:09 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HZ4LHPw1MTomChVohaOeEgTigT3O/nGbpLmMKaCXg/HKgKoIwlwNMVmP+G+ESmVVrzZBbIDAqrPVQ3ejxd5utP7BbYgmWB3Rbufg2mK4pyvrCPw+k8/EAx1sAeR6hCglbICwdAhnqPjez9sHQW1AJSwngH408OZVP9GWwqRuZNGCQ2uKjJ3KOnm+7gNWEsuBHE7QLj8jtmDxLtgNynFwDjYPrROWXgOGrOzW1YeeIrwZz0duSmJGQhGb7G9KALvd0juvn78oU8iAaXXjcqG0uJgHQFwY2A0FoJOUt0N6IGjBY8llBRekz3Tzi/5MmfFVMv6yJo79l9/nJfJuaibWEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JBUrUD/m5rpw2oCy0UssoP/QATypIt1hdtftbZA/kKk=; b=Wv14M63/Y2lAZOsiJ9HTt2JiALWEt077arC1Jp24nr5bJJgOSTcncucclrPnTMi5W5SQVTm0gR6dJHUNZEz4CsVsva9rjnmdjQ7aZ63p2e1SJEqAiVSm0oy3//XmsSWUix4atymvdOdnVsyosG99qghvhQ5joaCAYDZ37KBedheSACIUl3Ph8pWYKGbmyJaZpvV9rlU+e8VIgBtvc1o/ZQN2lMrT+adrPGr2YqjnURLhTm5d5ODp64mijkTElcup1AbHM7/oK7ByTXue98ANSR5OuP1jo9wN7b0zB1FAM9kuovnfYauv699AgN/jIFnn7diXCxKJEf3NwNPIqDIgBQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JBUrUD/m5rpw2oCy0UssoP/QATypIt1hdtftbZA/kKk=; b=iS8Pq2d2zms2UrfuSBvipSgdaBfqK54tUIgPDAt2JcSnFLglJkI6QbvW7J37V/23Sk+dq247UvjHx/VQweFbtHaMPaAqmTCDfIDRVsZpmj03sMh4M2rA9c7UnEzAVV5YtFtvyWt07xMbH9ImNFIpEOv9p3jS5dXJlGsGYHkOn+o= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by BY5PR12MB4225.namprd12.prod.outlook.com (2603:10b6:a03:211::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.28; Thu, 6 Oct 2022 09:38:07 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::d07a:463f:6f93:337f]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::d07a:463f:6f93:337f%7]) with mapi id 15.20.5676.036; Thu, 6 Oct 2022 09:38:07 +0000 Message-ID: <04cf3c88-aa51-4fd1-c2fa-2641683f0ed1@amd.com> Date: Thu, 6 Oct 2022 10:38:02 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.3.1 Subject: Re: [PATCH 07/15] net/dpaa2: use internal mempool for SG table Content-Language: en-US To: Gagandeep Singh , "dev@dpdk.org" References: <20220928052516.1279442-1-g.singh@nxp.com> <20220928052516.1279442-8-g.singh@nxp.com> <8a537bed-d720-bf74-b11f-291d70dc8bb5@amd.com> From: Ferruh Yigit In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: LO4P123CA0576.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:276::23) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|BY5PR12MB4225:EE_ X-MS-Office365-Filtering-Correlation-Id: fcfc5314-aaea-4f5c-570c-08daa77e7942 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xLTR+83y74R0jY02arof7kVqSHjjZYmqHw2a7Pq01leHWkX+vydM1bvkVPFcpbEGTd+q15jUw+8/ObJmmSkqqJNnHMsrNT5Eki+sAE8MnZ0oU8iZs2KifVo/S6VMcGh6AMDVrYogFMqaRaH5UOCqT5O0e48TGVsAI04/W3jrrU4Vbs6ukCMTc7rvHdY5OmNsgg2YnkpkJRntnoAF4GtzfBts0jxMhJyOaKRn3T3lpox2W04kfolXNJLFu43EmiKOOHVYnvl+zSv+6JOFrcuJ+yGhYItNXFGleqgxtLBy9r6OltJY2ilD/eo8h/D1TnCE1Wx9/TDVKae5SmdL++h2Xkf5YWExxhU1RcpInxTX7xOv2VVoQ0va0VvNlqUG+oNF6KPSENIT7GzPYKNhUrC4Z6BlbGzYnrtmuIbxiEEbfetz2ySDH/l4OSxwfG5Ls7orF7SiST1Y6x9GKyibjXScSzZjfdyZsrKvSgP6OcDeYUAuvypHMzUT5jwiUcnJGRxqGDoO/K84yOc4e8oyXuDLPfgLcLzJQWmH0fzv4IgUpdgdtuejAOGLQnTXR8aUZEdDrqwzJSucz4N0yHfJpbME4Y+DBGt18BnRwZgs580r/M10BQrvG1H+leqUjZOo0XT00jrhYZwYm9dlOP5kCTnrGhZifUS5gUOxznmVoP+xV9OEYCbgNCdaCJXV0iVCFttoLWp5PbvTSGKbR0xglDqIffDx5x34eDEgcf6pBWJohSJWcIBJ6feD6DvUEJxKugsZTgyVGevmr3dt3u5M/0UJ+LnOruk6TCm4//asVNebYUM= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(136003)(346002)(396003)(376002)(366004)(39860400002)(451199015)(478600001)(6486002)(31696002)(66476007)(66556008)(66946007)(86362001)(6512007)(5660300002)(41300700001)(26005)(44832011)(186003)(2616005)(8936002)(83380400001)(38100700002)(2906002)(53546011)(6506007)(6666004)(31686004)(8676002)(110136005)(36756003)(316002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YmtxR1ovQnlKTS8rcHpYM3AvMkttdmViN3BycCsxQmsxdHU3Z1o2bXR5SjFl?= =?utf-8?B?R1czWnNkZ0x0aFJWdEcrSHErc2FWRE5saGlNSWZwV2dPZDd3WFp3ZjB4am5a?= =?utf-8?B?MHo4bkw5SGZIeU1VVjdwZmQvY2dTUStxUUlEVWZnOE8wY21yanZVQ0krbVUx?= =?utf-8?B?RFRUWk5mVkdmTUN4Q0w1R0xEeGozMEpBYjBYWW1qVSt3T0c5WmJ0c3ZSRFlL?= =?utf-8?B?L3kyVXVyNjI5R0RTNWl0T1YxTDI3bDJDZTFLWCtUSnVxTUVCTFJqelp6WHVW?= =?utf-8?B?TC84eUQwdTlldlpxOFNmTWttVndYc0dMUWVrOEd5UGlMeXhsVmNPbUZlV0kz?= =?utf-8?B?cld5ZFhiQllyNnd0eGpjK0tGNDA2eGFsL2Q1RDJEWW5SWTR3UGpTNDFIRlVw?= =?utf-8?B?eXBtQjFrbituSXQ0eWVoNFh0dGpRV2JydXRKeEtXSG9HcWZKV1QremFMZ05h?= =?utf-8?B?eVlQYnhSQUVYUktxVUltL1VUMldQVjFTb3d0OVlDeHhuQXpnSnl2NnhpTEh5?= =?utf-8?B?TGdGalhOYWx2b1d5TFhlVWZnNytnMHJkTHFkOTdKNmZHcS9iaU5idnI2a3FG?= =?utf-8?B?cWFXR3VDNlBab1RYYWlqSDRaUkRlc1lDc2FoSTFoWUYxSnZFb1VuNUZ3d0tr?= =?utf-8?B?K3ppajIwVnNFQXV5RzJWWU96bU9kRzZsVzNDVDUreVVRR0tpU2NTcmJ3bUNi?= =?utf-8?B?RlRGa1pwOSt3NS9QWUJVTGV2WWxPOFJNV3lZWjVKejJrVGJ4Uk0zQS9JZXVO?= =?utf-8?B?cm95LzEvNXhJUmFFWGFCbUNISlhrSFRHZktSamVZV0pPUzVlTnU1SWx6b0I5?= =?utf-8?B?cFBoamducjRsK3lPM0hkaFh4RTBodklPejQ0NUVvMTFmM3JBSXM3N0svcGdU?= =?utf-8?B?M05Wd3N2RnJBUFpmeE1nSmk2eFRsQWwva3A2Y2k0a1FNZU4reHUrVmhuSG1O?= =?utf-8?B?RWpnZnZFcWpzWXFYbUlsN1ZwREpEdlNtWXR0cXI1VldBNm9Jd3VWUmVrZ0Ex?= =?utf-8?B?TkpuT3pTOThid2VyU05VSTFwVHpnR3hlUEdvMElwSHY3ZmphaWFTVDc1WmQy?= =?utf-8?B?UFA1WTVUZEo3eEhIcmpUWVlTdXFIZGNyejNMcXJ4dDBRMWhMQk5NOXhHcjMw?= =?utf-8?B?TXl0ZEdDS2JMVE8xajRKZ210YWdKa2VoOEJ3Tks5REpJMjk3TldrUzlwTk83?= =?utf-8?B?N3FWbW4xU2x3RmxzZ2NOVGdQK1VaTi8rcFpKY1FKeG9MSVB1aC9UbGxXcTlI?= =?utf-8?B?aHpJNUV3L0xta1ZBcVNoM1hUZGNmVU9lcVc3WE5UTG9hdk95ZER3Qk53VFA4?= =?utf-8?B?cm5lMzdBcGZWVk5JM0orY1k2RmhUM294dEFGK2JYajVERFRSZEhoS2tHRGF4?= =?utf-8?B?WWZzd0tEUENjMzRMb053MEdjU0tNMTE2T01hdkhYOEZUeFdqS0tucjBWMTk5?= =?utf-8?B?LzYxWEJpSkJSSm9RMkZ5VUpGUTlvWmUzME9HbGd3SUYzREtaR05JR0hjRUJ5?= =?utf-8?B?YmlzS1pPTjBONUNCYkU0bFFqNUpkdWhTeTFHVFZmM2k5TmVrQytzRXdvT29Y?= =?utf-8?B?Z095b3BjNHFOUG5rdEhjaHBndTF5Rk5uV0NkbnNqc2hWQUwyMW9LKzI2T1Nq?= =?utf-8?B?WTJxeVpUV05qcElTZHBBSmFMUzhkb1ZwbDg1MURkdVdoR2ZSOUFXbmlhTkhR?= =?utf-8?B?TjFMb3pjTmp4emNNSi9jaXJ6VHRUZmJiYUpyWHpsV1MyMlRhL3dybWJiNlZZ?= =?utf-8?B?OVArVko0TGdnWDc5eFA0emR0MlB4ZzB0MGFmWWladGdLazhKT2tOa3V6RGYr?= =?utf-8?B?WWJHQnE1V2pSMnZhOXJsLzF4SldpOHNWRC9xKzgxWGtKYy9rSms5bWRvVm9k?= =?utf-8?B?b0dXOHBqcXZvREowbHRmcDRHanJhZWx2WHBkTVdXK3B0TS9VNnlrOEYvdnFK?= =?utf-8?B?SjdTRnEvVlNjUXhJZDZhbEFtWVVjQXJGL28vNW1VZldhSEZ2R2IzSkwzUEh4?= =?utf-8?B?WDVrdjc0VnQ0TXdmQ1YzeXVzUWM5MTFOZXFYU2x0S3RFaFdPNGRXOTloVkJx?= =?utf-8?B?OVowdUlUeS9KeGhiai9CNExIODUyQTNCai9wNjlmci9Kd3VkZ3FsZ1NlTWFv?= =?utf-8?Q?zeS9SgioemXQZuvM44h3g6sVs?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: fcfc5314-aaea-4f5c-570c-08daa77e7942 X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2022 09:38:07.0342 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +5IalkpKK3kymWrPCcNlqHfbiSNe/KRxATyVnfCUrYCm1j49xHNpz+yDqUYQcL4M X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR12MB4225 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On 10/6/2022 9:49 AM, Gagandeep Singh wrote: > Hi > >> -----Original Message----- >> From: Ferruh Yigit >> Sent: Wednesday, October 5, 2022 7:51 PM >> To: Gagandeep Singh ; dev@dpdk.org >> Subject: Re: [PATCH 07/15] net/dpaa2: use internal mempool for SG table >> >> On 9/28/2022 6:25 AM, Gagandeep Singh wrote: >>> Creating and using driver's mempool for allocating the SG table memory >>> required for FD creation instead of relying on user mempool. >>> >> >> As far as I can see this is in the Tx path, can you please explain why >> driver need an internal pktmbuf pool? >> >> And shouldn't mempool needs to be freed on driver close and release. >> >> Same comment applies for dpaa version of the patch (12/15). >> > > In TX path, in case of SG packets, driver need additional SG table memory to store the data > of all segments to pass to HW. Earlier it was dependent on the user's pool to allocate memory > which is wrong. so allocating the new pool in the driver for this purpose. > Agree to have driver internal mempool for driver internal requirements. > We are creating only one pool for all the DPAA net devices on first DPAA net device probe, so it can > be free only on last device probe. So will it be worth to free the pool on last device probe as > memory will be released on process kill as well? > Yes, it will be cleared but it is good discipline to free memory on quit/close. What do you think to allocate one per-port, and store it in device private data, it makes easy to free it when port closed or released? >>> Signed-off-by: Gagandeep Singh >>> --- >>> drivers/net/dpaa2/dpaa2_ethdev.c | 14 ++++++++++++++ >>> drivers/net/dpaa2/dpaa2_ethdev.h | 9 +++++++++ >>> drivers/net/dpaa2/dpaa2_rxtx.c | 13 ++++++------- >>> 3 files changed, 29 insertions(+), 7 deletions(-) >>> >>> diff --git a/drivers/net/dpaa2/dpaa2_ethdev.c >> b/drivers/net/dpaa2/dpaa2_ethdev.c >>> index 37a8b43114..c7aae70300 100644 >>> --- a/drivers/net/dpaa2/dpaa2_ethdev.c >>> +++ b/drivers/net/dpaa2/dpaa2_ethdev.c >>> @@ -78,6 +78,8 @@ bool dpaa2_enable_err_queue; >>> #define MAX_NB_RX_DESC 11264 >>> int total_nb_rx_desc; >>> >>> +struct rte_mempool *dpaa2_tx_sg_pool; >>> + >>> struct rte_dpaa2_xstats_name_off { >>> char name[RTE_ETH_XSTATS_NAME_SIZE]; >>> uint8_t page_id; /* dpni statistics page id */ >>> @@ -2907,6 +2909,18 @@ rte_dpaa2_probe(struct rte_dpaa2_driver >> *dpaa2_drv, >>> /* Invoke PMD device initialization function */ >>> diag = dpaa2_dev_init(eth_dev); >>> if (diag == 0) { >>> + if (!dpaa2_tx_sg_pool) { >>> + dpaa2_tx_sg_pool = >>> + >> rte_pktmbuf_pool_create("dpaa2_mbuf_tx_sg_pool", >>> + DPAA2_POOL_SIZE, >>> + DPAA2_POOL_CACHE_SIZE, 0, >>> + DPAA2_MAX_SGS * sizeof(struct qbman_sge), >>> + rte_socket_id()); >>> + if (dpaa2_tx_sg_pool == NULL) { >>> + DPAA2_PMD_ERR("SG pool creation >> failed\n"); >>> + return -ENOMEM; >>> + } >>> + } >>> rte_eth_dev_probing_finish(eth_dev); >>> return 0; >>> } >>> diff --git a/drivers/net/dpaa2/dpaa2_ethdev.h >> b/drivers/net/dpaa2/dpaa2_ethdev.h >>> index 32ae762e4a..872dced517 100644 >>> --- a/drivers/net/dpaa2/dpaa2_ethdev.h >>> +++ b/drivers/net/dpaa2/dpaa2_ethdev.h >>> @@ -121,6 +121,15 @@ >>> #define DPAA2_PKT_TYPE_VLAN_1 0x0160 >>> #define DPAA2_PKT_TYPE_VLAN_2 0x0260 >>> >>> +/* Global pool used by driver for SG list TX */ >>> +extern struct rte_mempool *dpaa2_tx_sg_pool; >>> +/* Maximum SG segments */ >>> +#define DPAA2_MAX_SGS 128 >>> +/* SG pool size */ >>> +#define DPAA2_POOL_SIZE 2048 >>> +/* SG pool cache size */ >>> +#define DPAA2_POOL_CACHE_SIZE 256 >>> + >>> /* enable timestamp in mbuf*/ >>> extern bool dpaa2_enable_ts[]; >>> extern uint64_t dpaa2_timestamp_rx_dynflag; >>> diff --git a/drivers/net/dpaa2/dpaa2_rxtx.c >> b/drivers/net/dpaa2/dpaa2_rxtx.c >>> index bc0e49b0d4..dcd86c4056 100644 >>> --- a/drivers/net/dpaa2/dpaa2_rxtx.c >>> +++ b/drivers/net/dpaa2/dpaa2_rxtx.c >>> @@ -403,7 +403,7 @@ eth_fd_to_mbuf(const struct qbman_fd *fd, >>> static int __rte_noinline __rte_hot >>> eth_mbuf_to_sg_fd(struct rte_mbuf *mbuf, >>> struct qbman_fd *fd, >>> - struct rte_mempool *mp, uint16_t bpid) >>> + uint16_t bpid) >>> { >>> struct rte_mbuf *cur_seg = mbuf, *prev_seg, *mi, *temp; >>> struct qbman_sge *sgt, *sge = NULL; >>> @@ -433,12 +433,12 @@ eth_mbuf_to_sg_fd(struct rte_mbuf *mbuf, >>> } >>> DPAA2_SET_FD_OFFSET(fd, offset); >>> } else { >>> - temp = rte_pktmbuf_alloc(mp); >>> + temp = rte_pktmbuf_alloc(dpaa2_tx_sg_pool); >>> if (temp == NULL) { >>> DPAA2_PMD_DP_DEBUG("No memory to allocate >> S/G table\n"); >>> return -ENOMEM; >>> } >>> - DPAA2_SET_ONLY_FD_BPID(fd, bpid); >>> + DPAA2_SET_ONLY_FD_BPID(fd, >> mempool_to_bpid(dpaa2_tx_sg_pool)); >>> DPAA2_SET_FD_OFFSET(fd, temp->data_off); >>> #ifdef RTE_LIBRTE_MEMPOOL_DEBUG >>> rte_mempool_check_cookies(rte_mempool_from_obj((void >> *)temp), >>> @@ -1321,9 +1321,10 @@ dpaa2_dev_tx(void *queue, struct rte_mbuf >> **bufs, uint16_t nb_pkts) >>> >>> if (unlikely(RTE_MBUF_HAS_EXTBUF(*bufs))) { >>> if (unlikely((*bufs)->nb_segs > 1)) { >>> + mp = (*bufs)->pool; >>> if (eth_mbuf_to_sg_fd(*bufs, >>> &fd_arr[loop], >>> - mp, 0)) >>> + >> mempool_to_bpid(mp))) >>> goto send_n_return; >>> } else { >>> eth_mbuf_to_fd(*bufs, >>> @@ -1372,7 +1373,7 @@ dpaa2_dev_tx(void *queue, struct rte_mbuf >> **bufs, uint16_t nb_pkts) >>> if (unlikely((*bufs)->nb_segs > 1)) { >>> if (eth_mbuf_to_sg_fd(*bufs, >>> &fd_arr[loop], >>> - mp, bpid)) >>> + bpid)) >>> goto send_n_return; >>> } else { >>> eth_mbuf_to_fd(*bufs, >>> @@ -1646,7 +1647,6 @@ dpaa2_dev_tx_multi_txq_ordered(void **queue, >>> if (unlikely((*bufs)->nb_segs > 1)) { >>> if (eth_mbuf_to_sg_fd(*bufs, >>> &fd_arr[loop], >>> - mp, >>> bpid)) >>> goto send_frames; >>> } else { >>> @@ -1810,7 +1810,6 @@ dpaa2_dev_tx_ordered(void *queue, struct >> rte_mbuf **bufs, uint16_t nb_pkts) >>> if (unlikely((*bufs)->nb_segs > 1)) { >>> if (eth_mbuf_to_sg_fd(*bufs, >>> &fd_arr[loop], >>> - mp, >>> bpid)) >>> goto send_n_return; >>> } else { >