From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 9D6DCA0C4D; Mon, 8 Nov 2021 14:10:41 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 854D341137; Mon, 8 Nov 2021 14:10:41 +0100 (CET) Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by mails.dpdk.org (Postfix) with ESMTP id C669141122 for ; Mon, 8 Nov 2021 14:10:39 +0100 (CET) X-IronPort-AV: E=McAfee;i="6200,9189,10161"; a="295659513" X-IronPort-AV: E=Sophos;i="5.87,218,1631602800"; d="scan'208";a="295659513" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Nov 2021 05:10:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.87,218,1631602800"; d="scan'208";a="491208446" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orsmga007.jf.intel.com with ESMTP; 08 Nov 2021 05:10:38 -0800 Received: from orsmsx609.amr.corp.intel.com (10.22.229.22) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Mon, 8 Nov 2021 05:10:38 -0800 Received: from orsmsx604.amr.corp.intel.com (10.22.229.17) by ORSMSX609.amr.corp.intel.com (10.22.229.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12; Mon, 8 Nov 2021 05:10:38 -0800 Received: from ORSEDG601.ED.cps.intel.com (10.7.248.6) by orsmsx604.amr.corp.intel.com (10.22.229.17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.12 via Frontend Transport; Mon, 8 Nov 2021 05:10:38 -0800 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.176) by edgegateway.intel.com (134.134.137.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2242.12; Mon, 8 Nov 2021 05:10:37 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QMUuowY3ozcso4bqRfwttStwK07/Q/YrFrR/hIqyPH07K+URbQBsF4WuOUbkTXPBmGwDesTHouVMWWb3MO8EqV0XhTMMtxDqQ2LedHKq/KvDNEYjApRijBvO+mf8XBAhLeIKMmdesCIvf1AJrFEFihLLWAq3dsUlGXQM0+C6SiVLZRc3ktYzs+YiyKXHwZB5n72+XELxm0c4TmEfSn92b9USB5fgGUtDxf7aGcmeU60l+V2c3rPJycPbRevpJ3vCJP1b+mbHBdsBGvKJDLavMa2EOiv2tpVjCfQg6+7eugXdEeDM4mi2vTWGNPs+s/T9QpK70Ra91xZ3FkyvxkcPxg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=a/0AYN1iitVKqTENxQQhXHsySJPHbHOK/lkeoc0OZ18=; b=TY4pt40p1+Hf7CmUr/5v/JnFuONqn9IUlycwyChjz+rXxiKFP0uiIakZ5BV3KNvaGif+smgQWUZrhQXr3rBnUhNYiEV5Gs9WHX9gsEvXqxLcAlX6jGiD/I+1CenodQXdL2gEGw/qI68vHv0YQMZFCYXymzSE/8Eu0eb0rEXLU5Wevu7ckIBDmFAUD7p0mLVFWKtPlGXgz527BpUJ6OIM0UNhsuPw4FyPYWnDxqVlihFBsL/23bZpWeTdDjVRP3p7QHDL2/JF0e4yyeRDCSx+EyWCSVyuODM1ZIJ6e6tDmDmCFriHLGF1EPWVQSTTCGFKXrg6uraq70+q8YYzAPyuJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a/0AYN1iitVKqTENxQQhXHsySJPHbHOK/lkeoc0OZ18=; b=kwdnNtbSjkjN8DjJkZOw/VGzWGiJHkskLLumDh6d9Hcd0nyXeqTcFYOGNXVtNgUb0e2d56G6+07aD8tQguITHDT1U7m05TCK5JN8Zk2cG2i8oZ8x8nYPhSzYrHBMW9r502sGjG4b5xfCyBpp/RmeJJ76i+MDpXjHAuBwn/JQR6M= Authentication-Results: oss.nvidia.com; dkim=none (message not signed) header.d=none;oss.nvidia.com; dmarc=none action=none header.from=intel.com; Received: from PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) by PH0PR11MB4806.namprd11.prod.outlook.com (2603:10b6:510:31::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.11; Mon, 8 Nov 2021 13:10:36 +0000 Received: from PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::bd7d:29be:3342:632c]) by PH0PR11MB5000.namprd11.prod.outlook.com ([fe80::bd7d:29be:3342:632c%7]) with mapi id 15.20.4669.016; Mon, 8 Nov 2021 13:10:36 +0000 Message-ID: Date: Mon, 8 Nov 2021 13:10:29 +0000 Content-Language: en-US To: Dmitry Kozlyuk , CC: Raslan Darawsheh , Matan Azrad , Viacheslav Ovsiienko References: <20211101095646.840095-1-dkozlyuk@nvidia.com> From: Ferruh Yigit X-User: ferruhy In-Reply-To: <20211101095646.840095-1-dkozlyuk@nvidia.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO4P123CA0466.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1aa::21) To PH0PR11MB5000.namprd11.prod.outlook.com (2603:10b6:510:41::19) MIME-Version: 1.0 Received: from [192.168.0.206] (37.228.236.146) by LO4P123CA0466.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:1aa::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10 via Frontend Transport; Mon, 8 Nov 2021 13:10:34 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1db6009c-5e43-44b9-8e3c-08d9a2b92717 X-MS-TrafficTypeDiagnostic: PH0PR11MB4806: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2089; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1hdcRTiP///+DFLyth+dJ5VI3E1gKQPsNizzo2c2MkkNkJ1VApeYIlZRuwMvuKNA5V9pLR/Sz2jja099qzlG33g36X/Q9DHxkfY7etrywlZBzTiWJOSE1OtYOqsHe4nWFmagUhtsrYSsBjj2fPUCUtqaAgGIXQRNThC0DTUcnOk7evsHKecTGYNmP3XXBqWGEPDgdAYSYmqDQw4YO7KHiNk7drtHT1hx5bRX7tjtHqHrsGUNi1p19OEO860nGBRH15qHuscOBHLmq/oAuu6Oj27ukSf3xrT60uYsXKlIewhpTKLYkKIjR/c4HmangJXp0CCHfJbwQJQ7J1M2De4whlX/9G+2bXZC0/LVanXQ4M2plNS8l0kU5VBKls2OEiF6pBppUkLICkTQGjeQbwjQCYdYPSInkfK4BoU5nUuP17z+2MyBJH5FoOk+F5M5EflD+TApKrCAfdjguuyEXb51fuR4Y6U7YyegXOPniq7hg6VeeJw5N1+IImmjFxUB8HOj7EiODXi1FSuYEHQuUc3cZh9XJIJcjLv38pt0BlLjYIZbWIdOQ3kukVKCK9EJf2RyTq3O/Gmwz2RaHtf36bLcDrmlagaF6BrCo1kptwSUfLOX1qWuNWcLA3YAFFtnEL/qrD8r2+T9lIZ478dC8rcKpU4rQVUEca3Am4gX0l/kcSOcV7xqZHQr5THqvM/KNadh6Qs1bnu56yI+Ngll9JwSbzYod1SiFbhRkqIGLeJXiYI= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB5000.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(38100700002)(4326008)(186003)(2616005)(956004)(508600001)(83380400001)(66556008)(31686004)(53546011)(36756003)(8936002)(26005)(86362001)(316002)(2906002)(5660300002)(66946007)(6486002)(66476007)(6666004)(8676002)(44832011)(82960400001)(16576012)(54906003)(31696002)(83323001)(45980500001); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aWxWT1ViVFh0ejhWU3dqYnZNdkhSQlZrNG9pZzNoam9wbGYwanBCK1paMnB6?= =?utf-8?B?bmd1S1A2c0ZiWWFGcWk3QmFXQkI3VDRmaEFGdzV0R1hHOHd2aEt4WWlpRlBZ?= =?utf-8?B?T0dsZjFVZHdGV0N6SEc3Z0lvZElFRHl2V1RUay9UN256SE9tdnRFL2hKR0Iz?= =?utf-8?B?RmhCMVdEQ29RWG9JQWNpczBBT0RRT1p6dU9lOU9CODVpRnR0UzVSbkphSzRp?= =?utf-8?B?dHd0djFzRWpVUlFBN3QxTGY2bHZoUUROYUd0aXYyQ0x4aDRtdktoRzBLWDlp?= =?utf-8?B?WnhKOGY0K1JHdFhJVVhSR0JRRWJ3cGRHRldleTdWN0toVXZMSjNxUmh5SGdH?= =?utf-8?B?STFENjJCU3F6NVQ3NjdScnRodmgxZnhjV3haT0xaQ1lNa3pycER0b2M0b3dD?= =?utf-8?B?T1duQU9aZ0htYm54Y01Zc28zaDV2c0pyeTNDWFdhWmdhN0pHWHlwSDZKYmFN?= =?utf-8?B?SUpQellpa21BOW5xQ3dqenM4T3paTEgrYjMvN3AxNjhJNnRiUHNSY0RFeVAr?= =?utf-8?B?c091M2FCb2NWeUwzVUhjM2dYUEl2SVJ1WWVIKzRiY1prVitXVExHakNYQWM2?= =?utf-8?B?TGU2U3JlZi9aV1lFcUFBV1lTTlhqWFVqSGd4VXdyd1pYVGo3ek1KMHJMdGpo?= =?utf-8?B?Q3AwSlQrR0JxTThsZlV3WWhJNlJ0M2xnaFRhUXQ4SjRwUmJ2V3grY2JxeE5S?= =?utf-8?B?UjhhUUdFYWJiR3llZzhuLzhRQmhvQWxzcFRMVmtBb0NPYm9zZVY0bVE5L0tK?= =?utf-8?B?bENTY2xjT3dpRkVjNWhQK0xSY0c1QjlDdHc1RU8rdlJFcXYwL2NhUUFZcDU5?= =?utf-8?B?c3lDTm1hc0EwYVJSR1QwRzIvZGNmYi9tREFYcEVNY0UxMWVEZG50Rmd6V3dH?= =?utf-8?B?NWcyWm5GMEdsOXpDK1psckdUUWlOTWZiTkxrOGo2Q3J2dnVacklaNkpoOWdT?= =?utf-8?B?eWZrVVJVVkJ0OCs1bTFOY1VoL3RYRDhNckZZcE12b0h1ZXhuZDAyTWFvZTVC?= =?utf-8?B?clZjNWZVQXZJUmJwaUp1TGdTUEdMRjJKOEdkMG96b2kyVDc2M290eDBETjlr?= =?utf-8?B?cEwxR25MNVEwYk81V1g3YXRsUk5Wc1hPQmNBQUU1M2pSRHNrZ3RWa0JnM3Z2?= =?utf-8?B?NnROYm5DdFFYcGIvVzhseG45NDRsS3I0ckUwTXFqcmdYL0RUaWFpMGwzMmR1?= =?utf-8?B?ejFTdFpodnBvK04yd2JkdGdibjZVRXFlNGNhdm9BQ0xBR0lFcS8zc09GOVF5?= =?utf-8?B?M3BqZVhXblhYT0FIK3dZMUZLaFByRjUvNlNwR3RDbHVlVnBVd2Urb054UTdt?= =?utf-8?B?d3AyS3RwT3pSeXBFY25aTWx0V3VuSDJXSjEzVUZETjI3OFl0c3RpOU13WEV6?= =?utf-8?B?Nlp5NTBYdHJKNURxbXprMUJiZ1o3UXNtWk9xcWEzNXh0ekFzRVgwWU5KenZp?= =?utf-8?B?L2k0djJMQnRWVzJTV01IMHlHMzRsTkRHWnJSNEVJUitZVDdZUDltSXI5RjRu?= =?utf-8?B?eHk4eUY1d2ZCajdhNml1eU5PbmJhUnpiMW9XSlJQb0RFeElqZnUzMVhqTFJR?= =?utf-8?B?MU41aTcrOU5JVHp1QWlRVmJxZUwrN1Vwd09hN3AzV1dRYnQrTmIvTThhOFph?= =?utf-8?B?OHpOeWFqQjBVUm0vQ3c4WjdMUTFMcTZLTFMzbklYQmM2b01qdzg1ei94QkxY?= =?utf-8?B?TE9UcUlYbzBkajd0QzIvRkpsYXBCRGJNUjJrYisrYWRSc0IwWStMMGJpN0lO?= =?utf-8?B?bUFsMDNNUFF5Z20wQ1VkQ0wyWlRRallYK0F3N2FTUWNuUjE1RGo4RnpWbGIx?= =?utf-8?B?SXBJVkdMTGEwQ09RMWJHZEt4MlpiaTQ3RFg0VVpCR3JuSWhhQjhqQ2VUQWxj?= =?utf-8?B?N1NyTzZyY0JENkRlKzhmK1l1S3o2dVpHNVNjckJFYWQ1SkZLa1Z6UnptZEd4?= =?utf-8?B?ak82UDF3RHIyV1BJYnFqa1JMa2lrLzdhbEkxcUxFZ0d0cUxKL3JzNHh0R0pT?= =?utf-8?B?TEdaRXlPdUo2cXZuc3laT3dVL2dpcTgzdjVNVTRPd1ZBU20rUWpIaHBZL0F2?= =?utf-8?B?THVMWTRRcS9KK2c0Y2FMMTBPRCtCSUtzTkFkYVd5RUgyQzNIc0FnWFd3ck1q?= =?utf-8?B?T0lnT3R3b0VMVXdxVVlVNkpYUnBHOVhNZHBvWitMM2NWdWsxMXNyMXhYV3Z1?= =?utf-8?Q?eujHtC0K6LGHcAAraPqdx9s=3D?= X-MS-Exchange-CrossTenant-Network-Message-Id: 1db6009c-5e43-44b9-8e3c-08d9a2b92717 X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5000.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Nov 2021 13:10:36.0500 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yXMIrOMYGBCGNoFtOzZQ5JofW3Kg4Dm+wy59LXg892WCKxshowL8mWYkx0ego4qnookKDbNPgDrZJoGaXcD3Dg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR11MB4806 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] [PATCH] net/mlx5: fix split buffer Rx X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 11/1/2021 9:56 AM, Dmitry Kozlyuk wrote: > Routine to lookup LKey on Rx was assuming that the mbuf address > always belongs to a single mempool: the one associated with an RxQ > or the MPRQ mempool. This assumption is false for split buffers case. > A wrong LKey was looked up, resulting in completion errors. > Modify lookup routines to lookup LKey in the mbuf->pool > for non-MPRQ cases both on Rx datapath and on queue initialization. > > Fixes: fec28ca0e3a9 ("net/mlx5: support mempool registration") > > Signed-off-by: Dmitry Kozlyuk > Reviewed-by: Matan Azrad > Reviewed-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/mlx5_rx.c | 5 ++++- > drivers/net/mlx5/mlx5_rx.h | 37 +++++++++++++++++++++++++++++++++++-- > 2 files changed, 39 insertions(+), 3 deletions(-) > > diff --git a/drivers/net/mlx5/mlx5_rx.c b/drivers/net/mlx5/mlx5_rx.c > index 258a645314..7296e81534 100644 > --- a/drivers/net/mlx5/mlx5_rx.c > +++ b/drivers/net/mlx5/mlx5_rx.c > @@ -356,6 +356,7 @@ mlx5_rxq_initialize(struct mlx5_rxq_data *rxq) > volatile struct mlx5_wqe_data_seg *scat; > uintptr_t addr; > uint32_t byte_count; > + uint32_t lkey; > > if (mlx5_rxq_mprq_enabled(rxq)) { > struct mlx5_mprq_buf *buf = (*rxq->mprq_bufs)[i]; > @@ -366,6 +367,7 @@ mlx5_rxq_initialize(struct mlx5_rxq_data *rxq) > 1 << rxq->strd_num_n); > byte_count = (1 << rxq->strd_sz_n) * > (1 << rxq->strd_num_n); > + lkey = mlx5_rx_addr2mr(rxq, addr); > } else { > struct rte_mbuf *buf = (*rxq->elts)[i]; > > @@ -373,13 +375,14 @@ mlx5_rxq_initialize(struct mlx5_rxq_data *rxq) > rxq->wqes)[i]; > addr = rte_pktmbuf_mtod(buf, uintptr_t); > byte_count = DATA_LEN(buf); > + lkey = mlx5_rx_mb2mr(rxq, buf); > } > /* scat->addr must be able to store a pointer. */ > MLX5_ASSERT(sizeof(scat->addr) >= sizeof(uintptr_t)); > *scat = (struct mlx5_wqe_data_seg){ > .addr = rte_cpu_to_be_64(addr), > .byte_count = rte_cpu_to_be_32(byte_count), > - .lkey = mlx5_rx_addr2mr(rxq, addr), > + .lkey = lkey, > }; > } > rxq->consumed_strd = 0; > diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h > index 4952fe1455..5903776e7a 100644 > --- a/drivers/net/mlx5/mlx5_rx.h > +++ b/drivers/net/mlx5/mlx5_rx.h > @@ -266,7 +266,7 @@ uint16_t mlx5_rx_burst_mprq_vec(void *dpdk_rxq, struct rte_mbuf **pkts, > static int mlx5_rxq_mprq_enabled(struct mlx5_rxq_data *rxq); > > /** > - * Query LKey from a packet buffer for Rx. No need to flush local caches > + * Query LKey for an address on Rx. No need to flush local caches > * as the Rx mempool database entries are valid for the lifetime of the queue. > * > * @param rxq > @@ -301,7 +301,40 @@ mlx5_rx_addr2mr(struct mlx5_rxq_data *rxq, uintptr_t addr) > mr_ctrl, mp, addr); > } > > -#define mlx5_rx_mb2mr(rxq, mb) mlx5_rx_addr2mr(rxq, (uintptr_t)((mb)->buf_addr)) > +/** > + * Query LKey from a packet buffer for Rx. No need to flush local caches > + * as the Rx mempool database entries are valid for the lifetime of the queue. > + * > + * @param rxq > + * Pointer to Rx queue structure. > + * @param mb > + * Buffer to search the address of. > + * > + * @return > + * Searched LKey on success, UINT32_MAX on no match. > + * This function always succeeds on valid input. > + */ > +static __rte_always_inline uint32_t > +mlx5_rx_mb2mr(struct mlx5_rxq_data *rxq, struct rte_mbuf *mb) > +{ > + struct mlx5_mr_ctrl *mr_ctrl = &rxq->mr_ctrl; > + uintptr_t addr = (uintptr_t)mb->buf_addr; > + struct mlx5_rxq_ctrl *rxq_ctrl; > + uint32_t lkey; > + > + /* Linear search on MR cache array. */ > + lkey = mlx5_mr_lookup_lkey(mr_ctrl->cache, &mr_ctrl->mru, > + MLX5_MR_CACHE_N, addr); > + if (likely(lkey != UINT32_MAX)) > + return lkey; > + /* > + * Slower search in the mempool database on miss. > + * During queue creation rxq->sh is not yet set, so we use rxq_ctrl. > + */ > + rxq_ctrl = container_of(rxq, struct mlx5_rxq_ctrl, rxq); > + return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->cdev->mr_scache, > + mr_ctrl, mb->pool, addr); Hi Dmitry, I am getting build errors with this patch, although I can see CI build is passed, can you please check if I am missing something, build error is: ../drivers/net/mlx5/mlx5_rx.h: In function ‘mlx5_rx_mb2mr’: ../drivers/net/mlx5/mlx5_rx.h:354:47: error: ‘struct mlx5_rxq_ctrl’ has no member named ‘priv’ 354 | return mlx5_mr_mempool2mr_bh(&rxq_ctrl->priv->sh->cdev->mr_scache, | ^~