From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 406FB423AA; Wed, 11 Jan 2023 08:13:43 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CCA1640691; Wed, 11 Jan 2023 08:13:42 +0100 (CET) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id C75534014F for ; Wed, 11 Jan 2023 08:13:39 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673421220; x=1704957220; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=c//FsfNh+GqxElb3r/HBmiWwufOeNTXwrWAn8qFeIvk=; b=aBDBr6FTZ9hErbWlnJQ9Ze5Slls+ny6YpZJJ6rXC40Jm+1CUWbVAS3iv uAJonN9Kdcu3ofdk739+ipZmBqOzQhjfInGwexWNU/UttPCavXLrcrbhC 9mgADjJef9kWalnKFK6z/+pG2OZSPWKq4zxDKLicQiSWtCjAW9FC0viHB suJsfKxKgc8rfgmeroKPN4vwsg91eeuqSQyP+73gejoDtUl0njsg8Jaoe 87OzQwPeCSTOoej0OsG7o/sllCxx/4atoEV/GwisbHddAluZnG/ItJtno Se1PXXneRz6HKviZ01zHGEK6Tt38INt+ERx6tE492WpzoNLGFZiBWBYTf A==; X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="306864857" X-IronPort-AV: E=Sophos;i="5.96,315,1665471600"; d="scan'208";a="306864857" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Jan 2023 23:13:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10586"; a="720620940" X-IronPort-AV: E=Sophos;i="5.96,315,1665471600"; d="scan'208";a="720620940" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by fmsmga008.fm.intel.com with ESMTP; 10 Jan 2023 23:13:38 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16; Tue, 10 Jan 2023 23:13:37 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16; Tue, 10 Jan 2023 23:13:36 -0800 Received: from fmsedg601.ED.cps.intel.com (10.1.192.135) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.16 via Frontend Transport; Tue, 10 Jan 2023 23:13:36 -0800 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.176) by edgegateway.intel.com (192.55.55.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.16; Tue, 10 Jan 2023 23:13:36 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WtazsO8yJsHj9Q/w5zENGfXKLr5Eaz5YeBPiDsmy2qeK4fMlwO0w3L5DRQQPkzbsLKxzOL8QVNi3a5UvwC6zdrjTJCzHDtSRt8ZQ80tx7E0pG6JrKUeV0pQptTIgOLI2WNWWuh2A86+yJHkDaeGKfA87En8LfJswI7+AjJL0vZWCuBCa45XVrrQ+Gc25/uOlcRYEq9HvZ12Av9IvqSymPzRgQLrsKtYzSmw0++GIgXlzWeiFv3Wm3tkLfQnPXkK0kyRLrfZBBCx57c/zsW642fwYtJiWHuEyiZ8MsLcH8zYueX3O9i9hc4zh1QefAaoJKARLxVP1fd+hGilbWgtC8g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=s9bpKWfc0pkbOeRXT0G537wQQ3VNhxD9Zo5qY8q+G2E=; b=YJbcQJR+LddhkFybRPsqdKSunUqO3zCS67lT0Tg7C9JNv8/ayHdhTdlMLKOoxIYFac9QEr1FKyzU4CPRt1Ef/D5dhN4++dunriIXKi1VThj/btskFtjdKm72MmT422wRjtRuAbf7BFRWQsbnA6FYmpr/WcxWn6Jb3nq11dmt0Lfpek816MX/PN8tbjR8cPFbg3jRjUI6i5XHY0/BJcazvUgQWya+rjpuf6yF0kNFELT6+J2llQFX8RMyeCqolw3dKNJGRAiVTGJGcmxbiM4LjnCQuFdGq4RuYNQbdjgRpYewISSRbH7ZzitotnEfTVqlbaWbAGBd5MAGXYzCbiLGyA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from PH0PR11MB5877.namprd11.prod.outlook.com (2603:10b6:510:141::12) by IA1PR11MB7920.namprd11.prod.outlook.com (2603:10b6:208:3fc::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5986.18; Wed, 11 Jan 2023 07:13:33 +0000 Received: from PH0PR11MB5877.namprd11.prod.outlook.com ([fe80::8c1c:6df6:533f:282c]) by PH0PR11MB5877.namprd11.prod.outlook.com ([fe80::8c1c:6df6:533f:282c%6]) with mapi id 15.20.5986.018; Wed, 11 Jan 2023 07:13:33 +0000 From: "Liu, Mingxia" To: "Wu, Wenjun1" , "dev@dpdk.org" , "Wu, Jingjing" , "Xing, Beilei" CC: "Lu, Wenzhuo" , "Qiao, Wenjing" Subject: RE: [PATCH v3 1/1] common/idpf: add AVX512 data path for split queue model Thread-Topic: [PATCH v3 1/1] common/idpf: add AVX512 data path for split queue model Thread-Index: AQHZIZoAabWxJpkOFk+7EcB9AatRhq6Y04cA Date: Wed, 11 Jan 2023 07:13:33 +0000 Message-ID: References: <20221214060706.962541-1-wenjun1.wu@intel.com> <20230106064645.2657232-1-wenjun1.wu@intel.com> <20230106064645.2657232-2-wenjun1.wu@intel.com> In-Reply-To: <20230106064645.2657232-2-wenjun1.wu@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: PH0PR11MB5877:EE_|IA1PR11MB7920:EE_ x-ms-office365-filtering-correlation-id: bd08bee2-4e8f-4059-be99-08daf3a359ac x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 2SqZxQHWlMC3Y4CoTCAZ9YLUzbGSa4wL77dMBbOsL8nm2LqRM0oAYIlua/kbR+pnLxjsR7eybaQ2Sx+hX7wYisFOatia2bYWyZUzchZ5okymnoEoqRKlPM1puw579sTnYHgWh+Geflzwe4xXtvXEswMBxQ+kku0tc0ePN6yi3Kf/Sxvxe8iKAzDnhiRZUxq9X9vMnGVqAVYAHqqci1dOylpSzDb6WWyGjmOpx3HWL2pYbl3S3TRdNZ434anCb/GXyiq6+6D42dkteQYNjO1wqkn+N+BuhssrQx5BzY1ovMiUMsW7XHWOQMzrlYowe6o+iGE9eYjPdDb/WwEgNvNbIKp9VTNRlhu6e1ewg0qy8FykBjhBiY3h0xAAWtqC6Kdewz8O4k+fEGttzQMwvEz35WpINUsnfemK6FeVrWOgObzf+aEyJGMHK3kv1OTZEoRmzrR+YYP6TfWfcKZEcW5THvxNB2T7IV11tlOIVt6SZbncwj54fCnTcqzdI4C/JGoW/qBPdt7p+2z90N8OXI5H5zHYUrzeMYAESO+7NPRyIY3yiQOEiFi7u0TFlIfNOZ39g30pmyskTGDGSI+ItDfaF7FQaqCIKgNhJFu8kt7ypnmc20SWNxhJ5oXCidTUlzcDhmfgxj4ogvTVFPp3snSwzHgRp3Kyf61pCD4nkK+7ueCYQptaLOGcVf/tVD6PNxDGqQ+drkXw/UTbaE/LmZnNTQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PH0PR11MB5877.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(39860400002)(396003)(136003)(366004)(376002)(346002)(451199015)(9686003)(26005)(186003)(86362001)(55016003)(30864003)(7696005)(64756008)(8676002)(66556008)(52536014)(83380400001)(66446008)(4326008)(33656002)(66476007)(8936002)(76116006)(71200400001)(5660300002)(66946007)(2906002)(478600001)(316002)(6636002)(54906003)(110136005)(41300700001)(38070700005)(53546011)(6506007)(38100700002)(122000001)(82960400001)(107886003)(559001)(579004); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?XhU8V1WSdRgSr091AA/Mm3fuC/PH1h8BXda2zpC7SsAZidUHM6YjkR0VLFIe?= =?us-ascii?Q?duwEBgWCh4mkvxWDwUHFY5M10S1k2G/7uWlhpYKkNzNo17X805ExfePnaVHY?= =?us-ascii?Q?rZMlu+/yGLmRUXr1KAsUBDhmi/VSFGNzquHMgt9ISz91GcknYR7ExpqZjghb?= =?us-ascii?Q?QH91rdtmwpBYFQPe0Dv+jlMUeRRxPvjaJc57JEcCM55lGlccsIas3Zq86dcn?= =?us-ascii?Q?Cq6SH8tUKVDtfQHGYpqeWAiNgXcrsCvE/OzerdC0tWPyeN46dyp74tn1L4tJ?= =?us-ascii?Q?RP7qLKk9Aa6Gao6jdpfALDP97Af1TxDFsnr+6pwFDH7/09A7yvGDfMSBCpIU?= =?us-ascii?Q?aMKTHlKXdMwXiy3Ek1458YzQWtgBFXGxQJTMCFc+bGtlg4SpMqRL3bzBAmbJ?= =?us-ascii?Q?Nj5xD1MFpK8ZiG9OaSGJTu3RazAK7Qv91JnU9Ud1q1v0GBHwlYPuc43RTj5c?= =?us-ascii?Q?SlKy6yZ7m/l9Gbo23eGbttfNSmHuVnrR432acTOv/cdpi9sx4wf9WKSUL0Vl?= =?us-ascii?Q?VuLMStz0HB4G96OLQjowaQSsS/XBZpbeo7WP1QWfvvjK5EeL6mfNmlVyjgX9?= =?us-ascii?Q?rATOEMdoFr/dM3pMpp5D2ti6B5OZIfkfd+Fi3RjVjk74bXxhQ4XZ7Sh6riaR?= =?us-ascii?Q?t2ORC7d31XDEzeQBdc4PLmDtocI8FUlD6wSb0avTQQSBaXNI5xBDl/qN9geJ?= =?us-ascii?Q?vy/dAfKcyEJEtFKm6/NLuW5h7AOfLeM6rtzIaYVJYpU3HgSM4tgJU5Gyof2b?= =?us-ascii?Q?KBfkIG/8BnMQtH3Wecf+r8u8LHV9mKjCfcbgO1Orgk1yVu0vulPmgrAVPTHm?= =?us-ascii?Q?AoNWsfUY8JP/VguUinQaKluWHcU55L9brg9Cb8jS30Eo8EgrlocVCrrEf4nn?= =?us-ascii?Q?uXoHCF0/emekwKWLKNlCPB+tDXszt8n5anwXsp3fkuOD3H3mnuuq7ajPvCzh?= =?us-ascii?Q?lvcuX4N0Q7opqdGm+RG27H5UNO8OZWjW1+UABYHhcRxApB3xaXr634RswrlQ?= =?us-ascii?Q?qcSRsZBShuRnNpwRSB6xoJ1H4niOJizFsw8gUucHwGWaZZVkRLZrA6VZbWcJ?= =?us-ascii?Q?6wiQ+pvFuTY+urbIwvGLmCjreRvXX0QnN3GiSTjyvtR/Z2LoLKhQ55i7da4D?= =?us-ascii?Q?It2oxZIIdO+BL+McLyyJzORaJvq9bW/POZQQxCIMsYe7SCMh8fYUIbfQVJNx?= =?us-ascii?Q?1tHsvr7KIbEfG1ZHm9BUjvHW3GfeyMbik2RBYviqTvpFmvd4ZptwFERWGRmc?= =?us-ascii?Q?CyPheSfbqUmj8brsYWfHor1rlEOu9nBkLWwzVP0m0X2nK8ZXGRF6C7lglRYl?= =?us-ascii?Q?JOS2zc2wngoAkOuqZA/ANhE8TmEJvNH2cD9QamjnjPwq4BZNOcHqqMXhIpDz?= =?us-ascii?Q?fKR2f6DL+9kxYjGuuh/PmIDrIa38asajni/AoiQC3dAz7l8k+O1ETX99Z+c/?= =?us-ascii?Q?dyDyG5HcxqRCzwdYTdys6IMJjBcHIdjqBaq6peDMXxbED0fBWCYpcqFVcRcI?= =?us-ascii?Q?0VATitdRrrZHTz5JkdfuzOcISG2imIG6thfdgsxzn1ek3AcYIzN4b/CnufT8?= =?us-ascii?Q?zsihT615ltxUww2YUhIc0oe38jP50bjtLixAmulO?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PH0PR11MB5877.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: bd08bee2-4e8f-4059-be99-08daf3a359ac X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Jan 2023 07:13:33.5433 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: DKfu79L9PqsprBjggGA9+VqV/u4GybQ49FsaG+3kICbdq6y4vDBhJ0+RwRXvFbJXl5if3uu0mC6ySwYa+niMkw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR11MB7920 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org According to beilei's v2 patchset, struct idpf_rx_stats { rte_atomic64_t mbuf_alloc_failed; }; rx_bufq->rx_stats.mbuf_alloc_failed +=3D IDPF_RXQ_REARM_THRESH; should be changed to rte_atomic64_add(&rx_bufq->rx_stats.mbuf_alloc_failed, IDPF_RXQ_REARM_THRES= H); > -----Original Message----- > From: Wu, Wenjun1 > Sent: Friday, January 6, 2023 2:47 PM > To: dev@dpdk.org; Wu, Jingjing ; Xing, Beilei > > Cc: Liu, Mingxia ; Lu, Wenzhuo > ; Qiao, Wenjing ; Wu, > Wenjun1 > Subject: [PATCH v3 1/1] common/idpf: add AVX512 data path for split queue > model >=20 > Add support of AVX512 data path for split queue model. >=20 > Signed-off-by: Wenjun Wu > Reviewed-by: Wenjing Qiao > --- > drivers/common/idpf/idpf_common_rxtx.c | 22 +- > drivers/common/idpf/idpf_common_rxtx.h | 19 +- > drivers/common/idpf/idpf_common_rxtx_avx512.c | 797 > +++++++++++++++++- > drivers/common/idpf/version.map | 5 +- > drivers/net/idpf/idpf_rxtx.c | 46 +- > drivers/net/idpf/idpf_rxtx_vec_common.h | 55 +- > 6 files changed, 889 insertions(+), 55 deletions(-) >=20 > diff --git a/drivers/common/idpf/idpf_common_rxtx.c > b/drivers/common/idpf/idpf_common_rxtx.c > index 7618819e68..7f8311d8f6 100644 > --- a/drivers/common/idpf/idpf_common_rxtx.c > +++ b/drivers/common/idpf/idpf_common_rxtx.c > @@ -161,6 +161,9 @@ reset_split_rx_bufq(struct idpf_rx_queue *rxq) > /* The number of descriptors which can be refilled. */ > rxq->nb_rx_hold =3D rxq->nb_rx_desc - 1; >=20 > + rxq->rxrearm_nb =3D 0; > + rxq->rxrearm_start =3D 0; > + > rxq->bufq1 =3D NULL; > rxq->bufq2 =3D NULL; > } > @@ -236,6 +239,10 @@ reset_split_tx_descq(struct idpf_tx_queue *txq) > txq->last_desc_cleaned =3D 0; > txq->sw_tail =3D 0; > txq->nb_free =3D txq->nb_tx_desc - 1; > + > + memset(txq->ctype, 0, sizeof(txq->ctype)); > + txq->next_dd =3D txq->rs_thresh - 1; > + txq->next_rs =3D txq->rs_thresh - 1; > } >=20 > void > @@ -1426,12 +1433,12 @@ release_rxq_mbufs_vec(struct idpf_rx_queue > *rxq) > memset(rxq->sw_ring, 0, sizeof(rxq->sw_ring[0]) * rxq- > >nb_rx_desc); } >=20 > -static const struct idpf_rxq_ops def_singleq_rx_ops_vec =3D { > +static const struct idpf_rxq_ops def_rx_ops_vec =3D { > .release_mbufs =3D release_rxq_mbufs_vec, }; >=20 > static inline int > -idpf_singleq_rx_vec_setup_default(struct idpf_rx_queue *rxq) > +idpf_rxq_vec_setup_default(struct idpf_rx_queue *rxq) > { > uintptr_t p; > struct rte_mbuf mb_def =3D { .buf_addr =3D 0 }; /* zeroed mbuf */ @@ - > 1451,6 +1458,13 @@ idpf_singleq_rx_vec_setup_default(struct > idpf_rx_queue *rxq) int __rte_cold idpf_singleq_rx_vec_setup(struct > idpf_rx_queue *rxq) { > - rxq->ops =3D &def_singleq_rx_ops_vec; > - return idpf_singleq_rx_vec_setup_default(rxq); > + rxq->ops =3D &def_rx_ops_vec; > + return idpf_rxq_vec_setup_default(rxq); } > + > +int __rte_cold > +idpf_splitq_rx_vec_setup(struct idpf_rx_queue *rxq) { > + rxq->bufq2->ops =3D &def_rx_ops_vec; > + return idpf_rxq_vec_setup_default(rxq->bufq2); > } > diff --git a/drivers/common/idpf/idpf_common_rxtx.h > b/drivers/common/idpf/idpf_common_rxtx.h > index 8a6e04af56..e23484d031 100644 > --- a/drivers/common/idpf/idpf_common_rxtx.h > +++ b/drivers/common/idpf/idpf_common_rxtx.h > @@ -52,6 +52,8 @@ > #define IDPF_VPMD_TX_MAX_BURST 32 > #define IDPF_VPMD_DESCS_PER_LOOP 4 > #define IDPF_RXQ_REARM_THRESH 64 > +#define IDPD_TXQ_SCAN_CQ_THRESH 64 > +#define IDPF_TX_CTYPE_NUM 8 >=20 > /* MTS */ > #define GLTSYN_CMD_SYNC_0_0 (PF_TIMESYNC_BASE + 0x0) > @@ -185,6 +187,7 @@ struct idpf_tx_queue { > uint32_t tx_start_qid; > uint8_t expected_gen_id; > struct idpf_tx_queue *complq; > + uint16_t ctype[IDPF_TX_CTYPE_NUM]; > }; >=20 > /* Offload features */ > @@ -203,6 +206,12 @@ struct idpf_tx_vec_entry { > struct rte_mbuf *mbuf; > }; >=20 > +union idpf_tx_desc { > + struct idpf_base_tx_desc *tx_ring; > + struct idpf_flex_tx_sched_desc *desc_ring; > + struct idpf_splitq_tx_compl_desc *compl_ring; }; > + > struct idpf_rxq_ops { > void (*release_mbufs)(struct idpf_rx_queue *rxq); }; @@ -265,14 > +274,22 @@ uint16_t idpf_prep_pkts(void *tx_queue, struct rte_mbuf > **tx_pkts, __rte_internal int idpf_singleq_rx_vec_setup(struct > idpf_rx_queue *rxq); __rte_internal -int > idpf_singleq_tx_vec_setup_avx512(struct idpf_tx_queue *txq); > +int idpf_splitq_rx_vec_setup(struct idpf_rx_queue *rxq); __rte_internal > +int idpf_tx_vec_setup_avx512(struct idpf_tx_queue *txq); > __rte_internal > uint16_t idpf_singleq_recv_pkts_avx512(void *rx_queue, > struct rte_mbuf **rx_pkts, > uint16_t nb_pkts); > __rte_internal > +uint16_t idpf_splitq_recv_pkts_avx512(void *tx_queue, struct rte_mbuf > **tx_pkts, > + uint16_t nb_pkts); > +__rte_internal > uint16_t idpf_singleq_xmit_pkts_avx512(void *tx_queue, > struct rte_mbuf **tx_pkts, > uint16_t nb_pkts); > +__rte_internal > +uint16_t idpf_splitq_xmit_pkts_avx512(void *tx_queue, struct rte_mbuf > **tx_pkts, > + uint16_t nb_pkts); >=20 > #endif /* _IDPF_COMMON_RXTX_H_ */ > diff --git a/drivers/common/idpf/idpf_common_rxtx_avx512.c > b/drivers/common/idpf/idpf_common_rxtx_avx512.c > index de52c6e45f..5a91ed610e 100644 > --- a/drivers/common/idpf/idpf_common_rxtx_avx512.c > +++ b/drivers/common/idpf/idpf_common_rxtx_avx512.c > @@ -537,8 +537,462 @@ idpf_singleq_recv_pkts_avx512(void *rx_queue, > struct rte_mbuf **rx_pkts, > return _idpf_singleq_recv_raw_pkts_avx512(rx_queue, rx_pkts, > nb_pkts); } >=20 > +static __rte_always_inline void > +idpf_splitq_rearm_common(struct idpf_rx_queue *rx_bufq) { > + struct rte_mbuf **rxp =3D &rx_bufq->sw_ring[rx_bufq- > >rxrearm_start]; > + volatile union virtchnl2_rx_buf_desc *rxdp =3D rx_bufq->rx_ring; > + uint16_t rx_id; > + int i; > + > + rxdp +=3D rx_bufq->rxrearm_start; > + > + /* Pull 'n' more MBUFs into the software ring */ > + if (rte_mempool_get_bulk(rx_bufq->mp, > + (void *)rxp, > + IDPF_RXQ_REARM_THRESH) < 0) { > + if (rx_bufq->rxrearm_nb + IDPF_RXQ_REARM_THRESH >=3D > + rx_bufq->nb_rx_desc) { > + __m128i dma_addr0; > + > + dma_addr0 =3D _mm_setzero_si128(); > + for (i =3D 0; i < IDPF_VPMD_DESCS_PER_LOOP; i++) { > + rxp[i] =3D &rx_bufq->fake_mbuf; > + _mm_store_si128((__m128i *)&rxdp[i], > + dma_addr0); > + } > + } > + rx_bufq->rx_stats.mbuf_alloc_failed +=3D > IDPF_RXQ_REARM_THRESH; > + return; > + } > + > + /* Initialize the mbufs in vector, process 8 mbufs in one loop */ > + for (i =3D 0; i < IDPF_RXQ_REARM_THRESH; > + i +=3D 8, rxp +=3D 8, rxdp +=3D 8) { > + rxdp[0].split_rd.pkt_addr =3D rxp[0]->buf_iova + > RTE_PKTMBUF_HEADROOM; > + rxdp[1].split_rd.pkt_addr =3D rxp[1]->buf_iova + > RTE_PKTMBUF_HEADROOM; > + rxdp[2].split_rd.pkt_addr =3D rxp[2]->buf_iova + > RTE_PKTMBUF_HEADROOM; > + rxdp[3].split_rd.pkt_addr =3D rxp[3]->buf_iova + > RTE_PKTMBUF_HEADROOM; > + rxdp[4].split_rd.pkt_addr =3D rxp[4]->buf_iova + > RTE_PKTMBUF_HEADROOM; > + rxdp[5].split_rd.pkt_addr =3D rxp[5]->buf_iova + > RTE_PKTMBUF_HEADROOM; > + rxdp[6].split_rd.pkt_addr =3D rxp[6]->buf_iova + > RTE_PKTMBUF_HEADROOM; > + rxdp[7].split_rd.pkt_addr =3D rxp[7]->buf_iova + > RTE_PKTMBUF_HEADROOM; > + } > + > + rx_bufq->rxrearm_start +=3D IDPF_RXQ_REARM_THRESH; > + if (rx_bufq->rxrearm_start >=3D rx_bufq->nb_rx_desc) > + rx_bufq->rxrearm_start =3D 0; > + > + rx_bufq->rxrearm_nb -=3D IDPF_RXQ_REARM_THRESH; > + > + rx_id =3D (uint16_t)((rx_bufq->rxrearm_start =3D=3D 0) ? > + (rx_bufq->nb_rx_desc - 1) : (rx_bufq- > >rxrearm_start - 1)); > + > + /* Update the tail pointer on the NIC */ > + IDPF_PCI_REG_WRITE(rx_bufq->qrx_tail, rx_id); } > + > +static __rte_always_inline void > +idpf_splitq_rearm(struct idpf_rx_queue *rx_bufq) { > + int i; > + uint16_t rx_id; > + volatile union virtchnl2_rx_buf_desc *rxdp =3D rx_bufq->rx_ring; > + struct rte_mempool_cache *cache =3D > + rte_mempool_default_cache(rx_bufq->mp, rte_lcore_id()); > + struct rte_mbuf **rxp =3D &rx_bufq->sw_ring[rx_bufq- > >rxrearm_start]; > + > + rxdp +=3D rx_bufq->rxrearm_start; > + > + if (unlikely(!cache)) > + return idpf_splitq_rearm_common(rx_bufq); > + > + /* We need to pull 'n' more MBUFs into the software ring from > mempool > + * We inline the mempool function here, so we can vectorize the > copy > + * from the cache into the shadow ring. > + */ > + > + /* Can this be satisfied from the cache? */ > + if (cache->len < IDPF_RXQ_REARM_THRESH) { > + /* No. Backfill the cache first, and then fill from it */ > + uint32_t req =3D IDPF_RXQ_REARM_THRESH + (cache->size - > + cache->len); > + > + /* How many do we require i.e. number to fill the cache + > the request */ > + int ret =3D rte_mempool_ops_dequeue_bulk > + (rx_bufq->mp, &cache->objs[cache->len], > req); > + if (ret =3D=3D 0) { > + cache->len +=3D req; > + } else { > + if (rx_bufq->rxrearm_nb + > IDPF_RXQ_REARM_THRESH >=3D > + rx_bufq->nb_rx_desc) { > + __m128i dma_addr0; > + > + dma_addr0 =3D _mm_setzero_si128(); > + for (i =3D 0; i < IDPF_VPMD_DESCS_PER_LOOP; > i++) { > + rxp[i] =3D &rx_bufq->fake_mbuf; > + _mm_storeu_si128((__m128i > *)&rxdp[i], > + dma_addr0); > + } > + } > + rx_bufq->rx_stats.mbuf_alloc_failed +=3D > IDPF_RXQ_REARM_THRESH; > + return; > + } > + } > + > + const __m512i iova_offsets =3D _mm512_set1_epi64(offsetof > + (struct rte_mbuf, > buf_iova)); > + const __m512i headroom =3D > _mm512_set1_epi64(RTE_PKTMBUF_HEADROOM); > + > + /* Initialize the mbufs in vector, process 8 mbufs in one loop, taking > + * from mempool cache and populating both shadow and HW rings > + */ > + for (i =3D 0; i < IDPF_RXQ_REARM_THRESH / > IDPF_DESCS_PER_LOOP_AVX; i++) { > + const __m512i mbuf_ptrs =3D _mm512_loadu_si512 > + (&cache->objs[cache->len - > IDPF_DESCS_PER_LOOP_AVX]); > + _mm512_storeu_si512(rxp, mbuf_ptrs); > + > + const __m512i iova_base_addrs =3D _mm512_i64gather_epi64 > + (_mm512_add_epi64(mbuf_ptrs, > iova_offsets), > + 0, /* base */ > + 1 /* scale */); > + const __m512i iova_addrs =3D > _mm512_add_epi64(iova_base_addrs, > + headroom); > + > + const __m512i iova_addrs_1 =3D > _mm512_bsrli_epi128(iova_addrs, 8); > + > + rxdp[0].split_rd.pkt_addr =3D > + > _mm_cvtsi128_si64(_mm512_extracti32x4_epi32(iova_addrs, 0)); > + rxdp[1].split_rd.pkt_addr =3D > + > _mm_cvtsi128_si64(_mm512_extracti32x4_epi32(iova_addrs_1, 0)); > + rxdp[2].split_rd.pkt_addr =3D > + > _mm_cvtsi128_si64(_mm512_extracti32x4_epi32(iova_addrs, 1)); > + rxdp[3].split_rd.pkt_addr =3D > + > _mm_cvtsi128_si64(_mm512_extracti32x4_epi32(iova_addrs_1, 1)); > + rxdp[4].split_rd.pkt_addr =3D > + > _mm_cvtsi128_si64(_mm512_extracti32x4_epi32(iova_addrs, 2)); > + rxdp[5].split_rd.pkt_addr =3D > + > _mm_cvtsi128_si64(_mm512_extracti32x4_epi32(iova_addrs_1, 2)); > + rxdp[6].split_rd.pkt_addr =3D > + > _mm_cvtsi128_si64(_mm512_extracti32x4_epi32(iova_addrs, 3)); > + rxdp[7].split_rd.pkt_addr =3D > + > _mm_cvtsi128_si64(_mm512_extracti32x4_epi32(iova_addrs_1, 3)); > + > + rxp +=3D IDPF_DESCS_PER_LOOP_AVX; > + rxdp +=3D IDPF_DESCS_PER_LOOP_AVX; > + cache->len -=3D IDPF_DESCS_PER_LOOP_AVX; > + } > + > + rx_bufq->rxrearm_start +=3D IDPF_RXQ_REARM_THRESH; > + if (rx_bufq->rxrearm_start >=3D rx_bufq->nb_rx_desc) > + rx_bufq->rxrearm_start =3D 0; > + > + rx_bufq->rxrearm_nb -=3D IDPF_RXQ_REARM_THRESH; > + > + rx_id =3D (uint16_t)((rx_bufq->rxrearm_start =3D=3D 0) ? > + (rx_bufq->nb_rx_desc - 1) : (rx_bufq- > >rxrearm_start - 1)); > + > + /* Update the tail pointer on the NIC */ > + IDPF_PCI_REG_WRITE(rx_bufq->qrx_tail, rx_id); } > + > +static __rte_always_inline uint16_t > +_idpf_splitq_recv_raw_pkts_avx512(struct idpf_rx_queue *rxq, > + struct rte_mbuf **rx_pkts, > + uint16_t nb_pkts) > +{ > + const uint32_t *type_table =3D rxq->adapter->ptype_tbl; > + const __m256i mbuf_init =3D _mm256_set_epi64x(0, 0, 0, > + rxq->bufq2- > >mbuf_initializer); > + /* only handle bufq2 here */ > + struct rte_mbuf **sw_ring =3D &rxq->bufq2->sw_ring[rxq->rx_tail]; > + volatile union virtchnl2_rx_desc *rxdp =3D rxq->rx_ring; > + > + rxdp +=3D rxq->rx_tail; > + > + rte_prefetch0(rxdp); > + > + /* nb_pkts has to be floor-aligned to IDPF_DESCS_PER_LOOP_AVX > */ > + nb_pkts =3D RTE_ALIGN_FLOOR(nb_pkts, > IDPF_DESCS_PER_LOOP_AVX); > + > + /* See if we need to rearm the RX queue - gives the prefetch a bit > + * of time to act > + */ > + if (rxq->bufq2->rxrearm_nb > IDPF_RXQ_REARM_THRESH) > + idpf_splitq_rearm(rxq->bufq2); > + > + /* Before we start moving massive data around, check to see if > + * there is actually a packet available > + */ > + if (((rxdp->flex_adv_nic_3_wb.pktlen_gen_bufq_id & > + VIRTCHNL2_RX_FLEX_DESC_ADV_GEN_M) >> > + VIRTCHNL2_RX_FLEX_DESC_ADV_GEN_S) !=3D rxq- > >expected_gen_id) > + return 0; > + > + const __m512i dd_check =3D _mm512_set1_epi64(1); > + const __m512i gen_check =3D _mm512_set1_epi64((uint64_t)1<<46); > + > + /* mask to shuffle from desc. to mbuf (4 descriptors)*/ > + const __m512i shuf_msk =3D > + _mm512_set_epi32 > + (/* 1st descriptor */ > + 0xFFFFFFFF, /* octet 4~7, 32bits rss */ > + 0xFFFF0504, /* octet 2~3, low 16 bits vlan_macip */ > + /* octet 15~14, 16 bits data_len */ > + 0xFFFF0504, /* skip high 16 bits pkt_len, zero out */ > + /* octet 15~14, low 16 bits pkt_len */ > + 0xFFFFFFFF, /* pkt_type set as unknown */ > + /* 2nd descriptor */ > + 0xFFFFFFFF, /* octet 4~7, 32bits rss */ > + 0xFFFF0504, /* octet 2~3, low 16 bits vlan_macip */ > + /* octet 15~14, 16 bits data_len */ > + 0xFFFF0504, /* skip high 16 bits pkt_len, zero out */ > + /* octet 15~14, low 16 bits pkt_len */ > + 0xFFFFFFFF, /* pkt_type set as unknown */ > + /* 3rd descriptor */ > + 0xFFFFFFFF, /* octet 4~7, 32bits rss */ > + 0xFFFF0504, /* octet 2~3, low 16 bits vlan_macip */ > + /* octet 15~14, 16 bits data_len */ > + 0xFFFF0504, /* skip high 16 bits pkt_len, zero out */ > + /* octet 15~14, low 16 bits pkt_len */ > + 0xFFFFFFFF, /* pkt_type set as unknown */ > + /* 4th descriptor */ > + 0xFFFFFFFF, /* octet 4~7, 32bits rss */ > + 0xFFFF0504, /* octet 2~3, low 16 bits vlan_macip */ > + /* octet 15~14, 16 bits data_len */ > + 0xFFFF0504, /* skip high 16 bits pkt_len, zero out */ > + /* octet 15~14, low 16 bits pkt_len */ > + 0xFFFFFFFF /* pkt_type set as unknown */ > + ); > + /** > + * compile-time check the above crc and shuffle layout is correct. > + * NOTE: the first field (lowest address) is given last in set_epi > + * calls above. > + */ > + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, pkt_len) !=3D > + offsetof(struct rte_mbuf, rx_descriptor_fields1) + 4); > + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, data_len) !=3D > + offsetof(struct rte_mbuf, rx_descriptor_fields1) + 8); > + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, vlan_tci) !=3D > + offsetof(struct rte_mbuf, rx_descriptor_fields1) + > 10); > + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, hash) !=3D > + offsetof(struct rte_mbuf, rx_descriptor_fields1) + > 12); > + > + uint16_t i, received; > + > + for (i =3D 0, received =3D 0; i < nb_pkts; > + i +=3D IDPF_DESCS_PER_LOOP_AVX, > + rxdp +=3D IDPF_DESCS_PER_LOOP_AVX) { > + /* step 1, copy over 8 mbuf pointers to rx_pkts array */ > + _mm256_storeu_si256((void *)&rx_pkts[i], > + _mm256_loadu_si256((void *)&sw_ring[i])); > #ifdef > +RTE_ARCH_X86_64 > + _mm256_storeu_si256 > + ((void *)&rx_pkts[i + 4], > + _mm256_loadu_si256((void *)&sw_ring[i + 4])); > #endif > + > + __m512i raw_desc0_3, raw_desc4_7; > + const __m128i raw_desc7 =3D > + _mm_load_si128((void *)(rxdp + 7)); > + rte_compiler_barrier(); > + const __m128i raw_desc6 =3D > + _mm_load_si128((void *)(rxdp + 6)); > + rte_compiler_barrier(); > + const __m128i raw_desc5 =3D > + _mm_load_si128((void *)(rxdp + 5)); > + rte_compiler_barrier(); > + const __m128i raw_desc4 =3D > + _mm_load_si128((void *)(rxdp + 4)); > + rte_compiler_barrier(); > + const __m128i raw_desc3 =3D > + _mm_load_si128((void *)(rxdp + 3)); > + rte_compiler_barrier(); > + const __m128i raw_desc2 =3D > + _mm_load_si128((void *)(rxdp + 2)); > + rte_compiler_barrier(); > + const __m128i raw_desc1 =3D > + _mm_load_si128((void *)(rxdp + 1)); > + rte_compiler_barrier(); > + const __m128i raw_desc0 =3D > + _mm_load_si128((void *)(rxdp + 0)); > + > + raw_desc4_7 =3D _mm512_broadcast_i32x4(raw_desc4); > + raw_desc4_7 =3D _mm512_inserti32x4(raw_desc4_7, > raw_desc5, 1); > + raw_desc4_7 =3D _mm512_inserti32x4(raw_desc4_7, > raw_desc6, 2); > + raw_desc4_7 =3D _mm512_inserti32x4(raw_desc4_7, > raw_desc7, 3); > + raw_desc0_3 =3D _mm512_broadcast_i32x4(raw_desc0); > + raw_desc0_3 =3D _mm512_inserti32x4(raw_desc0_3, > raw_desc1, 1); > + raw_desc0_3 =3D _mm512_inserti32x4(raw_desc0_3, > raw_desc2, 2); > + raw_desc0_3 =3D _mm512_inserti32x4(raw_desc0_3, > raw_desc3, 3); > + > + /** > + * convert descriptors 4-7 into mbufs, adjusting length and > + * re-arranging fields. Then write into the mbuf > + */ > + const __m512i len_mask =3D _mm512_set_epi32(0xffffffff, > 0xffffffff, > + 0xffff3fff, 0xffffffff, > + 0xffffffff, 0xffffffff, > + 0xffff3fff, 0xffffffff, > + 0xffffffff, 0xffffffff, > + 0xffff3fff, 0xffffffff, > + 0xffffffff, 0xffffffff, > + 0xffff3fff, 0xffffffff); > + const __m512i desc4_7 =3D _mm512_and_epi32(raw_desc4_7, > len_mask); > + __m512i mb4_7 =3D _mm512_shuffle_epi8(desc4_7, > shuf_msk); > + > + /** > + * to get packet types, shift 64-bit values down 30 bits > + * and so ptype is in lower 8-bits in each > + */ > + const __m512i ptypes4_7 =3D _mm512_srli_epi64(desc4_7, 16); > + const __m256i ptypes6_7 =3D > _mm512_extracti64x4_epi64(ptypes4_7, 1); > + const __m256i ptypes4_5 =3D > _mm512_extracti64x4_epi64(ptypes4_7, 0); > + const uint8_t ptype7 =3D _mm256_extract_epi8(ptypes6_7, 16); > + const uint8_t ptype6 =3D _mm256_extract_epi8(ptypes6_7, 0); > + const uint8_t ptype5 =3D _mm256_extract_epi8(ptypes4_5, 16); > + const uint8_t ptype4 =3D _mm256_extract_epi8(ptypes4_5, 0); > + > + const __m512i ptype4_7 =3D _mm512_set_epi32 > + (0, 0, 0, type_table[ptype7], > + 0, 0, 0, type_table[ptype6], > + 0, 0, 0, type_table[ptype5], > + 0, 0, 0, type_table[ptype4]); > + mb4_7 =3D _mm512_mask_blend_epi32(0x1111, mb4_7, > ptype4_7); > + > + /** > + * convert descriptors 0-3 into mbufs, adjusting length and > + * re-arranging fields. Then write into the mbuf > + */ > + const __m512i desc0_3 =3D _mm512_and_epi32(raw_desc0_3, > len_mask); > + __m512i mb0_3 =3D _mm512_shuffle_epi8(desc0_3, > shuf_msk); > + > + /* get the packet types */ > + const __m512i ptypes0_3 =3D _mm512_srli_epi64(desc0_3, 16); > + const __m256i ptypes2_3 =3D > _mm512_extracti64x4_epi64(ptypes0_3, 1); > + const __m256i ptypes0_1 =3D > _mm512_extracti64x4_epi64(ptypes0_3, 0); > + const uint8_t ptype3 =3D _mm256_extract_epi8(ptypes2_3, 16); > + const uint8_t ptype2 =3D _mm256_extract_epi8(ptypes2_3, 0); > + const uint8_t ptype1 =3D _mm256_extract_epi8(ptypes0_1, 16); > + const uint8_t ptype0 =3D _mm256_extract_epi8(ptypes0_1, 0); > + > + const __m512i ptype0_3 =3D _mm512_set_epi32 > + (0, 0, 0, type_table[ptype3], > + 0, 0, 0, type_table[ptype2], > + 0, 0, 0, type_table[ptype1], > + 0, 0, 0, type_table[ptype0]); > + mb0_3 =3D _mm512_mask_blend_epi32(0x1111, mb0_3, > ptype0_3); > + > + /** > + * use permute/extract to get status and generation bit > content > + * After the operations, the packets status flags are in the > + * order (hi->lo): [1, 3, 5, 7, 0, 2, 4, 6] > + */ > + > + const __m512i dd_permute_msk =3D _mm512_set_epi64 > + (11, 15, 3, 7, 9, 13, 1, 5); > + const __m512i status0_7 =3D _mm512_permutex2var_epi64 > + (raw_desc4_7, dd_permute_msk, raw_desc0_3); > + const __m512i gen_permute_msk =3D _mm512_set_epi64 > + (10, 14, 2, 6, 8, 12, 0, 4); > + const __m512i raw_gen0_7 =3D _mm512_permutex2var_epi64 > + (raw_desc4_7, gen_permute_msk, raw_desc0_3); > + > + /* now do flag manipulation */ > + > + /** > + * At this point, we have the 8 sets of flags in the low 16-bits > + * of each 32-bit value in vlan0. > + * We want to extract these, and merge them with the mbuf > init > + * data so we can do a single write to the mbuf to set the > flags > + * and all the other initialization fields. Extracting the > + * appropriate flags means that we have to do a shift and > blend > + * for each mbuf before we do the write. However, we can > also > + * add in the previously computed rx_descriptor fields to > + * make a single 256-bit write per mbuf > + */ > + /* check the structure matches expectations */ > + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, ol_flags) !=3D > + offsetof(struct rte_mbuf, rearm_data) + 8); > + RTE_BUILD_BUG_ON(offsetof(struct rte_mbuf, > rearm_data) !=3D > + RTE_ALIGN(offsetof(struct rte_mbuf, > + rearm_data), > + 16)); > + /* build up data and do writes */ > + __m256i rearm0, rearm1, rearm2, rearm3, rearm4, rearm5, > + rearm6, rearm7; > + const __m256i mb4_5 =3D _mm512_extracti64x4_epi64(mb4_7, > 0); > + const __m256i mb6_7 =3D _mm512_extracti64x4_epi64(mb4_7, > 1); > + const __m256i mb0_1 =3D _mm512_extracti64x4_epi64(mb0_3, > 0); > + const __m256i mb2_3 =3D _mm512_extracti64x4_epi64(mb0_3, > 1); > + > + rearm6 =3D _mm256_permute2f128_si256(mbuf_init, mb6_7, > 0x20); > + rearm4 =3D _mm256_permute2f128_si256(mbuf_init, mb4_5, > 0x20); > + rearm2 =3D _mm256_permute2f128_si256(mbuf_init, mb2_3, > 0x20); > + rearm0 =3D _mm256_permute2f128_si256(mbuf_init, mb0_1, > 0x20); > + > + /* write to mbuf */ > + _mm256_storeu_si256((__m256i *)&rx_pkts[i + 6]- > >rearm_data, > + rearm6); > + _mm256_storeu_si256((__m256i *)&rx_pkts[i + 4]- > >rearm_data, > + rearm4); > + _mm256_storeu_si256((__m256i *)&rx_pkts[i + 2]- > >rearm_data, > + rearm2); > + _mm256_storeu_si256((__m256i *)&rx_pkts[i + 0]- > >rearm_data, > + rearm0); > + > + rearm7 =3D _mm256_blend_epi32(mbuf_init, mb6_7, 0xF0); > + rearm5 =3D _mm256_blend_epi32(mbuf_init, mb4_5, 0xF0); > + rearm3 =3D _mm256_blend_epi32(mbuf_init, mb2_3, 0xF0); > + rearm1 =3D _mm256_blend_epi32(mbuf_init, mb0_1, 0xF0); > + > + /* again write to mbufs */ > + _mm256_storeu_si256((__m256i *)&rx_pkts[i + 7]- > >rearm_data, > + rearm7); > + _mm256_storeu_si256((__m256i *)&rx_pkts[i + 5]- > >rearm_data, > + rearm5); > + _mm256_storeu_si256((__m256i *)&rx_pkts[i + 3]- > >rearm_data, > + rearm3); > + _mm256_storeu_si256((__m256i *)&rx_pkts[i + 1]- > >rearm_data, > + rearm1); > + > + const __mmask8 dd_mask =3D _mm512_cmpeq_epi64_mask( > + _mm512_and_epi64(status0_7, dd_check), > dd_check); > + const __mmask8 gen_mask =3D _mm512_cmpeq_epi64_mask( > + _mm512_and_epi64(raw_gen0_7, gen_check), > + _mm512_set1_epi64((uint64_t)rxq- > >expected_gen_id << 46)); > + const __mmask8 recv_mask =3D _kand_mask8(dd_mask, > gen_mask); > + uint16_t burst =3D > __builtin_popcount(_cvtmask8_u32(recv_mask)); > + > + received +=3D burst; > + if (burst !=3D IDPF_DESCS_PER_LOOP_AVX) > + break; > + } > + > + /* update tail pointers */ > + rxq->rx_tail +=3D received; > + rxq->expected_gen_id ^=3D ((rxq->rx_tail & rxq->nb_rx_desc) !=3D 0); > + rxq->rx_tail &=3D (rxq->nb_rx_desc - 1); > + if ((rxq->rx_tail & 1) =3D=3D 1 && received > 1) { /* keep aligned */ > + rxq->rx_tail--; > + received--; > + } > + > + rxq->bufq2->rxrearm_nb +=3D received; > + return received; > +} > + > +/* only bufq2 can receive pkts */ > +uint16_t > +idpf_splitq_recv_pkts_avx512(void *rx_queue, struct rte_mbuf **rx_pkts, > + uint16_t nb_pkts) > +{ > + return _idpf_splitq_recv_raw_pkts_avx512(rx_queue, rx_pkts, > + nb_pkts); > +} > + > static __rte_always_inline int > -idpf_tx_free_bufs_avx512(struct idpf_tx_queue *txq) > +idpf_tx_singleq_free_bufs_avx512(struct idpf_tx_queue *txq) > { > struct idpf_tx_vec_entry *txep; > uint32_t n; > @@ -657,7 +1111,7 @@ tx_backlog_entry_avx512(struct idpf_tx_vec_entry > *txep, >=20 > #define IDPF_FLEX_TXD_QW1_BUF_SZ_S 48 > static __rte_always_inline void > -idpf_vtx1(volatile struct idpf_flex_tx_desc *txdp, > +idpf_singleq_vtx1(volatile struct idpf_flex_tx_desc *txdp, > struct rte_mbuf *pkt, uint64_t flags) { > uint64_t high_qw =3D > @@ -673,7 +1127,7 @@ idpf_vtx1(volatile struct idpf_flex_tx_desc *txdp, > #define IDPF_TX_LEN_MASK 0xAA #define IDPF_TX_OFF_MASK 0x55 static > __rte_always_inline void -idpf_vtx(volatile struct idpf_flex_tx_desc *txd= p, > +idpf_singleq_vtx(volatile struct idpf_flex_tx_desc *txdp, > struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags) { > const uint64_t hi_qw_tmpl =3D (IDPF_TX_DESC_DTYPE_FLEX_DATA | > @@ -681,7 +1135,7 @@ idpf_vtx(volatile struct idpf_flex_tx_desc *txdp, >=20 > /* if unaligned on 32-bit boundary, do one to align */ > if (((uintptr_t)txdp & 0x1F) !=3D 0 && nb_pkts !=3D 0) { > - idpf_vtx1(txdp, *pkt, flags); > + idpf_singleq_vtx1(txdp, *pkt, flags); > nb_pkts--, txdp++, pkt++; > } >=20 > @@ -719,14 +1173,14 @@ idpf_vtx(volatile struct idpf_flex_tx_desc *txdp, >=20 > /* do any last ones */ > while (nb_pkts) { > - idpf_vtx1(txdp, *pkt, flags); > + idpf_singleq_vtx1(txdp, *pkt, flags); > txdp++, pkt++, nb_pkts--; > } > } >=20 > static __rte_always_inline uint16_t > -idpf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf > **tx_pkts, > - uint16_t nb_pkts) > +idpf_singleq_xmit_fixed_burst_vec_avx512(void *tx_queue, struct > rte_mbuf **tx_pkts, > + uint16_t nb_pkts) > { > struct idpf_tx_queue *txq =3D tx_queue; > volatile struct idpf_flex_tx_desc *txdp; @@ -739,7 +1193,7 @@ > idpf_xmit_fixed_burst_vec_avx512(void *tx_queue, struct rte_mbuf > **tx_pkts, > nb_pkts =3D RTE_MIN(nb_pkts, txq->rs_thresh); >=20 > if (txq->nb_free < txq->free_thresh) > - idpf_tx_free_bufs_avx512(txq); > + idpf_tx_singleq_free_bufs_avx512(txq); >=20 > nb_pkts =3D (uint16_t)RTE_MIN(txq->nb_free, nb_pkts); > nb_commit =3D nb_pkts; > @@ -757,11 +1211,11 @@ idpf_xmit_fixed_burst_vec_avx512(void > *tx_queue, struct rte_mbuf **tx_pkts, > if (nb_commit >=3D n) { > tx_backlog_entry_avx512(txep, tx_pkts, n); >=20 > - idpf_vtx(txdp, tx_pkts, n - 1, flags); > + idpf_singleq_vtx(txdp, tx_pkts, n - 1, flags); > tx_pkts +=3D (n - 1); > txdp +=3D (n - 1); >=20 > - idpf_vtx1(txdp, *tx_pkts++, rs); > + idpf_singleq_vtx1(txdp, *tx_pkts++, rs); >=20 > nb_commit =3D (uint16_t)(nb_commit - n); >=20 > @@ -776,7 +1230,7 @@ idpf_xmit_fixed_burst_vec_avx512(void *tx_queue, > struct rte_mbuf **tx_pkts, >=20 > tx_backlog_entry_avx512(txep, tx_pkts, nb_commit); >=20 > - idpf_vtx(txdp, tx_pkts, nb_commit, flags); > + idpf_singleq_vtx(txdp, tx_pkts, nb_commit, flags); >=20 > tx_id =3D (uint16_t)(tx_id + nb_commit); > if (tx_id > txq->next_rs) { > @@ -795,7 +1249,7 @@ idpf_xmit_fixed_burst_vec_avx512(void *tx_queue, > struct rte_mbuf **tx_pkts, } >=20 > static __rte_always_inline uint16_t > -idpf_xmit_pkts_vec_avx512_cmn(void *tx_queue, struct rte_mbuf > **tx_pkts, > +idpf_singleq_xmit_pkts_vec_avx512_cmn(void *tx_queue, struct > rte_mbuf > +**tx_pkts, > uint16_t nb_pkts) > { > uint16_t nb_tx =3D 0; > @@ -805,7 +1259,7 @@ idpf_xmit_pkts_vec_avx512_cmn(void *tx_queue, > struct rte_mbuf **tx_pkts, > uint16_t ret, num; >=20 > num =3D (uint16_t)RTE_MIN(nb_pkts, txq->rs_thresh); > - ret =3D idpf_xmit_fixed_burst_vec_avx512(tx_queue, > &tx_pkts[nb_tx], > + ret =3D idpf_singleq_xmit_fixed_burst_vec_avx512(tx_queue, > +&tx_pkts[nb_tx], > num); > nb_tx +=3D ret; > nb_pkts -=3D ret; > @@ -820,11 +1274,313 @@ uint16_t > idpf_singleq_xmit_pkts_avx512(void *tx_queue, struct rte_mbuf **tx_pkts, > uint16_t nb_pkts) > { > - return idpf_xmit_pkts_vec_avx512_cmn(tx_queue, tx_pkts, > nb_pkts); > + return idpf_singleq_xmit_pkts_vec_avx512_cmn(tx_queue, tx_pkts, > +nb_pkts); } > + > +static __rte_always_inline void > +idpf_splitq_scan_cq_ring(struct idpf_tx_queue *cq) { > + struct idpf_splitq_tx_compl_desc *compl_ring; > + struct idpf_tx_queue *txq; > + uint16_t genid, txq_qid, cq_qid, i; > + uint8_t ctype; > + > + cq_qid =3D cq->tx_tail; > + > + for (i =3D 0; i < IDPD_TXQ_SCAN_CQ_THRESH; i++) { > + if (cq_qid =3D=3D cq->nb_tx_desc) { > + cq_qid =3D 0; > + cq->expected_gen_id ^=3D 1; > + } > + compl_ring =3D &cq->compl_ring[cq_qid]; > + genid =3D (compl_ring->qid_comptype_gen & > + rte_cpu_to_le_64(IDPF_TXD_COMPLQ_GEN_M)) >> > IDPF_TXD_COMPLQ_GEN_S; > + if (genid !=3D cq->expected_gen_id) > + break; > + ctype =3D (rte_le_to_cpu_16(compl_ring->qid_comptype_gen) > & > + IDPF_TXD_COMPLQ_COMPL_TYPE_M) >> > IDPF_TXD_COMPLQ_COMPL_TYPE_S; > + txq_qid =3D (rte_le_to_cpu_16(compl_ring- > >qid_comptype_gen) & > + IDPF_TXD_COMPLQ_QID_M) >> > IDPF_TXD_COMPLQ_QID_S; > + txq =3D cq->txqs[txq_qid - cq->tx_start_qid]; > + txq->ctype[ctype]++; > + cq_qid++; > + } > + > + cq->tx_tail =3D cq_qid; > +} > + > +static __rte_always_inline int > +idpf_tx_splitq_free_bufs_avx512(struct idpf_tx_queue *txq) { > + struct idpf_tx_vec_entry *txep; > + uint32_t n; > + uint32_t i; > + int nb_free =3D 0; > + struct rte_mbuf *m, *free[txq->rs_thresh]; > + > + n =3D txq->rs_thresh; > + > + /* first buffer to free from S/W ring is at index > + * tx_next_dd - (tx_rs_thresh-1) > + */ > + txep =3D (void *)txq->sw_ring; > + txep +=3D txq->next_dd - (n - 1); > + > + if (txq->offloads & IDPF_TX_OFFLOAD_MBUF_FAST_FREE && (n & 31) > =3D=3D 0) { > + struct rte_mempool *mp =3D txep[0].mbuf->pool; > + struct rte_mempool_cache *cache =3D > rte_mempool_default_cache(mp, > + > rte_lcore_id()); > + void **cache_objs; > + > + if (!cache || cache->len =3D=3D 0) > + goto normal; > + > + cache_objs =3D &cache->objs[cache->len]; > + > + if (n > RTE_MEMPOOL_CACHE_MAX_SIZE) { > + rte_mempool_ops_enqueue_bulk(mp, (void *)txep, > n); > + goto done; > + } > + > + /* The cache follows the following algorithm > + * 1. Add the objects to the cache > + * 2. Anything greater than the cache min value (if it crosses > the > + * cache flush threshold) is flushed to the ring. > + */ > + /* Add elements back into the cache */ > + uint32_t copied =3D 0; > + /* n is multiple of 32 */ > + while (copied < n) { > + const __m512i a =3D > _mm512_loadu_si512(&txep[copied]); > + const __m512i b =3D > _mm512_loadu_si512(&txep[copied + 8]); > + const __m512i c =3D > _mm512_loadu_si512(&txep[copied + 16]); > + const __m512i d =3D > _mm512_loadu_si512(&txep[copied + 24]); > + > + _mm512_storeu_si512(&cache_objs[copied], a); > + _mm512_storeu_si512(&cache_objs[copied + 8], b); > + _mm512_storeu_si512(&cache_objs[copied + 16], c); > + _mm512_storeu_si512(&cache_objs[copied + 24], d); > + copied +=3D 32; > + } > + cache->len +=3D n; > + > + if (cache->len >=3D cache->flushthresh) { > + rte_mempool_ops_enqueue_bulk(mp, > + &cache->objs[cache->size], > + cache->len - cache->size); > + cache->len =3D cache->size; > + } > + goto done; > + } > + > +normal: > + m =3D rte_pktmbuf_prefree_seg(txep[0].mbuf); > + if (likely(m)) { > + free[0] =3D m; > + nb_free =3D 1; > + for (i =3D 1; i < n; i++) { > + m =3D rte_pktmbuf_prefree_seg(txep[i].mbuf); > + if (likely(m)) { > + if (likely(m->pool =3D=3D free[0]->pool)) { > + free[nb_free++] =3D m; > + } else { > + rte_mempool_put_bulk(free[0]- > >pool, > + (void *)free, > + nb_free); > + free[0] =3D m; > + nb_free =3D 1; > + } > + } > + } > + rte_mempool_put_bulk(free[0]->pool, (void **)free, > nb_free); > + } else { > + for (i =3D 1; i < n; i++) { > + m =3D rte_pktmbuf_prefree_seg(txep[i].mbuf); > + if (m) > + rte_mempool_put(m->pool, m); > + } > + } > + > +done: > + /* buffers were freed, update counters */ > + txq->nb_free =3D (uint16_t)(txq->nb_free + txq->rs_thresh); > + txq->next_dd =3D (uint16_t)(txq->next_dd + txq->rs_thresh); > + if (txq->next_dd >=3D txq->nb_tx_desc) > + txq->next_dd =3D (uint16_t)(txq->rs_thresh - 1); > + txq->ctype[IDPF_TXD_COMPLT_RS] -=3D txq->rs_thresh; > + > + return txq->rs_thresh; > +} > + > +#define IDPF_TXD_FLEX_QW1_TX_BUF_SZ_S 48 > + > +static __rte_always_inline void > +idpf_splitq_vtx1(volatile struct idpf_flex_tx_sched_desc *txdp, > + struct rte_mbuf *pkt, uint64_t flags) { > + uint64_t high_qw =3D > + (IDPF_TX_DESC_DTYPE_FLEX_FLOW_SCHE | > + ((uint64_t)flags) | > + ((uint64_t)pkt->data_len << > IDPF_TXD_FLEX_QW1_TX_BUF_SZ_S)); > + > + __m128i descriptor =3D _mm_set_epi64x(high_qw, > + pkt->buf_iova + pkt->data_off); > + _mm_storeu_si128((__m128i *)txdp, descriptor); } > + > +static __rte_always_inline void > +idpf_splitq_vtx(volatile struct idpf_flex_tx_sched_desc *txdp, > + struct rte_mbuf **pkt, uint16_t nb_pkts, uint64_t flags) { > + const uint64_t hi_qw_tmpl =3D > (IDPF_TX_DESC_DTYPE_FLEX_FLOW_SCHE | > + ((uint64_t)flags)); > + > + /* if unaligned on 32-bit boundary, do one to align */ > + if (((uintptr_t)txdp & 0x1F) !=3D 0 && nb_pkts !=3D 0) { > + idpf_splitq_vtx1(txdp, *pkt, flags); > + nb_pkts--, txdp++, pkt++; > + } > + > + /* do 4 at a time while possible, in bursts */ > + for (; nb_pkts > 3; txdp +=3D 4, pkt +=3D 4, nb_pkts -=3D 4) { > + uint64_t hi_qw3 =3D > + hi_qw_tmpl | > + ((uint64_t)pkt[3]->data_len << > + IDPF_TXD_FLEX_QW1_TX_BUF_SZ_S); > + uint64_t hi_qw2 =3D > + hi_qw_tmpl | > + ((uint64_t)pkt[2]->data_len << > + IDPF_TXD_FLEX_QW1_TX_BUF_SZ_S); > + uint64_t hi_qw1 =3D > + hi_qw_tmpl | > + ((uint64_t)pkt[1]->data_len << > + IDPF_TXD_FLEX_QW1_TX_BUF_SZ_S); > + uint64_t hi_qw0 =3D > + hi_qw_tmpl | > + ((uint64_t)pkt[0]->data_len << > + IDPF_TXD_FLEX_QW1_TX_BUF_SZ_S); > + > + __m512i desc0_3 =3D > + _mm512_set_epi64 > + (hi_qw3, > + pkt[3]->buf_iova + pkt[3]->data_off, > + hi_qw2, > + pkt[2]->buf_iova + pkt[2]->data_off, > + hi_qw1, > + pkt[1]->buf_iova + pkt[1]->data_off, > + hi_qw0, > + pkt[0]->buf_iova + pkt[0]->data_off); > + _mm512_storeu_si512((void *)txdp, desc0_3); > + } > + > + /* do any last ones */ > + while (nb_pkts) { > + idpf_splitq_vtx1(txdp, *pkt, flags); > + txdp++, pkt++, nb_pkts--; > + } > +} > + > +static __rte_always_inline uint16_t > +idpf_splitq_xmit_fixed_burst_vec_avx512(void *tx_queue, struct > rte_mbuf **tx_pkts, > + uint16_t nb_pkts) > +{ > + struct idpf_tx_queue *txq =3D (struct idpf_tx_queue *)tx_queue; > + volatile struct idpf_flex_tx_sched_desc *txdp; > + struct idpf_tx_vec_entry *txep; > + uint16_t n, nb_commit, tx_id; > + /* bit2 is reserved and must be set to 1 according to Spec */ > + uint64_t cmd_dtype =3D IDPF_TXD_FLEX_FLOW_CMD_EOP; > + > + tx_id =3D txq->tx_tail; > + > + /* cross rx_thresh boundary is not allowed */ > + nb_pkts =3D RTE_MIN(nb_pkts, txq->rs_thresh); > + > + nb_commit =3D nb_pkts =3D (uint16_t)RTE_MIN(txq->nb_free, nb_pkts); > + if (unlikely(nb_pkts =3D=3D 0)) > + return 0; > + > + tx_id =3D txq->tx_tail; > + txdp =3D &txq->desc_ring[tx_id]; > + txep =3D (void *)txq->sw_ring; > + txep +=3D tx_id; > + > + txq->nb_free =3D (uint16_t)(txq->nb_free - nb_pkts); > + > + n =3D (uint16_t)(txq->nb_tx_desc - tx_id); > + if (nb_commit >=3D n) { > + tx_backlog_entry_avx512(txep, tx_pkts, n); > + > + idpf_splitq_vtx((void *)txdp, tx_pkts, n - 1, cmd_dtype); > + tx_pkts +=3D (n - 1); > + txdp +=3D (n - 1); > + > + idpf_splitq_vtx1((void *)txdp, *tx_pkts++, cmd_dtype); > + > + nb_commit =3D (uint16_t)(nb_commit - n); > + > + tx_id =3D 0; > + txq->next_rs =3D (uint16_t)(txq->rs_thresh - 1); > + > + /* avoid reach the end of ring */ > + txdp =3D &txq->desc_ring[tx_id]; > + txep =3D (void *)txq->sw_ring; > + txep +=3D tx_id; > + } > + > + tx_backlog_entry_avx512(txep, tx_pkts, nb_commit); > + > + idpf_splitq_vtx((void *)txdp, tx_pkts, nb_commit, cmd_dtype); > + > + tx_id =3D (uint16_t)(tx_id + nb_commit); > + if (tx_id > txq->next_rs) > + txq->next_rs =3D > + (uint16_t)(txq->next_rs + txq->rs_thresh); > + > + txq->tx_tail =3D tx_id; > + > + IDPF_PCI_REG_WRITE(txq->qtx_tail, txq->tx_tail); > + > + return nb_pkts; > +} > + > +static __rte_always_inline uint16_t > +idpf_splitq_xmit_pkts_vec_avx512_cmn(void *tx_queue, struct rte_mbuf > **tx_pkts, > + uint16_t nb_pkts) > +{ > + struct idpf_tx_queue *txq =3D (struct idpf_tx_queue *)tx_queue; > + uint16_t nb_tx =3D 0; > + > + while (nb_pkts) { > + uint16_t ret, num; > + > + idpf_splitq_scan_cq_ring(txq->complq); > + > + if (txq->ctype[IDPF_TXD_COMPLT_RS] > txq->free_thresh) > + idpf_tx_splitq_free_bufs_avx512(txq); > + > + num =3D (uint16_t)RTE_MIN(nb_pkts, txq->rs_thresh); > + ret =3D idpf_splitq_xmit_fixed_burst_vec_avx512(tx_queue, > + &tx_pkts[nb_tx], > + num); > + nb_tx +=3D ret; > + nb_pkts -=3D ret; > + if (ret < num) > + break; > + } > + > + return nb_tx; > +} > + > +uint16_t > +idpf_splitq_xmit_pkts_avx512(void *tx_queue, struct rte_mbuf **tx_pkts, > + uint16_t nb_pkts) > +{ > + return idpf_splitq_xmit_pkts_vec_avx512_cmn(tx_queue, tx_pkts, > +nb_pkts); > } >=20 > static inline void > -idpf_singleq_tx_release_mbufs_avx512(struct idpf_tx_queue *txq) > +idpf_tx_release_mbufs_avx512(struct idpf_tx_queue *txq) > { > unsigned int i; > const uint16_t max_desc =3D (uint16_t)(txq->nb_tx_desc - 1); @@ - > 843,13 +1599,16 @@ idpf_singleq_tx_release_mbufs_avx512(struct > idpf_tx_queue *txq) > } > } >=20 > -static const struct idpf_txq_ops avx512_singleq_tx_vec_ops =3D { > - .release_mbufs =3D idpf_singleq_tx_release_mbufs_avx512, > +static const struct idpf_txq_ops avx512_tx_vec_ops =3D { > + .release_mbufs =3D idpf_tx_release_mbufs_avx512, > }; >=20 > int __rte_cold > -idpf_singleq_tx_vec_setup_avx512(struct idpf_tx_queue *txq) > +idpf_tx_vec_setup_avx512(struct idpf_tx_queue *txq) > { > - txq->ops =3D &avx512_singleq_tx_vec_ops; > + if (!txq) > + return 0; > + > + txq->ops =3D &avx512_tx_vec_ops; > return 0; > } > diff --git a/drivers/common/idpf/version.map > b/drivers/common/idpf/version.map index 33747a5c69..7018a1d695 100644 > --- a/drivers/common/idpf/version.map > +++ b/drivers/common/idpf/version.map > @@ -44,9 +44,12 @@ INTERNAL { > idpf_singleq_xmit_pkts; > idpf_prep_pkts; > idpf_singleq_rx_vec_setup; > - idpf_singleq_tx_vec_setup_avx512; > + idpf_splitq_rx_vec_setup; > + idpf_tx_vec_setup_avx512; > idpf_singleq_recv_pkts_avx512; > + idpf_splitq_recv_pkts_avx512; > idpf_singleq_xmit_pkts_avx512; > + idpf_splitq_xmit_pkts_avx512; >=20 > local: *; > }; > diff --git a/drivers/net/idpf/idpf_rxtx.c b/drivers/net/idpf/idpf_rxtx.c = index > b6d4e5abd0..d4e1c14402 100644 > --- a/drivers/net/idpf/idpf_rxtx.c > +++ b/drivers/net/idpf/idpf_rxtx.c > @@ -761,7 +761,8 @@ idpf_set_rx_function(struct rte_eth_dev *dev) > if (rte_vect_get_max_simd_bitwidth() >=3D > RTE_VECT_SIMD_512) #ifdef CC_AVX512_SUPPORT > if > (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) =3D=3D 1 && > - > rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) =3D=3D 1) > + > rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) =3D=3D 1 && > + > rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512DQ)) > vport->rx_use_avx512 =3D true; > #else > PMD_DRV_LOG(NOTICE, > @@ -774,10 +775,24 @@ idpf_set_rx_function(struct rte_eth_dev *dev) >=20 > #ifdef RTE_ARCH_X86 > if (vport->rxq_model =3D=3D VIRTCHNL2_QUEUE_MODEL_SPLIT) { > +#ifdef RTE_ARCH_X86 > + if (vport->rx_vec_allowed) { > + for (i =3D 0; i < dev->data->nb_rx_queues; i++) { > + rxq =3D dev->data->rx_queues[i]; > + (void)idpf_splitq_rx_vec_setup(rxq); > + } > +#ifdef CC_AVX512_SUPPORT > + if (vport->rx_use_avx512) { > + dev->rx_pkt_burst =3D > idpf_splitq_recv_pkts_avx512; > + return; > + } > +#endif > + } > +#endif > dev->rx_pkt_burst =3D idpf_splitq_recv_pkts; > } else { > if (vport->rx_vec_allowed) { > - for (i =3D 0; i < dev->data->nb_tx_queues; i++) { > + for (i =3D 0; i < dev->data->nb_rx_queues; i++) { > rxq =3D dev->data->rx_queues[i]; > (void)idpf_singleq_rx_vec_setup(rxq); > } > @@ -809,14 +824,22 @@ idpf_set_tx_function(struct rte_eth_dev *dev) > int i; > #endif /* CC_AVX512_SUPPORT */ >=20 > - if (idpf_rx_vec_dev_check_default(dev) =3D=3D IDPF_VECTOR_PATH && > + if (idpf_tx_vec_dev_check_default(dev) =3D=3D IDPF_VECTOR_PATH && > rte_vect_get_max_simd_bitwidth() >=3D RTE_VECT_SIMD_128) { > vport->tx_vec_allowed =3D true; > if (rte_vect_get_max_simd_bitwidth() >=3D > RTE_VECT_SIMD_512) #ifdef CC_AVX512_SUPPORT > + { > if > (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512F) =3D=3D 1 && >=20 > rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX512BW) =3D=3D 1) > vport->tx_use_avx512 =3D true; > + if (vport->tx_use_avx512) { > + for (i =3D 0; i < dev->data->nb_tx_queues; i++) > { > + txq =3D dev->data->tx_queues[i]; > + idpf_tx_vec_setup_avx512(txq); > + } > + } > + } > #else > PMD_DRV_LOG(NOTICE, > "AVX512 is not supported in build env"); @@ -827,6 > +850,17 @@ idpf_set_tx_function(struct rte_eth_dev *dev) #endif /* > RTE_ARCH_X86 */ >=20 > if (vport->txq_model =3D=3D VIRTCHNL2_QUEUE_MODEL_SPLIT) { > +#ifdef RTE_ARCH_X86 > + if (vport->tx_vec_allowed) { > +#ifdef CC_AVX512_SUPPORT > + if (vport->tx_use_avx512) { > + dev->tx_pkt_burst =3D > idpf_splitq_xmit_pkts_avx512; > + dev->tx_pkt_prepare =3D idpf_prep_pkts; > + return; > + } > +#endif > + } > +#endif > dev->tx_pkt_burst =3D idpf_splitq_xmit_pkts; > dev->tx_pkt_prepare =3D idpf_prep_pkts; > } else { > @@ -834,12 +868,6 @@ idpf_set_tx_function(struct rte_eth_dev *dev) > if (vport->tx_vec_allowed) { > #ifdef CC_AVX512_SUPPORT > if (vport->tx_use_avx512) { > - for (i =3D 0; i < dev->data->nb_tx_queues; i++) > { > - txq =3D dev->data->tx_queues[i]; > - if (txq =3D=3D NULL) > - continue; > - > idpf_singleq_tx_vec_setup_avx512(txq); > - } > dev->tx_pkt_burst =3D > idpf_singleq_xmit_pkts_avx512; > dev->tx_pkt_prepare =3D idpf_prep_pkts; > return; > diff --git a/drivers/net/idpf/idpf_rxtx_vec_common.h > b/drivers/net/idpf/idpf_rxtx_vec_common.h > index 0f4e10e154..366870d1db 100644 > --- a/drivers/net/idpf/idpf_rxtx_vec_common.h > +++ b/drivers/net/idpf/idpf_rxtx_vec_common.h > @@ -1,4 +1,4 @@ > -/* SPDX-License-Identifier: BSD-3-Clause > +/* SPDX-LIDPFnse-Identifier: BSD-3-Clause > * Copyright(c) 2022 Intel Corporation > */ >=20 > @@ -15,14 +15,15 @@ > #pragma GCC diagnostic ignored "-Wcast-qual" > #endif >=20 > -#define IDPF_VECTOR_PATH 0 > -#define ICE_RX_NO_VECTOR_FLAGS ( \ > +#define IDPF_SCALAR_PATH 0 > +#define IDPF_VECTOR_PATH 1 > +#define IDPF_RX_NO_VECTOR_FLAGS ( \ > RTE_ETH_RX_OFFLOAD_IPV4_CKSUM | \ > RTE_ETH_RX_OFFLOAD_UDP_CKSUM | \ > RTE_ETH_RX_OFFLOAD_TCP_CKSUM | \ > RTE_ETH_RX_OFFLOAD_OUTER_IPV4_CKSUM | \ > RTE_ETH_RX_OFFLOAD_TIMESTAMP) > -#define ICE_TX_NO_VECTOR_FLAGS ( \ > +#define IDPF_TX_NO_VECTOR_FLAGS ( \ > RTE_ETH_TX_OFFLOAD_TCP_TSO | \ > RTE_ETH_TX_OFFLOAD_MULTI_SEGS) >=20 > @@ -30,19 +31,19 @@ static inline int > idpf_rx_vec_queue_default(struct idpf_rx_queue *rxq) { > if (rxq =3D=3D NULL) > - return -1; > + return IDPF_SCALAR_PATH; >=20 > if (rte_is_power_of_2(rxq->nb_rx_desc) =3D=3D 0) > - return -1; > + return IDPF_SCALAR_PATH; >=20 > if (rxq->rx_free_thresh < IDPF_VPMD_RX_MAX_BURST) > - return -1; > + return IDPF_SCALAR_PATH; >=20 > if ((rxq->nb_rx_desc % rxq->rx_free_thresh) !=3D 0) > - return -1; > + return IDPF_SCALAR_PATH; >=20 > - if ((rxq->offloads & ICE_RX_NO_VECTOR_FLAGS) !=3D 0) > - return -1; > + if ((rxq->offloads & IDPF_RX_NO_VECTOR_FLAGS) !=3D 0) > + return IDPF_SCALAR_PATH; >=20 > return IDPF_VECTOR_PATH; > } > @@ -51,14 +52,23 @@ static inline int > idpf_tx_vec_queue_default(struct idpf_tx_queue *txq) { > if (txq =3D=3D NULL) > - return -1; > + return IDPF_SCALAR_PATH; >=20 > if (txq->rs_thresh < IDPF_VPMD_TX_MAX_BURST || > (txq->rs_thresh & 3) !=3D 0) > - return -1; > + return IDPF_SCALAR_PATH; >=20 > - if ((txq->offloads & ICE_TX_NO_VECTOR_FLAGS) !=3D 0) > - return -1; > + if ((txq->offloads & IDPF_TX_NO_VECTOR_FLAGS) !=3D 0) > + return IDPF_SCALAR_PATH; > + > + return IDPF_VECTOR_PATH; > +} > + > +static inline int > +idpf_rx_splitq_vec_default(struct idpf_rx_queue *rxq) { > + if (rxq->bufq2->rx_buf_len < rxq->max_pkt_len) > + return IDPF_SCALAR_PATH; >=20 > return IDPF_VECTOR_PATH; > } > @@ -66,15 +76,18 @@ idpf_tx_vec_queue_default(struct idpf_tx_queue > *txq) static inline int idpf_rx_vec_dev_check_default(struct rte_eth_de= v > *dev) { > - int i; > + struct idpf_vport *vport =3D dev->data->dev_private; > struct idpf_rx_queue *rxq; > - int ret =3D 0; > + int i, default_ret, splitq_ret, ret =3D IDPF_SCALAR_PATH; >=20 > for (i =3D 0; i < dev->data->nb_rx_queues; i++) { > rxq =3D dev->data->rx_queues[i]; > - ret =3D (idpf_rx_vec_queue_default(rxq)); > - if (ret < 0) > - return -1; > + splitq_ret =3D idpf_rx_splitq_vec_default(rxq); > + default_ret =3D idpf_rx_vec_queue_default(rxq); > + ret =3D (vport->rxq_model =3D=3D > VIRTCHNL2_QUEUE_MODEL_SPLIT) ? > + splitq_ret && default_ret : default_ret; > + if (ret =3D=3D IDPF_SCALAR_PATH) > + return IDPF_SCALAR_PATH; > } >=20 > return IDPF_VECTOR_PATH; > @@ -90,8 +103,8 @@ idpf_tx_vec_dev_check_default(struct rte_eth_dev > *dev) > for (i =3D 0; i < dev->data->nb_tx_queues; i++) { > txq =3D dev->data->tx_queues[i]; > ret =3D idpf_tx_vec_queue_default(txq); > - if (ret < 0) > - return -1; > + if (ret =3D=3D IDPF_SCALAR_PATH) > + return IDPF_SCALAR_PATH; > } >=20 > return IDPF_VECTOR_PATH; > -- > 2.34.1