From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 521B2A00C5; Thu, 15 Sep 2022 09:06:35 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3F5B64021D; Thu, 15 Sep 2022 09:06:35 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by mails.dpdk.org (Postfix) with ESMTP id 2A56F40156 for ; Thu, 15 Sep 2022 09:06:33 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28ENmx6s024421; Thu, 15 Sep 2022 00:06:26 -0700 Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2177.outbound.protection.outlook.com [104.47.57.177]) by mx0b-0016f401.pphosted.com (PPS) with ESMTPS id 3jjy0272s7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 15 Sep 2022 00:06:26 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Lhasik85VHipJGn2BGqwhZFVsf53iuEeow2WdBGydEDXfwKHXq5jmylaA8cIUCcTQSsE6yNCVaYv15Z6BU3aPmE20S4posBlcUFMaFtDEpaIvXbnOz7zvGo8xhq8HDsaeS8t0+qNZaTLmA7LSKcwJQmKzqUy4OO+Gb3VyM0RssvYHePYtHsYORVcyhOQnJd9/RY4rj9skVjkD7sowoouzN7aSbmeCk1MbUAkbMHdAw9VnvIHQia5SigIuf4/0UMaPfgY8rMrwe3wv6iTCmyT9OZ45yEXlWeVEJNcuxjJSbk1u9lxmZwLCdqgbk8Eb8KVLjHljWTvpmQUjQLgVoTLaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=OzrsIcPj6+2xteIEKfDtV2ausVcZOnplfaczEclhr8A=; b=Dvitzmwbc+CHeoyZbC7FAZSdARnCtajjoSEZ/hZ0iLbbVLWSKKHs1TjXRLEP3H44nnhnwUdALG/5bdx5/GCj+Fl+ygsEbSReRykAboVjv+wmbLpEe4CPWplKfTxlJu1VY2nQn7WtMAemBdkNeQDCm84Ow1IJ5w6p/H9Nww/7apYOi9LF9JzSqgY4b9AG7F+ubn53RP5O6BBVlzVHBO4/9AkYa9td9o0UJ1llaR9lX6+7C8FQV77RNBgzsMCN8d09kTvCkPmy2IjoRlYrgr6dtBCGo/IM0vEEBheSBslN8lc8cX3kfYwZF19GZnGi9MHRb4qIoAz1fQR7X773okyiGQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OzrsIcPj6+2xteIEKfDtV2ausVcZOnplfaczEclhr8A=; b=g4rNJENFJY9Rz7b2nNv6VuOxwVy+gSldVBTW2t/XhWYHnclKI8KiLLgNSj33UJjXDDVI7EraS8i7fKtms1sc7p0iOJmE3iU/5anCiQKqIOlFKWjwPSw3i8fjYnZZd0wbJDlk+aW1A8ruGmjuxCOFD53arneqQNSiNDHctJTgHeM= Received: from CO6PR18MB4484.namprd18.prod.outlook.com (2603:10b6:5:359::9) by DM4PR18MB4126.namprd18.prod.outlook.com (2603:10b6:5:39a::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5612.22; Thu, 15 Sep 2022 07:06:24 +0000 Received: from CO6PR18MB4484.namprd18.prod.outlook.com ([fe80::6d7f:3f2f:935d:7319]) by CO6PR18MB4484.namprd18.prod.outlook.com ([fe80::6d7f:3f2f:935d:7319%9]) with mapi id 15.20.5632.015; Thu, 15 Sep 2022 07:06:23 +0000 From: Akhil Goyal To: Fan Zhang , "dev@dpdk.org" CC: "maxime.coquelin@redhat.com" , "chandu@amd.com" , "ruifeng.wang@arm.com" , "ajit.khaparde@broadcom.com" , Anoob Joseph , "pablo.de.lara.guarch@intel.com" , "matan@nvidia.com" , "g.singh@nxp.com" , "jianjay.zhou@huawei.com" Subject: RE: [EXT] [PATCH 2/3] crypto/scheduler: use unified session Thread-Topic: [EXT] [PATCH 2/3] crypto/scheduler: use unified session Thread-Index: AQHYu8Mz3wMq6dPRQ0GxTja9il7ud63gK1jQ Date: Thu, 15 Sep 2022 07:06:23 +0000 Message-ID: References: <20220829160645.378406-1-roy.fan.zhang@intel.com> <20220829160645.378406-3-roy.fan.zhang@intel.com> In-Reply-To: <20220829160645.378406-3-roy.fan.zhang@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: CO6PR18MB4484:EE_|DM4PR18MB4126:EE_ x-ms-office365-filtering-correlation-id: e1f436fb-3947-4b5b-4804-08da96e8cc53 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Dwhj2yWnv5cpFm2XWXxYU6BxCM2yZEy/AoeIW9gHR+TItJ0WzpSb6vJgWuryKfbevh5ye5zPQlO9VK9TCRnfng3DeaPBD6n5+Z77J3ENgUgJHdXDoT2I7SKl41wLbzHhUfSGD+2466QIdqGgHigT2zos3jKqqTwipnPhkRA8oGch+MOydWpqnq9fap8MWix8bcItJ9MCbIH36sHF7bP3ku1B1nP6KIyhbVwPhc48l9P/dZwvLdnHFzLgW/rYLEVOHG8n04pKeQQcpTwntUlGEA4yI0lK1Q8+vARdeGGD56IxHcq/R/GSy7BUtUofnI6Ue4TZn3wV3IV3ST6Ofe2NMmSs4ZLOmG7BFK/9MgJSzb/dX4MoHmpQNCn6MGeSYX5r+mc3sziLwyQfkZk70fQQkpHIG9Xzbz222SOCQU3XSMIzynLCrP9Ocjiz3+PatgIQN7RDh9xaF+ZO7Qsg1AJWd70TdAnrxwcY2z/X3c3A9sJhtJ649qEyjQCH37tMcRSxEBxyXQ/wwU4QbsA8qNX+0hi7kiwjIv4G2aQlmBsDlM6Rdq/X2BbmYZ2eYpIiaVNzZnYHZNNjllhf2AWkzI2sJmdUVkHn+dzO/iEkHbMr0V8/wIrLsJ4LPsUmV/f+ZRNRuJsErpv83L+lRKusdjJQ8y4BDTXapleQuCHU9ZpiGQydUW1T5bdOlZI1j/nwnODgzGQq8M/yY9gFMj0y3pinyH6RMqxzGAAMNSyyiqvPxgzaFrkRYPSpnZunJfC2uDTKZ/BApv/AlTbhZqzjEMo1NA== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CO6PR18MB4484.namprd18.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230022)(4636009)(136003)(39860400002)(396003)(346002)(376002)(366004)(451199015)(30864003)(55016003)(110136005)(33656002)(4326008)(26005)(2906002)(316002)(8936002)(76116006)(52536014)(66946007)(41300700001)(6506007)(66446008)(66556008)(83380400001)(8676002)(64756008)(7416002)(186003)(86362001)(54906003)(71200400001)(478600001)(122000001)(9686003)(5660300002)(38100700002)(38070700005)(55236004)(7696005)(66476007)(579004)(559001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?tRfzhyLh8NTdjU4CW10QOgF+vBVwtIb+mJ9mtz0TshaD++DeIn1b0Q85RnUW?= =?us-ascii?Q?SifdSd3yTXlgJ4FwzPU0HjbQ0y9dd3CqOUHhkgrbDrosfW5z5PCPVABiYqIB?= =?us-ascii?Q?MFdIZ3YTZr/9CiByEyhVVOMwkwdiwQWxXnjZyaAPtmm+a1qo86y7PAf1eVOJ?= =?us-ascii?Q?/XGwjR+SF39DXx7WQ8CZLnEJxWsE3YtXklUndJRERxD2YEw3jYYyRm9xKIz3?= =?us-ascii?Q?5yku7kgm+B6hjZYOGB165UXuzV0nQ9xZBGv8rcEyxkTjb+QOsEm05XdPLMQw?= =?us-ascii?Q?XXBi/ZNkcHlEsl3G9sujRO92EbO0siUPMfJijsXRZycoDTefCMAyA6hLqOvX?= =?us-ascii?Q?LvhcToENaQDo9Yj3yWh7hyqoY3OUm/kD+VXwJ7BOcUEACLRb/57YNkGMuN+/?= =?us-ascii?Q?mW5GOrs35aPQbwFxxTWgVPG2zlSiO6S37RFFXci7c0NrbMhdZFKa/T348wB2?= =?us-ascii?Q?lpc3gX9FxSxOlQSHqQHzJqEUWAsbe3KFxtPiqmFAn6rf3GOeRUnMcqKDt1PI?= =?us-ascii?Q?mwVREmw48oYrVYiN/jLciYOKhByaq3FiG+0HHNa4p4rMjv1byaHFVIfZ8ZLg?= =?us-ascii?Q?GU9H4eGIB4X/gPiqpJPSSJp6CABjPAvxVQrDhibRZTcBpr3QHFo+iz2m10zB?= =?us-ascii?Q?tOFrsSKgC0l1mBs+YI94qyO1Ou91GPL9+WUpBhkWy+0ILTWbP5vQDV+UBC9h?= =?us-ascii?Q?0I5+sLn42TxeAuqq8zJyJp15aWocFGqwn3HtwVPUTLTCmjTz7/NrQokEYsYF?= =?us-ascii?Q?oIwoELPnyH3lfmXztxb9bdarvoecQf0vpQj4p1UhSicQza3tJ16t91qYVTwn?= =?us-ascii?Q?tWIwyDOCoVLYBUt1xEQB1mv8Rd3dIjwzwdyKkVUTJi6kC9Z4cEm9moxQpPts?= =?us-ascii?Q?zHs9FWOGoQIDTap2DaXKfRKUhJzEMaZA45eJhlvYZu9yVdJ+fGgyPrgR+2IH?= =?us-ascii?Q?2kM8ZrZE8GdDqaLY6kl2Go2YC+FaFD0vLfIgXmebSFfrOPOOpzHU+1k2s5KJ?= =?us-ascii?Q?YQ2Ca443e8LkyaeahdKXEbEoy0zA4KnQBdfhbeIpOJPLSkK1iqyURaY+paTM?= =?us-ascii?Q?44zMyVVj7nokBpmmipd8NkLaMqrqcgAKpyn+KU8PpJcLQJdRYpKDMU7VeVlh?= =?us-ascii?Q?BzZ99AXFwiXCCV4O9yl8HW0GcgpgkDJ5hW/1LJyt3J3E5hioeiW7VgNsXpuK?= =?us-ascii?Q?g5AWfZIoR1s/HFqz4Q1CHT3Rfae2x0Z3B1hc2VoVRZbbNw8eMmVFkD/3Gy+X?= =?us-ascii?Q?xIzkO1QtxofZTQwkqL7mLN+rE9GM/I6HobIMmSkck4qjqRsd+gp7lxBt2oF2?= =?us-ascii?Q?nXccWDADG+2qqXOBar45qEDNS4gNELpZSrleMxom6zHHsF1zUiAbD32uKEvG?= =?us-ascii?Q?IT2PTjDA7nAVgm3qP1RtthJGy+V5SSZxrrtcK5+sNGSTL5EZJnD36q4urQiU?= =?us-ascii?Q?iXOJTO5QcrZwa1IbBJyW+NlXyQvSQnN2wbbITDPeYnwplFWnVkQ3ICmrnB5n?= =?us-ascii?Q?odsInwB3MJhylGMtQyGkmu5lSiDz/8wmxs1sIu/l+6FwbfCndZAFI6ztUQO+?= =?us-ascii?Q?2iSg6+uoL/6NbFneAiYzbLZ1K2d87yssL4sSYuts?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: marvell.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CO6PR18MB4484.namprd18.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: e1f436fb-3947-4b5b-4804-08da96e8cc53 X-MS-Exchange-CrossTenant-originalarrivaltime: 15 Sep 2022 07:06:23.0779 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 3JsmKXCIlhmGLBgrK1O8MYkwqEUU/agTYATcSAd9HHwqVv0FYRa3kudrBU5CRQAdL+8sJ5bOp7mjPArg+2pbAA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR18MB4126 X-Proofpoint-GUID: 81qnz39DYrC232K4793bcsm5GwnuGuWp X-Proofpoint-ORIG-GUID: 81qnz39DYrC232K4793bcsm5GwnuGuWp X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-15_03,2022-09-14_04,2022-06-22_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Fan, Are these patches getting compiled at your end? Are these a working patchse= t on Intel platform? For me it is failing for each of the patch 'drivers/a715181@@tmp_rte_crypto_scheduler@sta/crypto_scheduler_scheduler_p= md_ops.c.o' -MF 'drivers/a715181@@tmp_rte_crypto_scheduler@sta/crypto_sched= uler_scheduler_pmd_ops.c.o.d' -o 'drivers/a715181@@tmp_rte_crypto_scheduler= @sta/crypto_scheduler_scheduler_pmd_ops.c.o' -c ../drivers/crypto/scheduler= /scheduler_pmd_ops.c ../drivers/crypto/scheduler/scheduler_pmd_ops.c: In function 'scheduler_pmd= _sym_session_configure': ../drivers/crypto/scheduler/scheduler_pmd_ops.c:525:1: error: label at end = of compound statement 525 | next_worker: | ^~~~~~~~~~~ ../drivers/crypto/scheduler/scheduler_pmd_ops.c: In function 'scheduler_pmd= _sym_session_clear': ../drivers/crypto/scheduler/scheduler_pmd_ops.c:569:1: error: label at end = of compound statement 569 | next_worker: | ^~~~~~~~~~~ > This patch updates the scheduler PMD to use unified session > data structure. Previously thanks to the private session > array in cryptodev sym session there are no necessary > change needed for scheduler PMD other than the way ops > are enqueued/dequeued. The patch inherits the same design > in the original session data structure to the scheduler PMD > so the cryptodev sym session can be as a linear buffer for > both session header and driver private data. >=20 > With the change there are inevitable extra cost on both memory > (64 bytes per session per driver type) and cycle count (set > the correct session for each cop based on the worker before > enqueue, and retrieve the original session after dequeue). >=20 > Signed-off-by: Fan Zhang > --- > app/test/test_cryptodev.c | 4 +- > drivers/crypto/scheduler/scheduler_failover.c | 19 ++- > .../crypto/scheduler/scheduler_multicore.c | 17 +++ > .../scheduler/scheduler_pkt_size_distr.c | 84 +++++++++--- > drivers/crypto/scheduler/scheduler_pmd_ops.c | 101 ++++++++++++++- > .../crypto/scheduler/scheduler_pmd_private.h | 120 +++++++++++++++++- > .../crypto/scheduler/scheduler_roundrobin.c | 11 +- > 7 files changed, 314 insertions(+), 42 deletions(-) >=20 > diff --git a/app/test/test_cryptodev.c b/app/test/test_cryptodev.c > index c975c38001..e181b0aa3e 100644 > --- a/app/test/test_cryptodev.c > +++ b/app/test/test_cryptodev.c > @@ -14986,8 +14986,8 @@ test_scheduler_attach_worker_op(void) > ts_params->session_mpool =3D > rte_cryptodev_sym_session_pool_create( > "test_sess_mp", > - MAX_NB_SESSIONS, 0, 0, 0, > - SOCKET_ID_ANY); > + MAX_NB_SESSIONS, > session_size, > + 0, 0, SOCKET_ID_ANY); > TEST_ASSERT_NOT_NULL(ts_params->session_mpool, > "session mempool allocation failed"); > } > diff --git a/drivers/crypto/scheduler/scheduler_failover.c > b/drivers/crypto/scheduler/scheduler_failover.c > index 2a0e29fa72..7fadcf66d0 100644 > --- a/drivers/crypto/scheduler/scheduler_failover.c > +++ b/drivers/crypto/scheduler/scheduler_failover.c > @@ -16,18 +16,19 @@ > struct fo_scheduler_qp_ctx { > struct scheduler_worker primary_worker; > struct scheduler_worker secondary_worker; > + uint8_t primary_worker_index; > + uint8_t secondary_worker_index; >=20 > uint8_t deq_idx; > }; >=20 > static __rte_always_inline uint16_t > failover_worker_enqueue(struct scheduler_worker *worker, > - struct rte_crypto_op **ops, uint16_t nb_ops) > + struct rte_crypto_op **ops, uint16_t nb_ops, uint8_t index) > { > - uint16_t i, processed_ops; > + uint16_t processed_ops; >=20 > - for (i =3D 0; i < nb_ops && i < 4; i++) > - rte_prefetch0(ops[i]->sym->session); > + scheduler_set_worker_session(ops, nb_ops, index); >=20 > processed_ops =3D rte_cryptodev_enqueue_burst(worker->dev_id, > worker->qp_id, ops, nb_ops); > @@ -47,13 +48,14 @@ schedule_enqueue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > return 0; >=20 > enqueued_ops =3D failover_worker_enqueue(&qp_ctx->primary_worker, > - ops, nb_ops); > + ops, nb_ops, PRIMARY_WORKER_IDX); >=20 > if (enqueued_ops < nb_ops) > enqueued_ops +=3D failover_worker_enqueue( > &qp_ctx->secondary_worker, > &ops[enqueued_ops], > - nb_ops - enqueued_ops); > + nb_ops - enqueued_ops, > + SECONDARY_WORKER_IDX); >=20 > return enqueued_ops; > } > @@ -94,7 +96,7 @@ schedule_dequeue(void *qp, struct rte_crypto_op **ops, > uint16_t nb_ops) > qp_ctx->deq_idx =3D (~qp_ctx->deq_idx) & WORKER_SWITCH_MASK; >=20 > if (nb_deq_ops =3D=3D nb_ops) > - return nb_deq_ops; > + goto retrieve_session; >=20 > worker =3D workers[qp_ctx->deq_idx]; >=20 > @@ -104,6 +106,9 @@ schedule_dequeue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > worker->nb_inflight_cops -=3D nb_deq_ops2; > } >=20 > +retrieve_session: > + scheduler_retrieve_session(ops, nb_deq_ops + nb_deq_ops2); > + > return nb_deq_ops + nb_deq_ops2; > } >=20 > diff --git a/drivers/crypto/scheduler/scheduler_multicore.c > b/drivers/crypto/scheduler/scheduler_multicore.c > index 900ab4049d..3dea850661 100644 > --- a/drivers/crypto/scheduler/scheduler_multicore.c > +++ b/drivers/crypto/scheduler/scheduler_multicore.c > @@ -183,11 +183,19 @@ mc_scheduler_worker(struct rte_cryptodev *dev) >=20 > while (!mc_ctx->stop_signal) { > if (pending_enq_ops) { > + scheduler_set_worker_session( > + &enq_ops[pending_enq_ops_idx], > pending_enq_ops, > + worker_idx); > processed_ops =3D > rte_cryptodev_enqueue_burst(worker->dev_id, > worker->qp_id, > &enq_ops[pending_enq_ops_idx], > pending_enq_ops); > + if (processed_ops < pending_deq_ops) > + scheduler_retrieve_session( > + &enq_ops[pending_enq_ops_idx + > + processed_ops], > + pending_deq_ops - processed_ops); > pending_enq_ops -=3D processed_ops; > pending_enq_ops_idx +=3D processed_ops; > inflight_ops +=3D processed_ops; > @@ -195,9 +203,16 @@ mc_scheduler_worker(struct rte_cryptodev *dev) > processed_ops =3D rte_ring_dequeue_burst(enq_ring, > (void *)enq_ops, >=20 > MC_SCHED_BUFFER_SIZE, NULL); > if (processed_ops) { > + scheduler_set_worker_session(enq_ops, > + processed_ops, worker_idx); > pending_enq_ops_idx =3D > rte_cryptodev_enqueue_burst( > worker->dev_id, worker- > >qp_id, > enq_ops, processed_ops); > + if (pending_enq_ops_idx < processed_ops) > + scheduler_retrieve_session( > + enq_ops + > pending_enq_ops_idx, > + processed_ops - > + pending_enq_ops_idx); > pending_enq_ops =3D processed_ops - > pending_enq_ops_idx; > inflight_ops +=3D pending_enq_ops_idx; > } > @@ -214,6 +229,8 @@ mc_scheduler_worker(struct rte_cryptodev *dev) > worker->dev_id, worker->qp_id, > deq_ops, > MC_SCHED_BUFFER_SIZE); > if (processed_ops) { > + scheduler_retrieve_session(deq_ops, > + processed_ops); > inflight_ops -=3D processed_ops; > if (reordering_enabled) { > uint16_t j; > diff --git a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c > b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c > index 933a5c6978..9204f6f608 100644 > --- a/drivers/crypto/scheduler/scheduler_pkt_size_distr.c > +++ b/drivers/crypto/scheduler/scheduler_pkt_size_distr.c > @@ -48,34 +48,54 @@ schedule_enqueue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > }; > struct psd_schedule_op *p_enq_op; > uint16_t i, processed_ops_pri =3D 0, processed_ops_sec =3D 0; > - uint32_t job_len; >=20 > if (unlikely(nb_ops =3D=3D 0)) > return 0; >=20 > for (i =3D 0; i < nb_ops && i < 4; i++) { > rte_prefetch0(ops[i]->sym); > - rte_prefetch0(ops[i]->sym->session); > + rte_prefetch0((uint8_t *)ops[i]->sym->session + > + sizeof(struct rte_cryptodev_sym_session)); > } >=20 > for (i =3D 0; (i < (nb_ops - 8)) && (nb_ops > 8); i +=3D 4) { > + struct scheduler_session_ctx *sess_ctx[4]; > + uint8_t target[4]; > + uint32_t job_len[4]; > + > rte_prefetch0(ops[i + 4]->sym); > - rte_prefetch0(ops[i + 4]->sym->session); > + rte_prefetch0((uint8_t *)ops[i + 4]->sym->session + > + sizeof(struct rte_cryptodev_sym_session)); > rte_prefetch0(ops[i + 5]->sym); > - rte_prefetch0(ops[i + 5]->sym->session); > + rte_prefetch0((uint8_t *)ops[i + 5]->sym->session + > + sizeof(struct rte_cryptodev_sym_session)); > rte_prefetch0(ops[i + 6]->sym); > - rte_prefetch0(ops[i + 6]->sym->session); > + rte_prefetch0((uint8_t *)ops[i + 6]->sym->session + > + sizeof(struct rte_cryptodev_sym_session)); > rte_prefetch0(ops[i + 7]->sym); > - rte_prefetch0(ops[i + 7]->sym->session); > + rte_prefetch0((uint8_t *)ops[i + 7]->sym->session + > + sizeof(struct rte_cryptodev_sym_session)); > + > + sess_ctx[0] =3D (void *)ops[i]->sym->session->driver_priv_data; > + sess_ctx[1] =3D > + (void *)ops[i + 1]->sym->session->driver_priv_data; > + sess_ctx[2] =3D > + (void *)ops[i + 2]->sym->session->driver_priv_data; > + sess_ctx[3] =3D > + (void *)ops[i + 3]->sym->session->driver_priv_data; >=20 > /* job_len is initialized as cipher data length, once > * it is 0, equals to auth data length > */ > - job_len =3D ops[i]->sym->cipher.data.length; > - job_len +=3D (ops[i]->sym->cipher.data.length =3D=3D 0) * > + job_len[0] =3D ops[i]->sym->cipher.data.length; > + job_len[0] +=3D (ops[i]->sym->cipher.data.length =3D=3D 0) * > ops[i]->sym->auth.data.length; > /* decide the target op based on the job length */ > - p_enq_op =3D &enq_ops[!(job_len & psd_qp_ctx->threshold)]; > + target[0] =3D !(job_len[0] & psd_qp_ctx->threshold); > + if (ops[i]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + ops[i]->sym->session =3D > + sess_ctx[0]->worker_sess[target[0]]; > + p_enq_op =3D &enq_ops[target[0]]; >=20 > /* stop schedule cops before the queue is full, this shall > * prevent the failed enqueue > @@ -89,10 +109,14 @@ schedule_enqueue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > sched_ops[p_enq_op->worker_idx][p_enq_op->pos] =3D ops[i]; > p_enq_op->pos++; >=20 > - job_len =3D ops[i+1]->sym->cipher.data.length; > - job_len +=3D (ops[i+1]->sym->cipher.data.length =3D=3D 0) * > + job_len[1] =3D ops[i + 1]->sym->cipher.data.length; > + job_len[1] +=3D (ops[i + 1]->sym->cipher.data.length =3D=3D 0) * > ops[i+1]->sym->auth.data.length; > - p_enq_op =3D &enq_ops[!(job_len & psd_qp_ctx->threshold)]; > + target[1] =3D !(job_len[1] & psd_qp_ctx->threshold); > + if (ops[i + 1]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + ops[i + 1]->sym->session =3D > + sess_ctx[1]->worker_sess[target[1]]; > + p_enq_op =3D &enq_ops[target[1]]; >=20 > if (p_enq_op->pos + in_flight_ops[p_enq_op->worker_idx] =3D=3D > qp_ctx->max_nb_objs) { > @@ -103,10 +127,14 @@ schedule_enqueue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > sched_ops[p_enq_op->worker_idx][p_enq_op->pos] =3D ops[i+1]; > p_enq_op->pos++; >=20 > - job_len =3D ops[i+2]->sym->cipher.data.length; > - job_len +=3D (ops[i+2]->sym->cipher.data.length =3D=3D 0) * > - ops[i+2]->sym->auth.data.length; > - p_enq_op =3D &enq_ops[!(job_len & psd_qp_ctx->threshold)]; > + job_len[2] =3D ops[i + 2]->sym->cipher.data.length; > + job_len[2] +=3D (ops[i + 2]->sym->cipher.data.length =3D=3D 0) * > + ops[i + 2]->sym->auth.data.length; > + target[2] =3D !(job_len[2] & psd_qp_ctx->threshold); > + if (ops[i + 2]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + ops[i + 2]->sym->session =3D > + sess_ctx[2]->worker_sess[target[2]]; > + p_enq_op =3D &enq_ops[target[2]]; >=20 > if (p_enq_op->pos + in_flight_ops[p_enq_op->worker_idx] =3D=3D > qp_ctx->max_nb_objs) { > @@ -117,10 +145,14 @@ schedule_enqueue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > sched_ops[p_enq_op->worker_idx][p_enq_op->pos] =3D ops[i+2]; > p_enq_op->pos++; >=20 > - job_len =3D ops[i+3]->sym->cipher.data.length; > - job_len +=3D (ops[i+3]->sym->cipher.data.length =3D=3D 0) * > - ops[i+3]->sym->auth.data.length; > - p_enq_op =3D &enq_ops[!(job_len & psd_qp_ctx->threshold)]; > + job_len[3] =3D ops[i + 3]->sym->cipher.data.length; > + job_len[3] +=3D (ops[i + 3]->sym->cipher.data.length =3D=3D 0) * > + ops[i + 3]->sym->auth.data.length; > + target[3] =3D !(job_len[3] & psd_qp_ctx->threshold); > + if (ops[i + 3]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + ops[i + 3]->sym->session =3D > + sess_ctx[1]->worker_sess[target[3]]; > + p_enq_op =3D &enq_ops[target[3]]; >=20 > if (p_enq_op->pos + in_flight_ops[p_enq_op->worker_idx] =3D=3D > qp_ctx->max_nb_objs) { > @@ -133,10 +165,18 @@ schedule_enqueue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > } >=20 > for (; i < nb_ops; i++) { > + struct scheduler_session_ctx *sess_ctx =3D > + (void *)ops[i]->sym->session->driver_priv_data; > + uint32_t job_len; > + uint8_t target; > + > job_len =3D ops[i]->sym->cipher.data.length; > job_len +=3D (ops[i]->sym->cipher.data.length =3D=3D 0) * > ops[i]->sym->auth.data.length; > - p_enq_op =3D &enq_ops[!(job_len & psd_qp_ctx->threshold)]; > + target =3D !(job_len & psd_qp_ctx->threshold); > + if (ops[i]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + ops[i]->sym->session =3D sess_ctx->worker_sess[target]; > + p_enq_op =3D &enq_ops[target]; >=20 > if (p_enq_op->pos + in_flight_ops[p_enq_op->worker_idx] =3D=3D > qp_ctx->max_nb_objs) { > @@ -199,6 +239,7 @@ schedule_dequeue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > if (worker->nb_inflight_cops) { > nb_deq_ops_pri =3D rte_cryptodev_dequeue_burst(worker- > >dev_id, > worker->qp_id, ops, nb_ops); > + scheduler_retrieve_session(ops, nb_deq_ops_pri); > worker->nb_inflight_cops -=3D nb_deq_ops_pri; > } >=20 > @@ -213,6 +254,7 @@ schedule_dequeue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > nb_deq_ops_sec =3D rte_cryptodev_dequeue_burst(worker- > >dev_id, > worker->qp_id, &ops[nb_deq_ops_pri], > nb_ops - nb_deq_ops_pri); > + scheduler_retrieve_session(ops, nb_deq_ops_sec); > worker->nb_inflight_cops -=3D nb_deq_ops_sec; >=20 > if (!worker->nb_inflight_cops) > diff --git a/drivers/crypto/scheduler/scheduler_pmd_ops.c > b/drivers/crypto/scheduler/scheduler_pmd_ops.c > index b93821783b..f8aa10af64 100644 > --- a/drivers/crypto/scheduler/scheduler_pmd_ops.c > +++ b/drivers/crypto/scheduler/scheduler_pmd_ops.c > @@ -9,6 +9,7 @@ > #include > #include > #include > +#include >=20 > #include "scheduler_pmd_private.h" >=20 > @@ -467,19 +468,107 @@ scheduler_pmd_sym_session_get_size(struct > rte_cryptodev *dev __rte_unused) > return max_priv_sess_size; > } >=20 > +struct scheduler_configured_sess_info { > + uint8_t dev_id; > + uint8_t driver_id; > + struct rte_cryptodev_sym_session *sess; > +}; > + > static int > -scheduler_pmd_sym_session_configure(struct rte_cryptodev *dev > __rte_unused, > - struct rte_crypto_sym_xform *xform __rte_unused, > - struct rte_cryptodev_sym_session *sess __rte_unused) > +scheduler_pmd_sym_session_configure(struct rte_cryptodev *dev, > + struct rte_crypto_sym_xform *xform, > + struct rte_cryptodev_sym_session *sess) > { > + struct scheduler_ctx *sched_ctx =3D dev->data->dev_private; > + struct rte_mempool *mp =3D rte_mempool_from_obj(sess); > + struct scheduler_session_ctx *sess_ctx =3D (void *)sess->driver_priv_da= ta; > + struct scheduler_configured_sess_info configured_sess[ > + RTE_CRYPTODEV_SCHEDULER_MAX_NB_WORKERS] =3D > { 0 }; > + uint32_t i, j, n_configured_sess =3D 0; > + int ret =3D 0; > + > + if (mp =3D=3D NULL) > + return -EINVAL; > + > + for (i =3D 0; i < sched_ctx->nb_workers; i++) { > + struct scheduler_worker *worker =3D &sched_ctx->workers[i]; > + struct rte_cryptodev_sym_session *worker_sess; > + > + for (j =3D 0; j < n_configured_sess; j++) { > + if (configured_sess[j].driver_id =3D=3D > + worker->driver_id) { > + sess_ctx->worker_sess[i] =3D > + configured_sess[j].sess; > + goto next_worker; > + } > + } > + > + if (rte_mempool_avail_count(mp) =3D=3D 0) { > + ret =3D -ENOMEM; > + goto error_exit; > + } > + > + worker_sess =3D rte_cryptodev_sym_session_create(worker- > >dev_id, > + xform, mp); > + if (worker_sess =3D=3D NULL) { > + ret =3D -rte_errno; > + goto error_exit; > + } > + > + worker_sess->opaque_data =3D (uint64_t)sess; > + sess_ctx->worker_sess[i] =3D worker_sess; > + configured_sess[n_configured_sess].driver_id =3D > + worker->driver_id; > + configured_sess[n_configured_sess].dev_id =3D worker->dev_id; > + configured_sess[n_configured_sess].sess =3D worker_sess; > + n_configured_sess++; > +next_worker: > + } > + > return 0; > +error_exit: > + sess_ctx->ref_cnt =3D sched_ctx->ref_cnt; > + for (i =3D 0; i < n_configured_sess; i++) > + rte_cryptodev_sym_session_free(configured_sess[i].dev_id, > + configured_sess[i].sess); > + return ret; > } >=20 > /** Clear the memory of session so it doesn't leave key material behind = */ > static void > -scheduler_pmd_sym_session_clear(struct rte_cryptodev *dev __rte_unused, > - struct rte_cryptodev_sym_session *sess __rte_unused) > -{} > +scheduler_pmd_sym_session_clear(struct rte_cryptodev *dev, > + struct rte_cryptodev_sym_session *sess) > +{ > + struct scheduler_ctx *sched_ctx =3D dev->data->dev_private; > + struct scheduler_session_ctx *sess_ctx =3D (void *)sess->driver_priv_da= ta; > + struct scheduler_configured_sess_info deleted_sess[ > + RTE_CRYPTODEV_SCHEDULER_MAX_NB_WORKERS] =3D > { 0 }; > + uint32_t i, j, n_deleted_sess =3D 0; > + > + if (sched_ctx->ref_cnt !=3D sess_ctx->ref_cnt) { > + CR_SCHED_LOG(WARNING, > + "Worker updated between session creation/deletion. " > + "The session may not be freed fully."); > + } > + > + for (i =3D 0; i < sched_ctx->nb_workers; i++) { > + struct scheduler_worker *worker =3D &sched_ctx->workers[i]; > + > + for (j =3D 0; j < n_deleted_sess; j++) { > + if (deleted_sess[j].driver_id =3D=3D worker->driver_id) { > + sess_ctx->worker_sess[i] =3D NULL; > + goto next_worker; > + } > + } > + > + rte_cryptodev_sym_session_free(worker->dev_id, > + sess_ctx->worker_sess[i]); > + > + deleted_sess[n_deleted_sess++].driver_id =3D worker->driver_id; > + sess_ctx->worker_sess[i] =3D NULL; > +next_worker: > + } > +} >=20 > static struct rte_cryptodev_ops scheduler_pmd_ops =3D { > .dev_configure =3D scheduler_pmd_config, > diff --git a/drivers/crypto/scheduler/scheduler_pmd_private.h > b/drivers/crypto/scheduler/scheduler_pmd_private.h > index 4d33b9ab44..0e508727a4 100644 > --- a/drivers/crypto/scheduler/scheduler_pmd_private.h > +++ b/drivers/crypto/scheduler/scheduler_pmd_private.h > @@ -22,7 +22,6 @@ struct scheduler_worker { > uint8_t dev_id; > uint16_t qp_id; > uint32_t nb_inflight_cops; > - > uint8_t driver_id; > }; >=20 > @@ -37,6 +36,8 @@ struct scheduler_ctx { >=20 > struct scheduler_worker > workers[RTE_CRYPTODEV_SCHEDULER_MAX_NB_WORKERS]; > uint32_t nb_workers; > + /* reference count when the workers are incremented/decremented */ > + uint32_t ref_cnt; >=20 > enum rte_cryptodev_scheduler_mode mode; >=20 > @@ -61,6 +62,11 @@ struct scheduler_qp_ctx { > struct rte_ring *order_ring; > } __rte_cache_aligned; >=20 > +struct scheduler_session_ctx { > + uint32_t ref_cnt; > + struct rte_cryptodev_sym_session *worker_sess[ > + RTE_CRYPTODEV_SCHEDULER_MAX_NB_WORKERS]; > +}; >=20 > extern uint8_t cryptodev_scheduler_driver_id; >=20 > @@ -101,6 +107,118 @@ scheduler_order_drain(struct rte_ring *order_ring, > return nb_ops_to_deq; > } >=20 > +static __rte_always_inline void > +scheduler_set_worker_session(struct rte_crypto_op **ops, uint16_t nb_ops= , > + uint8_t worker_index) > +{ > + struct rte_crypto_op **op =3D ops; > + uint16_t n =3D nb_ops; > + > + if (n >=3D 4) { > + rte_prefetch0(op[0]->sym->session); > + rte_prefetch0(op[1]->sym->session); > + rte_prefetch0(op[2]->sym->session); > + rte_prefetch0(op[3]->sym->session); > + } > + > + while (n >=3D 4) { > + if (n >=3D 8) { > + rte_prefetch0(op[4]->sym->session); > + rte_prefetch0(op[5]->sym->session); > + rte_prefetch0(op[6]->sym->session); > + rte_prefetch0(op[7]->sym->session); > + } > + > + if (op[0]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) { > + struct scheduler_session_ctx *sess_ctx =3D > + (void *)op[0]->sym->session->driver_priv_data; > + op[0]->sym->session =3D > + sess_ctx->worker_sess[worker_index]; > + } > + > + if (op[1]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) { > + struct scheduler_session_ctx *sess_ctx =3D > + (void *)op[1]->sym->session->driver_priv_data; > + op[1]->sym->session =3D > + sess_ctx->worker_sess[worker_index]; > + } > + > + if (op[2]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) { > + struct scheduler_session_ctx *sess_ctx =3D > + (void *)op[2]->sym->session->driver_priv_data; > + op[2]->sym->session =3D > + sess_ctx->worker_sess[worker_index]; > + } > + > + if (op[3]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) { > + struct scheduler_session_ctx *sess_ctx =3D > + (void *)op[3]->sym->session->driver_priv_data; > + op[3]->sym->session =3D > + sess_ctx->worker_sess[worker_index]; > + } > + > + op +=3D 4; > + n -=3D 4; > + } > + > + while (n--) { > + if (op[0]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) { > + struct scheduler_session_ctx *sess_ctx =3D > + (void *)op[0]->sym->session->driver_priv_data; > + > + op[0]->sym->session =3D > + sess_ctx->worker_sess[worker_index]; > + op++; > + } > + } > +} > + > +static __rte_always_inline void > +scheduler_retrieve_session(struct rte_crypto_op **ops, uint16_t nb_ops) > +{ > + uint16_t n =3D nb_ops; > + struct rte_crypto_op **op =3D ops; > + > + if (n >=3D 4) { > + rte_prefetch0(op[0]->sym->session); > + rte_prefetch0(op[1]->sym->session); > + rte_prefetch0(op[2]->sym->session); > + rte_prefetch0(op[3]->sym->session); > + } > + > + while (n >=3D 4) { > + if (n >=3D 8) { > + rte_prefetch0(op[4]->sym->session); > + rte_prefetch0(op[5]->sym->session); > + rte_prefetch0(op[6]->sym->session); > + rte_prefetch0(op[7]->sym->session); > + } > + > + if (op[0]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + op[0]->sym->session =3D > + (void *)op[0]->sym->session->opaque_data; > + if (op[1]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + op[1]->sym->session =3D > + (void *)op[1]->sym->session->opaque_data; > + if (op[2]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + op[2]->sym->session =3D > + (void *)op[2]->sym->session->opaque_data; > + if (op[3]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + op[3]->sym->session =3D > + (void *)op[3]->sym->session->opaque_data; > + > + op +=3D 4; > + n -=3D 4; > + } > + > + while (n--) { > + if (op[0]->sess_type =3D=3D RTE_CRYPTO_OP_WITH_SESSION) > + op[0]->sym->session =3D > + (void *)op[0]->sym->session->opaque_data; > + op++; > + } > +} > + > /** device specific operations function pointer structure */ > extern struct rte_cryptodev_ops *rte_crypto_scheduler_pmd_ops; >=20 > diff --git a/drivers/crypto/scheduler/scheduler_roundrobin.c > b/drivers/crypto/scheduler/scheduler_roundrobin.c > index ace2dec2ec..ad3f8b842a 100644 > --- a/drivers/crypto/scheduler/scheduler_roundrobin.c > +++ b/drivers/crypto/scheduler/scheduler_roundrobin.c > @@ -23,16 +23,17 @@ schedule_enqueue(void *qp, struct rte_crypto_op > **ops, uint16_t nb_ops) > ((struct scheduler_qp_ctx *)qp)->private_qp_ctx; > uint32_t worker_idx =3D rr_qp_ctx->last_enq_worker_idx; > struct scheduler_worker *worker =3D &rr_qp_ctx->workers[worker_idx]; > - uint16_t i, processed_ops; > + uint16_t processed_ops; >=20 > if (unlikely(nb_ops =3D=3D 0)) > return 0; >=20 > - for (i =3D 0; i < nb_ops && i < 4; i++) > - rte_prefetch0(ops[i]->sym->session); > - > + scheduler_set_worker_session(ops, nb_ops, worker_idx); > processed_ops =3D rte_cryptodev_enqueue_burst(worker->dev_id, > worker->qp_id, ops, nb_ops); > + if (processed_ops < nb_ops) > + scheduler_retrieve_session(ops + processed_ops, > + nb_ops - processed_ops); >=20 > worker->nb_inflight_cops +=3D processed_ops; >=20 > @@ -86,7 +87,7 @@ schedule_dequeue(void *qp, struct rte_crypto_op **ops, > uint16_t nb_ops) >=20 > nb_deq_ops =3D rte_cryptodev_dequeue_burst(worker->dev_id, > worker->qp_id, ops, nb_ops); > - > + scheduler_retrieve_session(ops, nb_deq_ops); > last_worker_idx +=3D 1; > last_worker_idx %=3D rr_qp_ctx->nb_workers; >=20 > -- > 2.34.1