From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10055.outbound.protection.outlook.com [40.107.1.55]) by dpdk.org (Postfix) with ESMTP id 7B49D1B48C for ; Wed, 9 Jan 2019 16:14:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nxp.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9LRj5CR7uueQP/QoyDZORcQqf9ji2TlaGl35+CeSzkE=; b=ZAJ3o4AukUfDCr0R4qh7RPTvlNL+cd9cvmLQkBFUk19685IiFOtfFqxl+wyXHR8URqPSFJc7OIwC1meZza/hZGsbDlMUK4iG6R5b2xrz77GFKRvh4z5EQf/72Gh6dgUjbcmSIWdy2rhV3ZXmDMk6eKBB4AC7D/x40ptv4Nc22wI= Received: from VI1PR04MB4893.eurprd04.prod.outlook.com (20.177.49.154) by VI1PR04MB4269.eurprd04.prod.outlook.com (52.134.31.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1495.7; Wed, 9 Jan 2019 15:14:18 +0000 Received: from VI1PR04MB4893.eurprd04.prod.outlook.com ([fe80::1c70:8924:814d:d1a]) by VI1PR04MB4893.eurprd04.prod.outlook.com ([fe80::1c70:8924:814d:d1a%3]) with mapi id 15.20.1516.010; Wed, 9 Jan 2019 15:14:18 +0000 From: Akhil Goyal To: "dev@dpdk.org" CC: Hemant Agrawal , "pablo.de.lara.guarch@intel.com" , Akhil Goyal Thread-Topic: [PATCH] crypto/dpaa_sec: support same session flows on multi cores Thread-Index: AQHUqC39FOQwogila0KUTeKbIvAyoQ== Date: Wed, 9 Jan 2019 15:14:17 +0000 Message-ID: <20190109150612.20803-2-akhil.goyal@nxp.com> References: <20190109150612.20803-1-akhil.goyal@nxp.com> In-Reply-To: <20190109150612.20803-1-akhil.goyal@nxp.com> Accept-Language: en-IN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BM1PR01CA0126.INDPRD01.PROD.OUTLOOK.COM (2603:1096:b00:40::20) To VI1PR04MB4893.eurprd04.prod.outlook.com (2603:10a6:803:56::26) x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: git-send-email 2.17.1 x-originating-ip: [14.143.30.134] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VI1PR04MB4269; 6:O6kuDJgtPyPCr/uAWrYr/nLUhhW9jgQ/b2AQOzKCEVUkSqQ2HXVPhjtXMsF9ug8EX/iGm4o8k9rkkQ9zndr4028GVQH54fN5R1e6y39oepvqEHVwyMi98xqDcvG8oq/S4UWlEm7VHIilfjHC5BXW1PGWxBMJtP3um2/sK0l7AqXby90tkK2GLZYdvfperruURPkoaj8qqA6K01RjjoFYjjDVAviXbmeHBPhATvAniO4MLDDyWN2QUECtAxZta/4yPKfKXCYFUBDuEzFJvUwRN90Ds3Pu06aO34mWVNSIaVu2g26R3YxeoNqc3rV1ZEaLWwGHKK8oo6lzXU65GsA9OAwEO95WKyNFNn/H6kZYQiUKLprKqzJ1N1gtAgbStIklW7MdAfI1PbF0eeXSjYJHhEwYBKCBmHDv9F+lyKpFD67lHg0B/mxtiEz8EM7caAi27dHdjxwsk2uK2+/1zlK+mg==; 5:Uf1KlUrl9eKqaFK0+gjZs/4dYiGLicRBJWHZDsweuFNMXCF0Hk4rh3CkYfiBfv8ulPsZOrNCATfwx2ofhXnSZyHwWXNjPvd/OMVHLHreaDBfO0znqg7sNxObNaQW79wg1ZRWw+QyHoDFa1ZmKtxOrtt8ixMz3euj06KbTjDZC9qiM4ylOb3CyOu4ocGG6YGsqykRCYaaLYIzAAw4GI3fQg==; 7:73FeMzPcUOf/k81aGi6djVxULrGXypmb+dLNuKL0gqOYrAk+leRNkGSPQspa5nN0PVqtzRLEZRYV4GZfc4Lpo3vkDuFzuLI8ghdzVSpOSP5Tf/ZOHdURfTSHewG1JO6CmgHhK2uyzsGYikALbZJKiQ== x-ms-office365-filtering-correlation-id: b1da2d4f-ea4d-4966-2542-08d676451fc3 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(7168020)(4627221)(201703031133081)(201702281549075)(8990200)(5600109)(711020)(4618075)(2017052603328)(7153060)(7193020); SRVR:VI1PR04MB4269; x-ms-traffictypediagnostic: VI1PR04MB4269: authentication-results: spf=none (sender IP is ) smtp.mailfrom=akhil.goyal@nxp.com; x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(3230021)(908002)(999002)(5005026)(6040522)(8220060)(2401047)(8121501046)(3002001)(93006095)(93001095)(3231475)(944501520)(52105112)(10201501046)(6055026)(6041310)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123562045)(20161123560045)(201708071742011)(7699051)(76991095); SRVR:VI1PR04MB4269; BCL:0; PCL:0; RULEID:; SRVR:VI1PR04MB4269; x-forefront-prvs: 0912297777 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(376002)(346002)(136003)(396003)(39860400002)(366004)(189003)(199004)(4326008)(25786009)(478600001)(53936002)(1076003)(2501003)(6512007)(6436002)(102836004)(6486002)(105586002)(106356001)(81156014)(81166006)(97736004)(1730700003)(8936002)(55236004)(2351001)(68736007)(186003)(26005)(5024004)(14444005)(6506007)(256004)(386003)(50226002)(305945005)(2906002)(5640700003)(7736002)(5660300001)(71190400001)(476003)(2616005)(71200400001)(8676002)(44832011)(78486014)(6916009)(76176011)(11346002)(486006)(316002)(14454004)(86362001)(3846002)(446003)(36756003)(54906003)(6116002)(52116002)(66066001)(99286004); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR04MB4269; H:VI1PR04MB4893.eurprd04.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: nxp.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: BKgQNBbciG/u8NKdC5n1DCYHcO28jzAqAsvT3cnqR6asxUN4LRgGcSlU7Idq/cdnLP5sHQC5cldVi2h0jQanKM97EE/Nem2SfUM8f1p+q3NmjSad2vA0G3LI/IBSwoIOGFKVVLTTm1H63ykKHPIwUpF4EM1zTwCXBcRrle5kbbL2orxZ0qxFjxNhld6MoLduc13Rh1ayQK2Hw9FDIdk21Zi680KTER2Z2RNkiN+gIWYHJgCNMbw5ImIUOkpD8zDwhpqkqN3rMIS3BjozW1gsn+yPrlQ45B2B/wGdK955vnaqbkwh9rdYnGMObzSaBBPD spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: nxp.com X-MS-Exchange-CrossTenant-Network-Message-Id: b1da2d4f-ea4d-4966-2542-08d676451fc3 X-MS-Exchange-CrossTenant-originalarrivaltime: 09 Jan 2019 15:14:16.8321 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 686ea1d3-bc2b-4c6f-a92c-d99c5c301635 X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR04MB4269 Subject: [dpdk-dev] [PATCH] crypto/dpaa_sec: support same session flows on multi cores X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jan 2019 15:14:19 -0000 In dpaa_sec, each session should have a separate in-queue, and each in-queue should be mapped to an outq. So if multiple flows of same SA comes to different cores due to rss distribution of NIC, enqueue from any core will try to do re-assignment of inq - outq mapping which will fail. In this patch, for each core a separate inq is allocated and used from each of the core. But the number of outq will remain the same and we save the pointer to outq in session for each of the cores. Signed-off-by: Akhil Goyal --- drivers/crypto/dpaa_sec/dpaa_sec.c | 65 +++++++++++++++++++----------- drivers/crypto/dpaa_sec/dpaa_sec.h | 9 +++-- 2 files changed, 46 insertions(+), 28 deletions(-) diff --git a/drivers/crypto/dpaa_sec/dpaa_sec.c b/drivers/crypto/dpaa_sec/d= paa_sec.c index b5896c4f7..6b05ac8bd 100644 --- a/drivers/crypto/dpaa_sec/dpaa_sec.c +++ b/drivers/crypto/dpaa_sec/dpaa_sec.c @@ -1683,15 +1683,18 @@ dpaa_sec_enqueue_burst(void *qp, struct rte_crypto_= op **ops, nb_ops =3D loop; goto send_pkts; } - if (unlikely(!ses->qp)) { + if (unlikely(!ses->qp[rte_lcore_id() % MAX_DPAA_CORES])) { if (dpaa_sec_attach_sess_q(qp, ses)) { frames_to_send =3D loop; nb_ops =3D loop; goto send_pkts; } - } else if (unlikely(ses->qp !=3D qp)) { + } else if (unlikely(ses->qp[rte_lcore_id() % + MAX_DPAA_CORES] !=3D qp)) { DPAA_SEC_DP_ERR("Old:sess->qp =3D %p" - " New qp =3D %p\n", ses->qp, qp); + " New qp =3D %p\n", + ses->qp[rte_lcore_id() % + MAX_DPAA_CORES], qp); frames_to_send =3D loop; nb_ops =3D loop; goto send_pkts; @@ -1743,7 +1746,7 @@ dpaa_sec_enqueue_burst(void *qp, struct rte_crypto_op= **ops, } =20 fd =3D &fds[loop]; - inq[loop] =3D ses->inq; + inq[loop] =3D ses->inq[rte_lcore_id() % MAX_DPAA_CORES]; fd->opaque_addr =3D 0; fd->cmd =3D 0; qm_fd_addr_set64(fd, dpaa_mem_vtop(cf->sg)); @@ -1970,7 +1973,7 @@ dpaa_sec_attach_sess_q(struct dpaa_sec_qp *qp, dpaa_s= ec_session *sess) { int ret; =20 - sess->qp =3D qp; + sess->qp[rte_lcore_id() % MAX_DPAA_CORES] =3D qp; ret =3D dpaa_sec_prep_cdb(sess); if (ret) { DPAA_SEC_ERR("Unable to prepare sec cdb"); @@ -1983,7 +1986,8 @@ dpaa_sec_attach_sess_q(struct dpaa_sec_qp *qp, dpaa_s= ec_session *sess) return ret; } } - ret =3D dpaa_sec_init_rx(sess->inq, dpaa_mem_vtop(&sess->cdb), + ret =3D dpaa_sec_init_rx(sess->inq[rte_lcore_id() % MAX_DPAA_CORES], + dpaa_mem_vtop(&sess->cdb), qman_fq_fqid(&qp->outq)); if (ret) DPAA_SEC_ERR("Unable to init sec queue"); @@ -1997,6 +2001,7 @@ dpaa_sec_set_session_parameters(struct rte_cryptodev = *dev, { struct dpaa_sec_dev_private *internals =3D dev->data->dev_private; dpaa_sec_session *session =3D sess; + uint32_t i; =20 PMD_INIT_FUNC_TRACE(); =20 @@ -2053,12 +2058,15 @@ dpaa_sec_set_session_parameters(struct rte_cryptode= v *dev, } session->ctx_pool =3D internals->ctx_pool; rte_spinlock_lock(&internals->lock); - session->inq =3D dpaa_sec_attach_rxq(internals); - rte_spinlock_unlock(&internals->lock); - if (session->inq =3D=3D NULL) { - DPAA_SEC_ERR("unable to attach sec queue"); - goto err1; + for (i =3D 0; i < MAX_DPAA_CORES; i++) { + session->inq[i] =3D dpaa_sec_attach_rxq(internals); + if (session->inq[i] =3D=3D NULL) { + DPAA_SEC_ERR("unable to attach sec queue"); + rte_spinlock_unlock(&internals->lock); + goto err1; + } } + rte_spinlock_unlock(&internals->lock); =20 return 0; =20 @@ -2118,8 +2126,9 @@ dpaa_sec_sym_session_clear(struct rte_cryptodev *dev, if (sess_priv) { struct rte_mempool *sess_mp =3D rte_mempool_from_obj(sess_priv); =20 - if (s->inq) - dpaa_sec_detach_rxq(qi, s->inq); + if (s->inq[rte_lcore_id() % MAX_DPAA_CORES]) + dpaa_sec_detach_rxq(qi, + s->inq[rte_lcore_id() % MAX_DPAA_CORES]); rte_free(s->cipher_key.data); rte_free(s->auth_key.data); memset(s, 0, sizeof(dpaa_sec_session)); @@ -2138,6 +2147,7 @@ dpaa_sec_set_ipsec_session(__rte_unused struct rte_cr= yptodev *dev, struct rte_crypto_auth_xform *auth_xform =3D NULL; struct rte_crypto_cipher_xform *cipher_xform =3D NULL; dpaa_sec_session *session =3D (dpaa_sec_session *)sess; + uint32_t i; =20 PMD_INIT_FUNC_TRACE(); =20 @@ -2257,12 +2267,15 @@ dpaa_sec_set_ipsec_session(__rte_unused struct rte_= cryptodev *dev, goto out; session->ctx_pool =3D internals->ctx_pool; rte_spinlock_lock(&internals->lock); - session->inq =3D dpaa_sec_attach_rxq(internals); - rte_spinlock_unlock(&internals->lock); - if (session->inq =3D=3D NULL) { - DPAA_SEC_ERR("unable to attach sec queue"); - goto out; + for (i =3D 0; i < MAX_DPAA_CORES; i++) { + session->inq[i] =3D dpaa_sec_attach_rxq(internals); + if (session->inq[i] =3D=3D NULL) { + DPAA_SEC_ERR("unable to attach sec queue"); + rte_spinlock_unlock(&internals->lock); + goto out; + } } + rte_spinlock_unlock(&internals->lock); =20 return 0; out: @@ -2283,6 +2296,7 @@ dpaa_sec_set_pdcp_session(struct rte_cryptodev *dev, struct rte_crypto_cipher_xform *cipher_xform =3D NULL; dpaa_sec_session *session =3D (dpaa_sec_session *)sess; struct dpaa_sec_dev_private *dev_priv =3D dev->data->dev_private; + uint32_t i; =20 PMD_INIT_FUNC_TRACE(); =20 @@ -2364,12 +2378,15 @@ dpaa_sec_set_pdcp_session(struct rte_cryptodev *dev= , =20 session->ctx_pool =3D dev_priv->ctx_pool; rte_spinlock_lock(&dev_priv->lock); - session->inq =3D dpaa_sec_attach_rxq(dev_priv); - rte_spinlock_unlock(&dev_priv->lock); - if (session->inq =3D=3D NULL) { - DPAA_SEC_ERR("unable to attach sec queue"); - goto out; + for (i =3D 0; i < MAX_DPAA_CORES; i++) { + session->inq[i] =3D dpaa_sec_attach_rxq(dev_priv); + if (session->inq[i] =3D=3D NULL) { + DPAA_SEC_ERR("unable to attach sec queue"); + rte_spinlock_unlock(&dev_priv->lock); + goto out; + } } + rte_spinlock_unlock(&dev_priv->lock); return 0; out: rte_free(session->auth_key.data); @@ -2631,7 +2648,7 @@ dpaa_sec_dev_init(struct rte_cryptodev *cryptodev) =20 flags =3D QMAN_FQ_FLAG_LOCKED | QMAN_FQ_FLAG_DYNAMIC_FQID | QMAN_FQ_FLAG_TO_DCPORTAL; - for (i =3D 0; i < internals->max_nb_sessions; i++) { + for (i =3D 0; i < MAX_DPAA_CORES * internals->max_nb_sessions; i++) { /* create rx qman fq for sessions*/ ret =3D qman_create_fq(0, flags, &internals->inq[i]); if (unlikely(ret !=3D 0)) { diff --git a/drivers/crypto/dpaa_sec/dpaa_sec.h b/drivers/crypto/dpaa_sec/d= paa_sec.h index 6049c1d52..75c0960a9 100644 --- a/drivers/crypto/dpaa_sec/dpaa_sec.h +++ b/drivers/crypto/dpaa_sec/dpaa_sec.h @@ -10,6 +10,7 @@ #define CRYPTODEV_NAME_DPAA_SEC_PMD crypto_dpaa_sec /**< NXP DPAA - SEC PMD device name */ =20 +#define MAX_DPAA_CORES 4 #define NUM_POOL_CHANNELS 4 #define DPAA_SEC_BURST 7 #define DPAA_SEC_ALG_UNSUPPORT (-1) @@ -26,7 +27,7 @@ #define CTX_POOL_NUM_BUFS 32000 #define CTX_POOL_BUF_SIZE sizeof(struct dpaa_sec_op_ctx) #define CTX_POOL_CACHE_SIZE 512 -#define RTE_DPAA_SEC_PMD_MAX_NB_SESSIONS 2048 +#define RTE_DPAA_SEC_PMD_MAX_NB_SESSIONS 1024 =20 #define DIR_ENC 1 #define DIR_DEC 0 @@ -142,8 +143,8 @@ typedef struct dpaa_sec_session_entry { }; struct sec_pdcp_ctxt pdcp; }; - struct dpaa_sec_qp *qp; - struct qman_fq *inq; + struct dpaa_sec_qp *qp[MAX_DPAA_CORES]; + struct qman_fq *inq[MAX_DPAA_CORES]; struct sec_cdb cdb; /**< cmd block associated with qp */ struct rte_mempool *ctx_pool; /* session mempool for dpaa_sec_op_ctx */ } dpaa_sec_session; @@ -158,7 +159,7 @@ struct dpaa_sec_qp { }; =20 #define RTE_DPAA_MAX_NB_SEC_QPS 2 -#define RTE_DPAA_MAX_RX_QUEUE RTE_DPAA_SEC_PMD_MAX_NB_SESSIONS +#define RTE_DPAA_MAX_RX_QUEUE (MAX_DPAA_CORES * RTE_DPAA_SEC_PMD_MAX_NB_SE= SSIONS) #define DPAA_MAX_DEQUEUE_NUM_FRAMES 63 =20 /* internal sec queue interface */ --=20 2.17.1