From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id CC252A04DD; Tue, 26 Nov 2019 16:07:05 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 09A843195; Tue, 26 Nov 2019 16:07:05 +0100 (CET) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30075.outbound.protection.outlook.com [40.107.3.75]) by dpdk.org (Postfix) with ESMTP id 525BDCF3 for ; Tue, 26 Nov 2019 16:07:04 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EFjTiIGV3z+C+esT+Cp2gSjeZ1ELO1n1OWdbjz885fdCw3KmCKMq3/6/rUkrPH0vctHumIuozvB2RS06D0iie74mzY0elbiLpRPM4wlkfLUjMqTRNz01N4iIglsl7qoRGsv0zi44+XqXThC38vLn2xhoz5caAjKpClCyZaHM37AboSaYJ7X29g0ZAlZdp+mFvssrxYUYSSIb7pq/LqAtzRxIjo3KhYvTBVWuQ/dOT3az/7A3yiUt8LIQ7bpm2C549Ix4ENMivKAqPd23yuKxdHh0cAgRWyuiSRM9tjoeNRfbXizkhKORGI26YhLkkjBHo29XqBwNtpixIuGMmxWQHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oGe95lRWCzq5od5uNZum1IGORW6aD8gBqfnSEYRrFsc=; b=REXg4gLVmrCPdE48BazToMw++aNgh93neqBBu5SkMswgJZV7o8Ce/ho4yShnNKQPbyIWZULVKOfMr20LDxC8QWKSA65xAN/5qg4ovXaH7EX8J56D495SpDlvhKZAzxF4LQaaIw7qPmRbXLoJEVlEk/37a6Yp+N/iITBu9oP4I+kavw/mUHu7azarttoT16ktTAmISUte5IwdoJ8FD+x4TUCEO8i2jmbn0dLsBOrpTYLUR364UFxfwZIp+wEuGRI+LM+PwlJ3w3o/AD3jV7Ikk34YajoLtMJrX9s4wPni4xL/9Ld9+irkcnR1w3rNxz2BdHV6oIh0nB5YECmWNbxhhA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=oGe95lRWCzq5od5uNZum1IGORW6aD8gBqfnSEYRrFsc=; b=BQfChWXRL30OAjMeugM0HYhtthTlNz/SLhFyv4trs2LrNu3KYz0ntlbaXU5seIX+9qhuNDUtZlsviIa/mX5YZuCH4lSD9lY7zofawDHlmxvxcJh9xa4yTcq8I5JrLpgu5L0vfBBg2PRK8MTnMEXIgBLb8pKf88uSEDVe9ZGU/VQ= Received: from DB3PR0502MB3964.eurprd05.prod.outlook.com (52.134.65.161) by DB3PR0502MB4090.eurprd05.prod.outlook.com (52.134.72.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2474.19; Tue, 26 Nov 2019 15:07:03 +0000 Received: from DB3PR0502MB3964.eurprd05.prod.outlook.com ([fe80::a850:bac9:c90f:f2f5]) by DB3PR0502MB3964.eurprd05.prod.outlook.com ([fe80::a850:bac9:c90f:f2f5%6]) with mapi id 15.20.2474.023; Tue, 26 Nov 2019 15:07:03 +0000 From: Raslan Darawsheh To: Slava Ovsiienko , "dev@dpdk.org" CC: Matan Azrad , Ori Kam Thread-Topic: [PATCH] net/mlx5: fix legacy multi-packet Tx descriptors Thread-Index: AQHVpEadtOG1nv3k6Uqo2WKt+KmsLaedjbDA Date: Tue, 26 Nov 2019 15:07:03 +0000 Message-ID: References: <1574765103-24499-1-git-send-email-viacheslavo@mellanox.com> In-Reply-To: <1574765103-24499-1-git-send-email-viacheslavo@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=rasland@mellanox.com; x-originating-ip: [212.29.221.74] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 651d56e1-ea20-4434-0411-08d772824b80 x-ms-traffictypediagnostic: DB3PR0502MB4090:|DB3PR0502MB4090: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0233768B38 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(346002)(136003)(366004)(39860400002)(396003)(376002)(13464003)(189003)(199004)(52536014)(7696005)(66556008)(66476007)(64756008)(4326008)(2501003)(71190400001)(66446008)(66946007)(229853002)(9686003)(8676002)(5660300002)(25786009)(66066001)(86362001)(2906002)(55016002)(11346002)(446003)(8936002)(99286004)(76116006)(7736002)(3846002)(6116002)(74316002)(305945005)(81156014)(54906003)(33656002)(110136005)(107886003)(6246003)(256004)(14444005)(6506007)(30864003)(26005)(478600001)(102836004)(53546011)(186003)(81166006)(316002)(6436002)(76176011)(71200400001)(14454004); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB4090; H:DB3PR0502MB3964.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 4RnSAYVFHEgjxYWKoAAWUOE9V3bKzphEvHMA+DO2w7w1ZU78KwOMLpZPi8bk4tDNPIU+XgT1T90tqTKggLQ2N+RUh57CXO9qKVqohnF40e1fYWpYOGtW4ifAFOrQHYZ8h7OGKab5bTJ2XeJ3HNrJEN2pmXfvB76qzyTvjRA3njTuaqNyQYRyuSYMr+P9cbrhdO7jl47EHCzbLOqckjkMcviV8ahzOBHkk2qty3T3pF5+33odF9RDofajsTuz9ZY9THpTKGzS1vsl7QIFsh9p+SkMS6qM0emUzv2QfBEmNGRCcdXaTe4L1oLJtzPHrnWVtUMxGxKga0pUg/u9lgzQ4XK8BsjS9Tw/7KEAd3xvKrOkfpoqjb1YImetn9jdD4c0JncLATcHtxP8gktSpjdvA6l8IrQ1M2CiH4yBiUn9RTPsfgE4X6QLkI3pTGWBbMpi Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 651d56e1-ea20-4434-0411-08d772824b80 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Nov 2019 15:07:03.3220 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: sR4bQi7W+z8934pyVbcQSvsLFD9cXlhmKCBHjrcRTw/881Ee+6/zxBzrz7O8odetVo0+KBbkNPS4a5/g3NNMiw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB4090 Subject: Re: [dpdk-dev] [PATCH] net/mlx5: fix legacy multi-packet Tx descriptors X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, > -----Original Message----- > From: Viacheslav Ovsiienko > Sent: Tuesday, November 26, 2019 12:45 PM > To: dev@dpdk.org > Cc: Matan Azrad ; Raslan Darawsheh > ; Ori Kam > Subject: [PATCH] net/mlx5: fix legacy multi-packet Tx descriptors >=20 > ConnectX-4LX supports multiple packets within the single Tx > descriptor. This feature is named as "Legacy Multi-Packet Write" > and imposes a lot of limitations: >=20 > - no ACLs, it means no NIC Tx Flows are supported and Tx metadata > become meaningless > - the required minimal inline data must be zero > - no SR-IOV, it means no support in E-Switch configurations, > - no priority and dscp forcing > - no VLAN insertion > - no TSO > - all packets within MPW session must have the same size >=20 > This legacy MPW feature is mainly intended for test purposes. > To explicitly engage the feature on ConnectX-4LX the devargs > should be specified: >=20 > - txq_mpw_en=3D1 >=20 > This feature was dropped in 19.08, this patch reverts it back. >=20 > Fixes: 18a1c20044c0 ("net/mlx5: implement Tx burst template") >=20 > Signed-off-by: Viacheslav Ovsiienko > Acked-by: Matan Azrad > --- > drivers/net/mlx5/mlx5.c | 3 +- > drivers/net/mlx5/mlx5_rxtx.c | 98 > ++++++++++++++++++++++++++++++++++++++------ > drivers/net/mlx5/mlx5_txq.c | 2 +- > 3 files changed, 89 insertions(+), 14 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c > index 59ae408..487c2be 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -2479,7 +2479,8 @@ struct mlx5_flow_id_pool * > else > config.mps =3D config.mps ? mps : MLX5_MPW_DISABLED; > DRV_LOG(INFO, "%sMPS is %s", > - config.mps =3D=3D MLX5_MPW_ENHANCED ? "enhanced " : "", > + config.mps =3D=3D MLX5_MPW_ENHANCED ? "enhanced " : > + config.mps =3D=3D MLX5_MPW ? "legacy " : "", > config.mps !=3D MLX5_MPW_DISABLED ? "enabled" : > "disabled"); > if (config.cqe_comp && !cqe_comp) { > DRV_LOG(WARNING, "Rx CQE compression isn't > supported"); > diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c > index 3762002..73c16de 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.c > +++ b/drivers/net/mlx5/mlx5_rxtx.c > @@ -62,6 +62,7 @@ enum mlx5_txcmp_code { > #define MLX5_TXOFF_CONFIG_VLAN (1u << 5) /* VLAN insertion > supported.*/ > #define MLX5_TXOFF_CONFIG_METADATA (1u << 6) /* Flow metadata. */ > #define MLX5_TXOFF_CONFIG_EMPW (1u << 8) /* Enhanced MPW > supported.*/ > +#define MLX5_TXOFF_CONFIG_MPW (1u << 9) /* Legacy MPW > supported.*/ >=20 > /* The most common offloads groups. */ > #define MLX5_TXOFF_CONFIG_NONE 0 > @@ -2240,6 +2241,9 @@ enum mlx5_txcmp_code { > { > struct mlx5_wqe_cseg *restrict cs =3D &wqe->cseg; >=20 > + /* For legacy MPW replace the EMPW by TSO with modifier. */ > + if (MLX5_TXOFF_CONFIG(MPW) && opcode =3D=3D > MLX5_OPCODE_ENHANCED_MPSW) > + opcode =3D MLX5_OPCODE_TSO | MLX5_OPC_MOD_MPW << > 24; > cs->opcode =3D rte_cpu_to_be_32((txq->wqe_ci << 8) | opcode); > cs->sq_ds =3D rte_cpu_to_be_32(txq->qp_num_8s | ds); > cs->flags =3D RTE_BE32(MLX5_COMP_ONLY_FIRST_ERR << > @@ -3669,6 +3673,7 @@ enum mlx5_txcmp_code { >=20 > /** > * Check the next packet attributes to match with the eMPW batch ones. > + * In addition, for legacy MPW the packet length is checked either. > * > * @param txq > * Pointer to TX queue structure. > @@ -3676,6 +3681,8 @@ enum mlx5_txcmp_code { > * Pointer to Ethernet Segment of eMPW batch. > * @param loc > * Pointer to burst routine local context. > + * @param dlen > + * Length of previous packet in MPW descriptor. > * @param olx > * Configured Tx offloads mask. It is fully defined at > * compile time and may be used for optimization. > @@ -3688,6 +3695,7 @@ enum mlx5_txcmp_code { > mlx5_tx_match_empw(struct mlx5_txq_data *restrict txq __rte_unused, > struct mlx5_wqe_eseg *restrict es, > struct mlx5_txq_local *restrict loc, > + uint32_t dlen, > unsigned int olx) > { > uint8_t swp_flags =3D 0; > @@ -3706,6 +3714,10 @@ enum mlx5_txcmp_code { > es->metadata !=3D (loc->mbuf->ol_flags & > PKT_TX_DYNF_METADATA ? > *RTE_FLOW_DYNF_METADATA(loc->mbuf) : > 0)) > return false; > + /* Legacy MPW can send packets with the same lengt only. */ > + if (MLX5_TXOFF_CONFIG(MPW) && > + dlen !=3D rte_pktmbuf_data_len(loc->mbuf)) > + return false; > /* There must be no VLAN packets in eMPW loop. */ > if (MLX5_TXOFF_CONFIG(VLAN)) > assert(!(loc->mbuf->ol_flags & PKT_TX_VLAN_PKT)); > @@ -3906,6 +3918,10 @@ enum mlx5_txcmp_code { > eseg =3D &loc->wqe_last->eseg; > dseg =3D &loc->wqe_last->dseg[0]; > loop =3D part; > + /* Store the packet length for legacy MPW. */ > + if (MLX5_TXOFF_CONFIG(MPW)) > + eseg->mss =3D rte_cpu_to_be_16 > + (rte_pktmbuf_data_len(loc->mbuf)); > for (;;) { > uint32_t dlen =3D rte_pktmbuf_data_len(loc->mbuf); > #ifdef MLX5_PMD_SOFT_COUNTERS > @@ -3964,8 +3980,9 @@ enum mlx5_txcmp_code { > * - check sum settings > * - metadata value > * - software parser settings > + * - packets length (legacy MPW only) > */ > - if (!mlx5_tx_match_empw(txq, eseg, loc, olx)) { > + if (!mlx5_tx_match_empw(txq, eseg, loc, dlen, olx)) { > assert(loop); > part -=3D loop; > mlx5_tx_sdone_empw(txq, loc, part, slen, > olx); > @@ -4059,6 +4076,10 @@ enum mlx5_txcmp_code { > olx & ~MLX5_TXOFF_CONFIG_VLAN); > eseg =3D &loc->wqe_last->eseg; > dseg =3D &loc->wqe_last->dseg[0]; > + /* Store the packet length for legacy MPW. */ > + if (MLX5_TXOFF_CONFIG(MPW)) > + eseg->mss =3D rte_cpu_to_be_16 > + (rte_pktmbuf_data_len(loc->mbuf)); > room =3D RTE_MIN(MLX5_WQE_SIZE_MAX / MLX5_WQE_SIZE, > loc->wqe_free) * MLX5_WQE_SIZE - > MLX5_WQE_CSEG_SIZE - > @@ -4209,8 +4230,9 @@ enum mlx5_txcmp_code { > * - check sum settings > * - metadata value > * - software parser settings > + * - packets length (legacy MPW only) > */ > - if (!mlx5_tx_match_empw(txq, eseg, loc, olx)) > + if (!mlx5_tx_match_empw(txq, eseg, loc, dlen, olx)) > break; > /* Packet attributes match, continue the same > eMPW. */ > if ((uintptr_t)dseg >=3D (uintptr_t)txq->wqes_end) > @@ -4313,8 +4335,9 @@ enum mlx5_txcmp_code { > * free the packet immediately. > */ > rte_pktmbuf_free_seg(loc->mbuf); > - } else if (!MLX5_TXOFF_CONFIG(EMPW) && > - txq->inlen_mode) { > + } else if ((!MLX5_TXOFF_CONFIG(EMPW) || > + MLX5_TXOFF_CONFIG(MPW)) && > + txq->inlen_mode) { > /* > * If minimal inlining is requested the eMPW > * feature should be disabled due to data is > @@ -4949,6 +4972,30 @@ enum mlx5_txcmp_code { > MLX5_TXOFF_CONFIG_METADATA) >=20 > /* > + * Generate routines with Legacy Multi-Packet Write support. > + * This mode is supported by ConnectX-4LX only and imposes > + * offload limitations, not supported: > + * - ACL/Flows (metadata are becoming meaningless) > + * - WQE Inline headers > + * - SRIOV (E-Switch offloads) > + * - VLAN insertion > + * - tunnel encapsulation/decapsulation > + * - TSO > + */ > +MLX5_TXOFF_DECL(none_mpw, > + MLX5_TXOFF_CONFIG_NONE | > MLX5_TXOFF_CONFIG_EMPW | > + MLX5_TXOFF_CONFIG_MPW) > + > +MLX5_TXOFF_DECL(mci_mpw, > + MLX5_TXOFF_CONFIG_MULTI | > MLX5_TXOFF_CONFIG_CSUM | > + MLX5_TXOFF_CONFIG_INLINE | > MLX5_TXOFF_CONFIG_EMPW | > + MLX5_TXOFF_CONFIG_MPW) > + > +MLX5_TXOFF_DECL(i_mpw, > + MLX5_TXOFF_CONFIG_INLINE | > MLX5_TXOFF_CONFIG_EMPW | > + MLX5_TXOFF_CONFIG_MPW) > + > +/* > * Array of declared and compiled Tx burst function and corresponding > * supported offloads set. The array is used to select the Tx burst > * function for specified offloads set at Tx queue configuration time. > @@ -5050,7 +5097,6 @@ enum mlx5_txcmp_code { > MLX5_TXOFF_CONFIG_INLINE | > MLX5_TXOFF_CONFIG_METADATA) >=20 > - > MLX5_TXOFF_INFO(mtv, > MLX5_TXOFF_CONFIG_MULTI | MLX5_TXOFF_CONFIG_TSO > | > MLX5_TXOFF_CONFIG_VLAN | > @@ -5091,6 +5137,19 @@ enum mlx5_txcmp_code { > MLX5_TXOFF_INFO(iv, > MLX5_TXOFF_CONFIG_INLINE | > MLX5_TXOFF_CONFIG_VLAN | > MLX5_TXOFF_CONFIG_METADATA) > + > +MLX5_TXOFF_INFO(none_mpw, > + MLX5_TXOFF_CONFIG_NONE | > MLX5_TXOFF_CONFIG_EMPW | > + MLX5_TXOFF_CONFIG_MPW) > + > +MLX5_TXOFF_INFO(mci_mpw, > + MLX5_TXOFF_CONFIG_MULTI | > MLX5_TXOFF_CONFIG_CSUM | > + MLX5_TXOFF_CONFIG_INLINE | > MLX5_TXOFF_CONFIG_EMPW | > + MLX5_TXOFF_CONFIG_MPW) > + > +MLX5_TXOFF_INFO(i_mpw, > + MLX5_TXOFF_CONFIG_INLINE | > MLX5_TXOFF_CONFIG_EMPW | > + MLX5_TXOFF_CONFIG_MPW) > }; >=20 > /** > @@ -5173,11 +5232,8 @@ enum mlx5_txcmp_code { > if (config->mps =3D=3D MLX5_MPW_ENHANCED && > config->txq_inline_min <=3D 0) { > /* > - * The NIC supports Enhanced Multi-Packet Write. > - * We do not support legacy MPW due to its > - * hardware related problems, so we just ignore > - * legacy MLX5_MPW settings. There should be no > - * minimal required inline data. > + * The NIC supports Enhanced Multi-Packet Write > + * and does not require minimal inline data. > */ > olx |=3D MLX5_TXOFF_CONFIG_EMPW; > } > @@ -5185,6 +5241,20 @@ enum mlx5_txcmp_code { > /* We should support Flow metadata. */ > olx |=3D MLX5_TXOFF_CONFIG_METADATA; > } > + if (config->mps =3D=3D MLX5_MPW) { > + /* > + * The NIC supports Legacy Multi-Packet Write. > + * The MLX5_TXOFF_CONFIG_MPW controls the > + * descriptor building method in combination > + * with MLX5_TXOFF_CONFIG_EMPW. > + */ > + if (!(olx & (MLX5_TXOFF_CONFIG_TSO | > + MLX5_TXOFF_CONFIG_SWP | > + MLX5_TXOFF_CONFIG_VLAN | > + MLX5_TXOFF_CONFIG_METADATA))) > + olx |=3D MLX5_TXOFF_CONFIG_EMPW | > + MLX5_TXOFF_CONFIG_MPW; > + } > /* > * Scan the routines table to find the minimal > * satisfying routine with requested offloads. > @@ -5253,7 +5323,11 @@ enum mlx5_txcmp_code { > DRV_LOG(DEBUG, "\tVLANI (VLAN insertion)"); > if (txoff_func[m].olx & MLX5_TXOFF_CONFIG_METADATA) > DRV_LOG(DEBUG, "\tMETAD (tx Flow metadata)"); > - if (txoff_func[m].olx & MLX5_TXOFF_CONFIG_EMPW) > - DRV_LOG(DEBUG, "\tEMPW (Enhanced MPW)"); > + if (txoff_func[m].olx & MLX5_TXOFF_CONFIG_EMPW) { > + if (txoff_func[m].olx & MLX5_TXOFF_CONFIG_MPW) > + DRV_LOG(DEBUG, "\tMPW (Legacy MPW)"); > + else > + DRV_LOG(DEBUG, "\tEMPW (Enhanced MPW)"); > + } > return txoff_func[m].func; > } > diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c > index 8160516..5e45748 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -957,7 +957,7 @@ struct mlx5_txq_obj * > (unsigned int)config->txq_inline_mpw; > inlen_mode =3D (config->txq_inline_min =3D=3D MLX5_ARG_UNSET) ? > 0 : (unsigned int)config->txq_inline_min; > - if (config->mps !=3D MLX5_MPW_ENHANCED) > + if (config->mps !=3D MLX5_MPW_ENHANCED && config->mps !=3D > MLX5_MPW) > inlen_empw =3D 0; > /* > * If there is requested minimal amount of data to inline > -- > 1.8.3.1 Patch applied to next-net-mlx, Kindest regards, Raslan Darawsheh