From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <matan@mellanox.com>
Received: from EUR03-AM5-obe.outbound.protection.outlook.com
(mail-eopbgr30086.outbound.protection.outlook.com [40.107.3.86])
by dpdk.org (Postfix) with ESMTP id 7D9C11B666
for <dev@dpdk.org>; Thu, 2 Nov 2017 17:43:22 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version;
bh=3/qYNs/PvXfJR5/iuRTOSwCvDDROa1ja3T1+bB25vLM=;
b=a16xjp2BJKQWHsaCuttgziOqdI6q/TJfLRgDSf/q/wxUMi80aFS8snDUltl/7s4bUAyaaoS2BJDq/XMdsM9j8HxhCTomXIW1zVOZW2KIH1eNmB7OpcuBsDpWvmVm1yDQ1YC5P0cYiryWmNgdNZf5gFE2GUkzr5+hs+qPoYPBIPQ=
Authentication-Results: spf=none (sender IP is )
smtp.mailfrom=matan@mellanox.com;
Received: from mellanox.com (37.142.13.130) by
HE1PR0502MB3659.eurprd05.prod.outlook.com (2603:10a6:7:85::17) with Microsoft
SMTP Server (version=TLS1_2,
cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.197.13; Thu, 2
Nov 2017 16:43:20 +0000
From: Matan Azrad <matan@mellanox.com>
To: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Cc: dev@dpdk.org,
Ophir Munk <ophirmu@mellanox.com>
Date: Thu, 2 Nov 2017 16:42:49 +0000
Message-Id: <1509640971-8637-7-git-send-email-matan@mellanox.com>
X-Mailer: git-send-email 1.8.3.1
In-Reply-To: <1509640971-8637-1-git-send-email-matan@mellanox.com>
References: <1509358049-18854-1-git-send-email-matan@mellanox.com>
<1509640971-8637-1-git-send-email-matan@mellanox.com>
MIME-Version: 1.0
Content-Type: text/plain
X-Originating-IP: [37.142.13.130]
X-ClientProxiedBy: VI1PR0801CA0068.eurprd08.prod.outlook.com
(2603:10a6:800:7d::12) To HE1PR0502MB3659.eurprd05.prod.outlook.com
(2603:10a6:7:85::17)
X-MS-PublicTrafficType: Email
X-MS-Office365-Filtering-Correlation-Id: 9263e41b-e686-4cae-2859-08d52210d3ce
X-MS-Office365-Filtering-HT: Tenant
X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0;
RULEID:(22001)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(48565401081)(2017052603199);
SRVR:HE1PR0502MB3659;
X-Microsoft-Exchange-Diagnostics: 1; HE1PR0502MB3659;
3:IILSMOwjTJzrWd3CyKvNlCKyDyuYCTv8XYk+ngfMeVXPiWfirPKtxI9okEzY/vP69oTOgsHDY1wjnREMsc83eidkgPnuFHkDSgtIQXCQyjV4f/CWvp7PGrFc9t9Mp3fsIW9Dw6/EZS929NfNWt87N64YiTC/8Ea8Lihvm+IhWTDhu+pY57xhA125tL5IqGEVSnC6X1WytRIAQVMOtRvjfGiRrylixx8LoqjGgWDKqIHUkGYHScliADPDRFrFm/25;
25:/rf9FQfheuoNX4ph+5sMloQuwP+OeYlDnslTlaeS06IdfSsUkOzt3h+fb/9L1pWJ9+g5SqwRKWNSaQzHmhiLzaJU48XW+v4HDnmxa/a0TLNVZRmDYHB9wexIWO/nSat0XF/NCN0m4eEmEYqNVE/TFZ7RIzezg9eGbbxo+93/naL6JC62Tjh1XU1tg9i1B3qJdPDht4d5Y9PaOde7OfbceVRSDsCh2LgdSPi8vLZr/xRg9/JF+/P/m/cTG4ntRySE9OOI+HH7Hb757raf1M9aHmYzJ+Nabn9A8oL7gZs4NKrCH+MKExrIE9GW3ox8vKqE2KSr/J3zjAd88deXLbIVwA==;
31:8nxX2auMPO8TlKIFhHuiiD/xlTW4GUT3cgqlNZCLVJYBgERmGlMpfkOpcYt2db1rE/+Fp6j/Eoaok6yHBmRzgFjvt6l15EyrVgMZM2psmQ9M+ktcNg+MXBjXXdMDtaStCx56oUo8c3W8ve//a9pfHX9AgSycd+SryzgozDy4H5UErkjeePttaEUQg8nanpea9UYtad7InKJGORgs1t4hl4Ksi+cCcSqkhhm89ccTfaQ=
X-MS-TrafficTypeDiagnostic: HE1PR0502MB3659:
X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr
X-Microsoft-Exchange-Diagnostics: 1; HE1PR0502MB3659;
20:KsWDWcbe1vKkUpMjy3izreawXAGkZWl5XZH9TIOYlCrF0NQafyXk7UgC75VlO+221Lxxu/ZQ8CyfyKUn6NkP7F4stfTF+RbsIrURtMNUpr/6ZbFmVg3ZqLtDvvVVMCl7pA39seIT9ojmYxSU4d+QiG4ogI5md22d3DtyiYlyTaUEEyLwPRi4TfAJ5mTwBPfLTzUARzjrq2xRGSX6P/I4I4y2DvkOKuTh6W8Cgqvaa0C+Do19OFeyoa36blbOQj+f+7CZBYUCV5HvBOGyajauIhZxoGi6IqfsnrFpZhRo9KPtCN+QuhlXrRiOV3sJ+7474Id/UG5UgLVRU/ql+Zg87bAEkGM09gQ+hxbJN4kZzcgkNeeRchF6GbY8kJ3nutB2iV384eNnM9nnIQhngyUsKwO269eyr+LJuS+6y20Sen83nyqQ3ChXAbE8wmnneeGkxJG559SmEgeFmLwDVXdApXS3bd8b2XoaEasXmKtx2ZwBG2Qra1E2T2bdxmyW7HGt;
4:LocxKhoiO0vEVvJ0HFMHSb4zN3u3EGVSIlUM1XAHMkBfrtlCit5Du8s5jRlD/NFQ7U2xqeBGrc6XwgfzEM6vyx+YcJ0QuOCeDdPXXJVTFgyQ0DYGa7AVF/EMJSTmlW0a0HJjY7hSO/jcigk03C3q7sVPlEZj++Ywup93hoX1Jn7XLsvTqkVvVzQrXFv2EYaQo1x3YjmLoaGM0W4FbzLi5fPvqcvXEospayF0s9GrymeFYDZTrCSdGsQp8pI7xjyDt3moYCRnLbohAP4yMWYS1X0lGlifyl1G95nTNncxB1kaM1TgXooyZ7wNnfbKFXaN
X-Exchange-Antispam-Report-Test: UriScan:(60795455431006);
X-Microsoft-Antispam-PRVS: <HE1PR0502MB36596CB44B2099E826B7AE1CD25C0@HE1PR0502MB3659.eurprd05.prod.outlook.com>
X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0;
RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(10201501046)(3231020)(93006095)(93001095)(3002001)(100000703101)(100105400095)(6055026)(6041248)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123555025)(20161123564025)(20161123562025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);
SRVR:HE1PR0502MB3659; BCL:0; PCL:0;
RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);
SRVR:HE1PR0502MB3659;
X-Forefront-PRVS: 047999FF16
X-Forefront-Antispam-Report: SFV:NSPM;
SFS:(10009020)(6009001)(376002)(346002)(39860400002)(189002)(199003)(16586007)(50226002)(316002)(106356001)(105586002)(101416001)(21086003)(25786009)(8936002)(478600001)(4720700003)(8676002)(81156014)(189998001)(36756003)(33026002)(81166006)(97736004)(4326008)(5003940100001)(2906002)(55016002)(68736007)(33646002)(5660300001)(305945005)(107886003)(7736002)(50466002)(53936002)(48376002)(66066001)(16526018)(2950100002)(6916009)(6666003)(50986999)(47776003)(76176999)(3846002)(86362001)(69596002)(6116002);
DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0502MB3659; H:mellanox.com; FPR:; SPF:None;
PTR:InfoNoRecords; MX:1; A:1; LANG:en;
Received-SPF: None (protection.outlook.com: mellanox.com does not designate
permitted sender hosts)
X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; HE1PR0502MB3659;
23:zS+GPfPuuJDFxO5nLjjQNbus8n8UX4yyt8YmJyu?=
=?us-ascii?Q?4EytQmbaVsdfe9qyxev87zzbW0A27OT53F4mtdXp+sEc2LBYDqtIcsaEnVqa?=
=?us-ascii?Q?sk4ZEG95LZII4LAL6Ce2ikvySP5m8t0w/6gHJVcEOCpV9zy1G33W2wSBikki?=
=?us-ascii?Q?9sWmMuEE0rMkQ1AdzmeaKW6kAhbuBKgCp18iMISVAFUvmuPMlwMPXbcqf7UV?=
=?us-ascii?Q?fF5+qFVZMwLCQ4bngJDcLuNEv+u0xEwCgTpCxJS5BTCDKeWjym25hh7lRymq?=
=?us-ascii?Q?EHF9DeaC3pWOt+jLAI58JAYsOqoOv5sI75c0gZm/H3qE72vKEK9b2RDx+1Qa?=
=?us-ascii?Q?nCKBGWlQ4+R2BvB8kZCH903mvL+qngzvk7EbRVM1feTWUgP7P61kSftbpDB1?=
=?us-ascii?Q?x/hOjsI90LodYLjIsmDysjI/D1+pVc/aksRMOWtePwKPgWKrik3Dm0zjeO7E?=
=?us-ascii?Q?sKdPqqXAbHow/1cWfgIzlpQH6oQxOiXtcPDShMNR65KGhcScJ38P1Ggy4b/H?=
=?us-ascii?Q?hA4PjPH9FbIWENGtNjmD9LKKIuxzNJtOTJg73LKGAV0v9kvVLIZisr3z9m+3?=
=?us-ascii?Q?CjIbW+h/UwguptKI+gQ/ETGCEpetY+rA5qJjtdl4xSYtdI+nowYJJ+JtQDx8?=
=?us-ascii?Q?+Dp+BJwWrwa4w+ZhqgcinurQZx8EB9Or5WwyAFdlBh1Etnqm7jvapbuBV8BN?=
=?us-ascii?Q?ID5uE3yf6F/6em9D/Us/7mBEUuxnopQKlTI/CV3YBP4wdzQFNlEJxcSEO2zF?=
=?us-ascii?Q?ddvJrIFjjHSGY0Q6v9Dxi/iFQleiyEY1qoO7BPvjHL1nD5+sx7T0ptqZZXBq?=
=?us-ascii?Q?JYuCbfFtJQLmQmda32T9dE4gdUcAlsF7FU81FXqNLSJwNrILZ3i8/r0ksWhA?=
=?us-ascii?Q?i0ZZ3oBy86qxKs/SK5nbrZsiyIPUfGQTMPA3mPJoHFfXMY9PHgDbzHjbHa7z?=
=?us-ascii?Q?OFUwPj6j8s5639Tef4RePsNU84DtgNWewIKQA0ZH73OeCi52FJaPPp6a0mJA?=
=?us-ascii?Q?zKBOr30qRjF5MnE8f88jo6aRrVRlDCUd0CXdjhbpuWTXTE56P4ZvcVsgLmcZ?=
=?us-ascii?Q?vdqtkEIO/omNSpCWXCs/nqzpowVBXtJrz0hxCEJCwbshJvbpchBcmF0wZ7IB?=
=?us-ascii?Q?uaOQJIVKmP0mhJcAcya5kGwYBUSjDLchmjP8sbg+7YbREICNOjMES7A=3D?=
=?us-ascii?Q?=3D?=
X-Microsoft-Exchange-Diagnostics: 1; HE1PR0502MB3659;
6:hW19gxiujoj2pVmr4+vuO8MSdDKUlss1VsMbRb9+LdnN/YQNhgJY1X8/FaKRE5l6eS51ymY+4Wt7aZaQUvsUQ9WgfUFXKDC0TM6rBpGyTlT3DQyeGr3f/5EVUC5CfSnfzVcR0Arqcgvwf4afXZaRLSssuHKPLeN0h5YHTLuCkd4ckofUPXUjTh3wlrx5V/f2I5BbkEwBUuvEXWcO+rELbqA/cEuq2cowc3DHIC6omUwNrIatSXWy/nS0cFUmBEyEhO5n/Ch3Gt7yx2J8pjQWviz2Cc7n7ZeivWHBVOXJhwAs2w+fHOFoCXDXhRBhSZYGRUjlD8ZVKq5F9+CG6U3oAcunNu3yJmx75V/8cKrddiU=;
5:JBQN7dQp9cagZbviRfIAp2lEkFwYwcZY37FL1Q0674fC+++S2bhq8kApRwizXSNJ2Xkk68CJnNhRi3onW4zy7nbMC9Ln9yp69idYnD72IUvZd0EJ6qpXyAxj5M5PdXMgISLhzYobufgEB07Qululc9Dyi9K8NDhGy0ImmuBpMH4=;
24:pWXYqAsgRHzchM0lvYl0sPBd/6SN/Y4yDnXBiSPIY0l0avw9Wubfr3fug9S4rhm9eDgJd+/wUUeF/Eg/IX4wTfc4TKQk25HLCw4eZe2hhM4=;
7:fZoIlM/v3f4Vvejw58yA7QkwrI6zu/Lo+0Qu3ucsdkVqmmM2lXFTbql1CH/sghQ64D7Z4MIhR9GyFXEwSBf5FHiWImsx+CAessqIPErap2bvBznHrIOqj9C/VlY5mtVi0fkn5ES3zU+KZQ50f5GeQ7Xi1VaTWjvRvZRXHZo8LgW9hsUdSi4h3qnSUn2cT5OL9kxSVKWZmu9WAkem6JBRJaVQ7FSbj0faF4BJBua3CpYflkMHilM1Ac8BIzxYX39p
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Nov 2017 16:43:20.3461 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 9263e41b-e686-4cae-2859-08d52210d3ce
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0502MB3659
Subject: [dpdk-dev] [PATCH v5 6/8] net/mlx4: separate Tx segment cases
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Thu, 02 Nov 2017 16:43:22 -0000
Optimize single segment case by processing it in different block which
prevents checks, calculations and barriers relevant only for multi
segment case.
Call a dedicated function for handling multi segments case.
Signed-off-by: Matan Azrad <matan@mellanox.com>
Signed-off-by: Ophir Munk <ophirmu@mellanox.com>
Acked-by: Adrien Mazarguil <adrien.mazarguil@6wind.com>
---
drivers/net/mlx4/mlx4_rxtx.c | 237 +++++++++++++++++++++++++++----------------
1 file changed, 152 insertions(+), 85 deletions(-)
diff --git a/drivers/net/mlx4/mlx4_rxtx.c b/drivers/net/mlx4/mlx4_rxtx.c
index abd125f..d13c8d2 100644
--- a/drivers/net/mlx4/mlx4_rxtx.c
+++ b/drivers/net/mlx4/mlx4_rxtx.c
@@ -61,6 +61,9 @@
#include "mlx4_rxtx.h"
#include "mlx4_utils.h"
+#define WQE_ONE_DATA_SEG_SIZE \
+ (sizeof(struct mlx4_wqe_ctrl_seg) + sizeof(struct mlx4_wqe_data_seg))
+
/**
* Pointer-value pair structure used in tx_post_send for saving the first
* DWORD (32 byte) of a TXBB.
@@ -140,21 +143,18 @@ struct pv {
* 0 on success, -1 on failure.
*/
static int
-mlx4_txq_complete(struct txq *txq)
+mlx4_txq_complete(struct txq *txq, const unsigned int elts_n,
+ struct mlx4_sq *sq)
{
unsigned int elts_comp = txq->elts_comp;
unsigned int elts_tail = txq->elts_tail;
- const unsigned int elts_n = txq->elts_n;
struct mlx4_cq *cq = &txq->mcq;
- struct mlx4_sq *sq = &txq->msq;
struct mlx4_cqe *cqe;
uint32_t cons_index = cq->cons_index;
uint16_t new_index;
uint16_t nr_txbbs = 0;
int pkts = 0;
- if (unlikely(elts_comp == 0))
- return 0;
/*
* Traverse over all CQ entries reported and handle each WQ entry
* reported by them.
@@ -237,6 +237,120 @@ struct pv {
return buf->pool;
}
+static int
+mlx4_tx_burst_segs(struct rte_mbuf *buf, struct txq *txq,
+ struct mlx4_wqe_ctrl_seg **pctrl)
+{
+ int wqe_real_size;
+ int nr_txbbs;
+ struct pv *pv = (struct pv *)txq->bounce_buf;
+ struct mlx4_sq *sq = &txq->msq;
+ uint32_t head_idx = sq->head & sq->txbb_cnt_mask;
+ struct mlx4_wqe_ctrl_seg *ctrl;
+ struct mlx4_wqe_data_seg *dseg;
+ struct rte_mbuf *sbuf;
+ uint32_t lkey;
+ uintptr_t addr;
+ uint32_t byte_count;
+ int pv_counter = 0;
+
+ /* Calculate the needed work queue entry size for this packet. */
+ wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) +
+ buf->nb_segs * sizeof(struct mlx4_wqe_data_seg);
+ nr_txbbs = MLX4_SIZE_TO_TXBBS(wqe_real_size);
+ /*
+ * Check that there is room for this WQE in the send queue and that
+ * the WQE size is legal.
+ */
+ if (((sq->head - sq->tail) + nr_txbbs +
+ sq->headroom_txbbs) >= sq->txbb_cnt ||
+ nr_txbbs > MLX4_MAX_WQE_TXBBS) {
+ return -1;
+ }
+ /* Get the control and data entries of the WQE. */
+ ctrl = (struct mlx4_wqe_ctrl_seg *)mlx4_get_send_wqe(sq, head_idx);
+ dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl +
+ sizeof(struct mlx4_wqe_ctrl_seg));
+ *pctrl = ctrl;
+ /* Fill the data segments with buffer information. */
+ for (sbuf = buf; sbuf != NULL; sbuf = sbuf->next, dseg++) {
+ addr = rte_pktmbuf_mtod(sbuf, uintptr_t);
+ rte_prefetch0((volatile void *)addr);
+ /* Handle WQE wraparound. */
+ if (dseg >= (struct mlx4_wqe_data_seg *)sq->eob)
+ dseg = (struct mlx4_wqe_data_seg *)sq->buf;
+ dseg->addr = rte_cpu_to_be_64(addr);
+ /* Memory region key (big endian) for this memory pool. */
+ lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(sbuf));
+ dseg->lkey = rte_cpu_to_be_32(lkey);
+#ifndef NDEBUG
+ /* Calculate the needed work queue entry size for this packet */
+ if (unlikely(dseg->lkey == rte_cpu_to_be_32((uint32_t)-1))) {
+ /* MR does not exist. */
+ DEBUG("%p: unable to get MP <-> MR association",
+ (void *)txq);
+ /*
+ * Restamp entry in case of failure.
+ * Make sure that size is written correctly
+ * Note that we give ownership to the SW, not the HW.
+ */
+ wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) +
+ buf->nb_segs * sizeof(struct mlx4_wqe_data_seg);
+ ctrl->fence_size = (wqe_real_size >> 4) & 0x3f;
+ mlx4_txq_stamp_freed_wqe(sq, head_idx,
+ (sq->head & sq->txbb_cnt) ? 0 : 1);
+ return -1;
+ }
+#endif /* NDEBUG */
+ if (likely(sbuf->data_len)) {
+ byte_count = rte_cpu_to_be_32(sbuf->data_len);
+ } else {
+ /*
+ * Zero length segment is treated as inline segment
+ * with zero data.
+ */
+ byte_count = RTE_BE32(0x80000000);
+ }
+ /*
+ * If the data segment is not at the beginning of a
+ * Tx basic block (TXBB) then write the byte count,
+ * else postpone the writing to just before updating the
+ * control segment.
+ */
+ if ((uintptr_t)dseg & (uintptr_t)(MLX4_TXBB_SIZE - 1)) {
+ /*
+ * Need a barrier here before writing the byte_count
+ * fields to make sure that all the data is visible
+ * before the byte_count field is set.
+ * Otherwise, if the segment begins a new cacheline,
+ * the HCA prefetcher could grab the 64-byte chunk and
+ * get a valid (!= 0xffffffff) byte count but stale
+ * data, and end up sending the wrong data.
+ */
+ rte_io_wmb();
+ dseg->byte_count = byte_count;
+ } else {
+ /*
+ * This data segment starts at the beginning of a new
+ * TXBB, so we need to postpone its byte_count writing
+ * for later.
+ */
+ pv[pv_counter].dseg = dseg;
+ pv[pv_counter++].val = byte_count;
+ }
+ }
+ /* Write the first DWORD of each TXBB save earlier. */
+ if (pv_counter) {
+ /* Need a barrier here before writing the byte_count. */
+ rte_io_wmb();
+ for (--pv_counter; pv_counter >= 0; pv_counter--)
+ pv[pv_counter].dseg->byte_count = pv[pv_counter].val;
+ }
+ /* Fill the control parameters for this packet. */
+ ctrl->fence_size = (wqe_real_size >> 4) & 0x3f;
+ return nr_txbbs;
+}
+
/**
* DPDK callback for Tx.
*
@@ -260,10 +374,11 @@ struct pv {
unsigned int i;
unsigned int max;
struct mlx4_sq *sq = &txq->msq;
- struct pv *pv = (struct pv *)txq->bounce_buf;
+ int nr_txbbs;
assert(txq->elts_comp_cd != 0);
- mlx4_txq_complete(txq);
+ if (likely(txq->elts_comp != 0))
+ mlx4_txq_complete(txq, elts_n, sq);
max = (elts_n - (elts_head - txq->elts_tail));
if (max > elts_n)
max -= elts_n;
@@ -282,7 +397,6 @@ struct pv {
uint32_t owner_opcode = MLX4_OPCODE_SEND;
struct mlx4_wqe_ctrl_seg *ctrl;
struct mlx4_wqe_data_seg *dseg;
- struct rte_mbuf *sbuf;
union {
uint32_t flags;
uint16_t flags16[2];
@@ -290,10 +404,6 @@ struct pv {
uint32_t head_idx = sq->head & sq->txbb_cnt_mask;
uint32_t lkey;
uintptr_t addr;
- uint32_t byte_count;
- int wqe_real_size;
- int nr_txbbs;
- int pv_counter = 0;
/* Clean up old buffer. */
if (likely(elt->buf != NULL)) {
@@ -312,38 +422,29 @@ struct pv {
} while (tmp != NULL);
}
RTE_MBUF_PREFETCH_TO_FREE(elt_next->buf);
- /*
- * Calculate the needed work queue entry size
- * for this packet.
- */
- wqe_real_size = sizeof(struct mlx4_wqe_ctrl_seg) +
- buf->nb_segs * sizeof(struct mlx4_wqe_data_seg);
- nr_txbbs = MLX4_SIZE_TO_TXBBS(wqe_real_size);
- /*
- * Check that there is room for this WQE in the send
- * queue and that the WQE size is legal.
- */
- if (((sq->head - sq->tail) + nr_txbbs +
- sq->headroom_txbbs) >= sq->txbb_cnt ||
- nr_txbbs > MLX4_MAX_WQE_TXBBS) {
- elt->buf = NULL;
- break;
- }
- /* Get the control and data entries of the WQE. */
- ctrl = (struct mlx4_wqe_ctrl_seg *)
- mlx4_get_send_wqe(sq, head_idx);
- dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl +
- sizeof(struct mlx4_wqe_ctrl_seg));
- /* Fill the data segments with buffer information. */
- for (sbuf = buf; sbuf != NULL; sbuf = sbuf->next, dseg++) {
- addr = rte_pktmbuf_mtod(sbuf, uintptr_t);
+ if (buf->nb_segs == 1) {
+ /*
+ * Check that there is room for this WQE in the send
+ * queue and that the WQE size is legal
+ */
+ if (((sq->head - sq->tail) + 1 + sq->headroom_txbbs) >=
+ sq->txbb_cnt || 1 > MLX4_MAX_WQE_TXBBS) {
+ elt->buf = NULL;
+ break;
+ }
+ /* Get the control and data entries of the WQE. */
+ ctrl = (struct mlx4_wqe_ctrl_seg *)
+ mlx4_get_send_wqe(sq, head_idx);
+ dseg = (struct mlx4_wqe_data_seg *)((uintptr_t)ctrl +
+ sizeof(struct mlx4_wqe_ctrl_seg));
+ addr = rte_pktmbuf_mtod(buf, uintptr_t);
rte_prefetch0((volatile void *)addr);
/* Handle WQE wraparound. */
if (dseg >= (struct mlx4_wqe_data_seg *)sq->eob)
dseg = (struct mlx4_wqe_data_seg *)sq->buf;
dseg->addr = rte_cpu_to_be_64(addr);
/* Memory region key (big endian). */
- lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(sbuf));
+ lkey = mlx4_txq_mp2mr(txq, mlx4_txq_mb2mp(buf));
dseg->lkey = rte_cpu_to_be_32(lkey);
#ifndef NDEBUG
if (unlikely(dseg->lkey ==
@@ -357,61 +458,27 @@ struct pv {
* Note that we give ownership to the SW,
* not the HW.
*/
- ctrl->fence_size = (wqe_real_size >> 4) & 0x3f;
+ ctrl->fence_size =
+ (WQE_ONE_DATA_SEG_SIZE >> 4) & 0x3f;
mlx4_txq_stamp_freed_wqe(sq, head_idx,
(sq->head & sq->txbb_cnt) ? 0 : 1);
elt->buf = NULL;
break;
}
#endif /* NDEBUG */
- if (likely(sbuf->data_len)) {
- byte_count = rte_cpu_to_be_32(sbuf->data_len);
- } else {
- /*
- * Zero length segment is treated as inline
- * segment with zero data.
- */
- byte_count = RTE_BE32(0x80000000);
- }
- /*
- * If the data segment is not at the beginning
- * of a Tx basic block (TXBB) then write the
- * byte count, else postpone the writing to
- * just before updating the control segment.
- */
- if ((uintptr_t)dseg & (uintptr_t)(MLX4_TXBB_SIZE - 1)) {
- /*
- * Need a barrier here before writing the
- * byte_count fields to make sure that all the
- * data is visible before the byte_count field
- * is set. otherwise, if the segment begins a
- * new cacheline, the HCA prefetcher could grab
- * the 64-byte chunk and get a valid
- * (!= 0xffffffff) byte count but stale data,
- * and end up sending the wrong data.
- */
- rte_io_wmb();
- dseg->byte_count = byte_count;
- } else {
- /*
- * This data segment starts at the beginning of
- * a new TXBB, so we need to postpone its
- * byte_count writing for later.
- */
- pv[pv_counter].dseg = dseg;
- pv[pv_counter++].val = byte_count;
- }
- }
- /* Write the first DWORD of each TXBB save earlier. */
- if (pv_counter) {
- /* Need a barrier before writing the byte_count. */
+ /* Need a barrier here before byte count store. */
rte_io_wmb();
- for (--pv_counter; pv_counter >= 0; pv_counter--)
- pv[pv_counter].dseg->byte_count =
- pv[pv_counter].val;
+ dseg->byte_count = rte_cpu_to_be_32(buf->data_len);
+ /* Fill the control parameters for this packet. */
+ ctrl->fence_size = (WQE_ONE_DATA_SEG_SIZE >> 4) & 0x3f;
+ nr_txbbs = 1;
+ } else {
+ nr_txbbs = mlx4_tx_burst_segs(buf, txq, &ctrl);
+ if (nr_txbbs < 0) {
+ elt->buf = NULL;
+ break;
+ }
}
- /* Fill the control parameters for this packet. */
- ctrl->fence_size = (wqe_real_size >> 4) & 0x3f;
/*
* For raw Ethernet, the SOLICIT flag is used to indicate
* that no ICRC should be calculated.
--
1.8.3.1