From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-eopbgr50050.outbound.protection.outlook.com [40.107.5.50]) by dpdk.org (Postfix) with ESMTP id EE67A7CB6; Wed, 13 Sep 2017 21:52:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=v6MWbKJfI/o9/oLVgbaQWYuRnaPT6ryclYQ2swXy7Ws=; b=cKZUYSlEuwowXTTNyseIDoaBgsxXbwswFobAxD6Tow9pgrPlB94E2cuyjDeBbfDFekBxa6obQnlSOK5W5oj2FgsHjnaj5BBv7AGgrIAYEzE23slhivITTdNJDjF5KfJK/1It8WnwOE7oz0KsmxtDZIO0HL/WmnqVe11SI9E+dNc= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; Received: from yongseok-MBP.local (209.116.155.178) by HE1PR0501MB2042.eurprd05.prod.outlook.com (2603:10a6:3:35::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.35.12; Wed, 13 Sep 2017 19:52:07 +0000 Date: Wed, 13 Sep 2017 12:51:55 -0700 From: Yongseok Koh To: Shahaf Shuler Cc: nelio.laranjeiro@6wind.com, adrien.mazarguil@6wind.com, dev@dpdk.org, stable@dpdk.org Message-ID: <20170913195154.GA4263@yongseok-MBP.local> References: <7b6d42c0c5e1a04f8483b6546ea0b1db8fb7ceee.1505133966.git.shahafs@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.7.2 (2016-11-26) X-Originating-IP: [209.116.155.178] X-ClientProxiedBy: DM5PR13CA0032.namprd13.prod.outlook.com (2603:10b6:3:7b::18) To HE1PR0501MB2042.eurprd05.prod.outlook.com (2603:10a6:3:35::20) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: de14ef45-a919-483a-d523-08d4fae0eabe X-MS-Office365-Filtering-HT: Tenant X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(48565401081)(300000503095)(300135400095)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:HE1PR0501MB2042; X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2042; 3:3tAnB7+5oBdJLqXQA6JMHEZVZFvKy5JHg+E2oyN4gxqPXkR4k1ipijEHmKNMIh8Uvs9TH5uvi7xYGBElj8xjIvrKL7LVpzzcONEQi0PImg01M/p48EzIq1Ww47lxdsGsdPJQuZ0/UVavnJSfREhZHn36PezLqt/eDoinNd/UYOPpOLL0PbqoNDTqNjMO86OgOHw1rhQAkkYHDKyXIuNqcgWYsPVx349AcSO9udnOzO8f20S78sBOcg679UPn0OWM; 25:X65DSfYGVcIHLii6s0y7hQ6JJB871A74tMVOfkxpe0gs7MqrCz1KYhbPP7SWQf4DAu7EvBeNKTIBNokYBnJh+W7qQlGWYyRfzJ6Iz3tnLzcnNJp8xB9hrZzpAc47Cy8DIjUacL3E2uHwo9yhTCZlCuJ/5kWZ6OODCkyXsBvakR6/OY9+JXuHPm6ym7McImmDDistRAyGetJr/Ix4j7/n0kiLWb+1LwQsqEOOsUA844jfiHYDaNSSN65mhC+aWiP+q3RZ5EF6FEBnOKYJQeK9kx2tEylB0bVcblc5A4071zsdVc4aRhKBgiqcorof42CLYlqjPGjW8SoaeWCywiwFfQ==; 31:TdmGj4H/NEjR7pkvfBvxklEhonbeHKXRjxm7lca/6UGIc+9T8A3Ki0e5DNumBI1UYubMY95/K6o0a4sUfr6753Om0RVvVZU5r3Tu0iyhV2tI0DtJAmBugd2mRhkgLOr6LLHfR8Lss9leNDG6mEH0huKfQjfoG5ICjDAyD6WPNDI5PiDaDYCgGeVLtvtwrFcfjo8SkZKCc+SEVqVnU7WyMMFJqLQDw6euxAkW5jbrd/s= X-MS-TrafficTypeDiagnostic: HE1PR0501MB2042: X-LD-Processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2042; 20:7mWFJO7Ob5hEUvle1ePtrgFBco8V5emdmqhwj3KC6FRrmpgJpdANOMGX7fMzkE1LfP2tjZit265ipdVb1ncD9fWeZqLo6BoQV+eZ/ml57XtClfpzwGUAlRqaCIm1xqpySaNgNylmZJu+0eqfZC69PEyOXmyY565+y0ASm4kWiTOHD/h2OfLFES3KJr1Zy/W3HwR6jGK8YvtUW/ksKUlfWhDhkGXZDq84Knk3u7mTwFUvRAQs7AN495oYGrVc4lCge+AwPBV9YyG8GXtlbuerY/9IhQd5gcJVjBiwCa6AVUMsmi1oDLjaEXEaIYet3pIt9HDq9m3J52dUaDLYyRiJdePXkDMhWwryyAfj8YhvetwF8dFm2ya2rB6WWElAjgkzv3Wh3oUBOr5IiKjvte5TD6wxjx1jYloCj168iaKs9Qx6twh4u3ggPJ7RgzCXE3BijPIOEfSTvuOmu4dRs+ML/uElv/xQnwCOkM8uPmz+JrLP38ncA7i2fR6dkdhoSJss; 4:cYgmxYUT8UMJtFDk1s9qoTwB4lI+nHCUCXKnjnwaKjOUvLmvSjgaDnl6j11+zz2rsM45ExwrEE+Npmo3J0Ou4YO9DJBDGiRgF1c+jufDuJoulEKLNjZNID1axvRiErIht+rCgMN0w7XPKXNOazIYUZzZsbhuB5VUQKRuf/KEdGdOHm2/ufoYly2Vhj6Gt28fnIt7r+a8AhlPxvJHjjq0uyrc62Cr6zqm1BcpY/uUVov/C+L7e4SIEWxAM9e0A1tv X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(93006095)(93001095)(100000703101)(100105400095)(3002001)(10201501046)(6055026)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123560025)(20161123564025)(20161123562025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:HE1PR0501MB2042; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:HE1PR0501MB2042; X-Forefront-PRVS: 042957ACD7 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(7370300001)(6009001)(346002)(376002)(189002)(24454002)(199003)(66066001)(305945005)(6666003)(97736004)(25786009)(47776003)(316002)(6636002)(7736002)(83506001)(53936002)(2950100002)(1076002)(16526017)(229853002)(86362001)(575784001)(7350300001)(23726003)(6116002)(3846002)(16586007)(50466002)(478600001)(6506006)(5660300001)(8676002)(81156014)(81166006)(98436002)(2906002)(105586002)(101416001)(110136004)(9686003)(106356001)(55016002)(33656002)(54356999)(4001350100001)(6246003)(4326008)(68736007)(6862004)(76176999)(50986999)(189998001)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:HE1PR0501MB2042; H:yongseok-MBP.local; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; HE1PR0501MB2042; 23:Mh4JMsUXbpXFufRAWeA1nCB97GGaq9eXvVxhqU0?= =?us-ascii?Q?1EoB6b/1rVNov5AnpGzJaSVLmqVsMdS+Q1A/ikZP3QGE3gUiC9Z2jFVi/nks?= =?us-ascii?Q?j0rgdYzx0Z/qhNBE36fiYP0M34fiTsq4QWTzPjeHRMFndPPB0aCFoxFu9a7s?= =?us-ascii?Q?Ebq2sEp+Ev9g/psck1HKMkYOy9IRpxSMJ+5llxdOBV1K7pLyCuZXJE/GzN+M?= =?us-ascii?Q?y8cb4SeN2oPX+a04eKJJVaciq9ZJ+3nu6GNYgKc5tlZNHVek1hHHeFB6veq/?= =?us-ascii?Q?LiYi6dqIW7ew+5DHHDiXi59HRRsE+lgJZLUxI+2ZHb2tl8AI+9T+f16B7wa4?= =?us-ascii?Q?aGZzeKYIgKKp++tC0YPa4kugK6ERo+g3B10dLVX8LxEVGxBDIpmTwEJNoD8Z?= =?us-ascii?Q?VcyDaQs6zhuNyA4PcxRlxic4lhcEf+v0IwC3+/YFYsBHTqIfZleq+er8aQ+4?= =?us-ascii?Q?uEEe8563PxjQ7vS/TIJdOp6fbd8633OlVNl3IsgUMMZ+HvvcipF89p5jIDLX?= =?us-ascii?Q?npOAZ8v1NPGv+ycH6szFKEnW8RkSqrm8MgvxKsuuwy9IuYhA/SPFqDnUFXct?= =?us-ascii?Q?tUCr1sFnA4jSJcOpdVffGyIE8kjBs17WQiTZ1RaMXysXzFlYeq6fKB+BHd34?= =?us-ascii?Q?co3A0xtuvp48Yln1ietKju2yo4cY/13bn3F1eWTSl1TeOP5RG1dbT/BRRtw6?= =?us-ascii?Q?xeh5X/PI/8k4E8CmmQXZo8V2/nogq9DAui0tNryXf0a5AaP74D1p8lbcBfD0?= =?us-ascii?Q?58RrD3IHq363m+hukxCCLUCOS2prqaP4XlMOXTsJZGkW5gu19sFqGObC5z5I?= =?us-ascii?Q?+cOFvrpy35fnFtq5BhROS8Iu2SRGF2MX3Hz8WL4ZWiX3Xvhh3BPIdA6Qlopk?= =?us-ascii?Q?J57yIhwMSHnDdb36v6CPX0o9oq6vlI2B6y0HXrsvSPvqmKcUut2PD0JHxwXR?= =?us-ascii?Q?sGeW3u3o1aROeqcW8paAq0OAHWnFlQdoLbFFGjJAZBuZFTT+N1/w5pu3QIFW?= =?us-ascii?Q?+wLlrET56OsWY8+GZ8WeB3n2BUDJNx6nOIrsXd9wLKG+lG93ZqEe/rxkYEnI?= =?us-ascii?Q?eBfd3jLN3lQq71/C7qMjPJm+92NH1Mj4ejm/c0ltzwC9DVDRlpZ15TS4GMFA?= =?us-ascii?Q?NtNeWKjg22MsJ6UgtKD0knxddD2lDtWZ2SQwRkCa93INCH2UCbdLQQBVX1ZO?= =?us-ascii?Q?zvf1ZECpsUTBLwWfJH8XaCCG8p1AueE8u1E9bIjtw71UG3+3CyernF7Xora3?= =?us-ascii?Q?DzDF/dQfR4Qkfusn645YSfPJrxGQNAKnRUuIeC2yeEIxc+SC1aKKtr3js2qb?= =?us-ascii?Q?2Tocmd/hQkSZ407sY4BMW1hw=3D?= X-Microsoft-Exchange-Diagnostics: 1; HE1PR0501MB2042; 6:PC2RM+rIe7TDtYgYT0MJcPnPKigkZ4cNAcYgpcq7Ntmkc2OJly5jvsE6k48/ecCTyDaAKD30p6GP5dU6X+SkKrcOP+YPu+Lb6wKPnNk/kuVtW+LdRvDF/uiHRJwaBXBJXnYGkQb//q3468QbBFo9jBzM1QFe5Tn36cT30j3L/9kALvfAqjhgAmc3/qpbZS5PnfH0l1aQRwCEnZIUz5KrJ/1pjdwSxoZl2DH4jjzyAk9HkBdgJgNt9oHwQCHKmjMmQsWj9Dpy5H1ch77rrl6Dtd3foK6bYnDRupwV1BbDc3LVD9AiYAQsDSOtxMuzPCyETqPbUBCtUBoJrfJ4uGZIyA==; 5:DynyqOIt5AF0L++5wv661sHLFqYOM2CbZKhdDXgsMXm99vOPkK5bet3zw/inV3IEFCRYjk+D81vyzPBwCyHplcIvwAgAgV1g8MMRAxlse7xSkue+ls4fwJ2McqjoMcQyTpdXwNBFyw+K9FiFkdKc5g==; 24:8D8DkC5Qg0EbfXNrvEhyIt/Zla5mqBtvvZQuKgMv+Mf9uqVOlsroeVV7CNJuNgBt576gO9HwMNHqwl8nq/W3CpODxoxtmE8VecBXEA0fy/k=; 7:3amHBcN4zxEx8tF9iEbPiVloPrlOVjJFHHyweKAr/P8B1XfnDSgHGqh2+FLs4x3I8q3PI6tmN1VpkYeyWYqxYlDKB4tjpBcSnngRH7BtFd/0/BnauLvnhsCp+SOxLo2ItnMZJsTeJvPKXvmd7Fl5R0wr9s3ojGNiHfK4PJejMfb0JvPE+kn77xoWyGAbWAsIRh+PXpaxlkgmVAze+Fo6WzNVQ9/TUE96AdeqpMizjm4= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Sep 2017 19:52:07.0793 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0501MB2042 Subject: Re: [dpdk-stable] [PATCH v4 4/4] net/mlx5: enforce Tx num of segments limitation X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2017 19:52:13 -0000 On Wed, Sep 13, 2017 at 01:50:39PM +0300, Shahaf Shuler wrote: > Mellanox NICs has a limitation on the number of mbuf segments a multi > segment mbuf can have. The max number depends on the Tx offloads requested. > > The current code not enforce such limitation, which might cause > malformed work requests to be written to the device. > > This commit adds verification for the number of mbuf segments posted > to the device. In case of overflow the packet will not be sent. > > In addition update the nic documentation with the limitation. > Considering device limitation is 63 data segments in a work request, the > maximum number of segment in mbuf was calculated taking TSO as the worst > case: > > max_nb_segs = 63 - (control_segment + ethernet segment + > TSO headers inline + inline segment + > extra inline to align to cacheline) > > Cc: stable@dpdk.org > > Signed-off-by: Shahaf Shuler > --- > doc/guides/nics/mlx5.rst | 2 ++ > drivers/net/mlx5/mlx5_defs.h | 3 ++- > drivers/net/mlx5/mlx5_prm.h | 3 +++ > drivers/net/mlx5/mlx5_rxtx.c | 4 ++++ > drivers/net/mlx5/mlx5_rxtx_vec_sse.c | 5 +++++ > drivers/net/mlx5/mlx5_txq.c | 27 +++++++++++++++++++++++++++ > 6 files changed, 43 insertions(+), 1 deletion(-) > > diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst > index f4cb18bca..d8244de97 100644 > --- a/doc/guides/nics/mlx5.rst > +++ b/doc/guides/nics/mlx5.rst > @@ -124,6 +124,8 @@ Limitations > > Will match any ipv4 packet (VLAN included). > > +- A multi segment mbuf must have less than 50 segments. That means mbuf->nb_segs < 50. Isn't it better to use either "multiple segment packet" or "multi-segment packet"? Also, more information might be needed here. If MPW/eMPW is enabled, the code restricts the max number of segments up to MLX5_MPW_DSEG_MAX(5). > + > Configuration > ------------- > > diff --git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h > index a76bc6f65..3de0e5d81 100644 > --- a/drivers/net/mlx5/mlx5_defs.h > +++ b/drivers/net/mlx5/mlx5_defs.h > @@ -100,7 +100,8 @@ > > /* > * Maximum size of burst for vectorized Tx. This is related to the maximum size > - * of Enhaned MPW (eMPW) WQE as vectorized Tx is supported with eMPW. > + * of Enhanced MPW (eMPW) WQE as vectorized Tx is supported with eMPW. > + * Careful when changing, large value can cause wqe DS to overlap. wqe -> WQE. > */ > #define MLX5_VPMD_TX_MAX_BURST 32U > > diff --git a/drivers/net/mlx5/mlx5_prm.h b/drivers/net/mlx5/mlx5_prm.h > index 608072f7e..bc2b72333 100644 > --- a/drivers/net/mlx5/mlx5_prm.h > +++ b/drivers/net/mlx5/mlx5_prm.h > @@ -154,6 +154,9 @@ > /* Default mark value used when none is provided. */ > #define MLX5_FLOW_MARK_DEFAULT 0xffffff > > +/* Maximum number of DS in WQE. */ > +#define MLX5_MAX_DS 63 How about make it consistent with MLX5_MPW_DSEG_MAX by naming MLX5_DSEG_MAX? > + > /* Subset of struct mlx5_wqe_eth_seg. */ > struct mlx5_wqe_eth_seg_small { > uint32_t rsvd0; > diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c > index 7567f2329..fdd7067da 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.c > +++ b/drivers/net/mlx5/mlx5_rxtx.c > @@ -661,6 +661,10 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, uint16_t pkts_n) > else > j += sg; > next_pkt: > + if (ds > MLX5_MAX_DS) { > + txq->stats.oerrors++; > + break; > + } > ++elts_head; > ++pkts; > ++i; > diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c > index f89762ff8..3583e6780 100644 > --- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.c > +++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.c > @@ -248,6 +248,10 @@ txq_scatter_v(struct txq *txq, struct rte_mbuf **pkts, uint16_t pkts_n) > if (segs_n == 1 || > max_elts < segs_n || max_wqe < 2) > break; > + if (segs_n > MLX5_MPW_DSEG_MAX) { > + txq->stats.oerrors++; > + break; > + } > wqe = &((volatile struct mlx5_wqe64 *) > txq->wqes)[wqe_ci & wq_mask].hdr; > if (buf->ol_flags & > @@ -365,6 +369,7 @@ txq_burst_v(struct txq *txq, struct rte_mbuf **pkts, uint16_t pkts_n, > max_elts = (elts_n - (elts_head - txq->elts_tail)); > max_wqe = (1u << txq->wqe_n) - (txq->wqe_ci - txq->wqe_pi); > pkts_n = RTE_MIN((unsigned int)RTE_MIN(pkts_n, max_wqe), max_elts); > + assert(pkts_n <= MLX5_MAX_DS - nb_dword_in_hdr); > if (unlikely(!pkts_n)) > return 0; > elts = &(*txq->elts)[elts_head & elts_m]; > diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c > index 4b0b532b1..091b1a93d 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -288,6 +288,8 @@ txq_ctrl_setup(struct rte_eth_dev *dev, struct txq_ctrl *txq_ctrl, > .comp_mask = IBV_EXP_QP_INIT_ATTR_PD, > }; > if (priv->txq_inline && (priv->txqs_n >= priv->txqs_inline)) { > + unsigned int ds_cnt; > + > tmpl.txq.max_inline = > ((priv->txq_inline + (RTE_CACHE_LINE_SIZE - 1)) / > RTE_CACHE_LINE_SIZE); > @@ -320,6 +322,31 @@ txq_ctrl_setup(struct rte_eth_dev *dev, struct txq_ctrl *txq_ctrl, > attr.init.cap.max_inline_data = > tmpl.txq.max_inline * RTE_CACHE_LINE_SIZE; > } > + /* > + * Check if the inline size is too large in a way which > + * can make the wqe DS to overflow. wqe -> WQE. > + * Considering in calculation: > + * WQE CTRL (1 DS) > + * WQE ETH (1 DS) > + * inline part (N DS) inline -> Inline ? > + */ > + ds_cnt = 2 + > + (attr.init.cap.max_inline_data / MLX5_WQE_DWORD_SIZE); > + if (ds_cnt > MLX5_MAX_DS) { > + unsigned int max_inline = (MLX5_MAX_DS - 2) * > + MLX5_WQE_DWORD_SIZE; > + > + /* Ceil down*/ Missing space and period. Rather, this comment could be unnecessary as the following code is so obvious. Or, you might want to explain why you make it aligned. > + max_inline = max_inline - (max_inline % > + RTE_CACHE_LINE_SIZE); > + WARN("txq inline is too large (%d) setting it to " > + "the maximum possible: %d\n", > + priv->txq_inline, max_inline); > + tmpl.txq.max_inline = max_inline / RTE_CACHE_LINE_SIZE; > + attr.init.cap.max_inline_data = max_inline; > + if (priv->mps == MLX5_MPW_ENHANCED) > + tmpl.txq.inline_max_packet_sz = max_inline; No need to set inline_max_packet_sz. inline_max_packet_sz is to limit the max size of a packet which can be inlined in eMPW mode. As long as txq->max_inline is correctly set, txq->inline_max_packet_sz doesn't affect the total number of DSEGs in a WQE. Thanks, Yongseok