From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0046.outbound.protection.outlook.com [104.47.2.46]) by dpdk.org (Postfix) with ESMTP id E856C2BF9 for ; Thu, 28 Jun 2018 17:19:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cIjOpX0Su4WESBTj0ZNd8FydN8smcf/y2txlpffFQR0=; b=gnhVjteQqEXQHmGHyWXtM1OJ3Te6jSasAVHSvWcYS7yuhyLCsxr3oAIeL5jKBckJ0MtLwSlHSW381Wf0JWUjF9U2DNveTHw8uMkrx3sz71H1JwUZpr5lMswv2t9/cyYg07tdQc1ddNl56M40HcU09To/zv1YzMRTIIz3iE+Uifc= Received: from VI1PR0501MB2608.eurprd05.prod.outlook.com (10.168.137.20) by VI1PR0501MB2845.eurprd05.prod.outlook.com (10.172.11.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.884.22; Thu, 28 Jun 2018 15:19:33 +0000 Received: from VI1PR0501MB2608.eurprd05.prod.outlook.com ([fe80::9dd0:9bdb:fd59:b615]) by VI1PR0501MB2608.eurprd05.prod.outlook.com ([fe80::9dd0:9bdb:fd59:b615%7]) with mapi id 15.20.0906.023; Thu, 28 Jun 2018 15:19:33 +0000 From: Matan Azrad To: Mordechay Haimovsky , Adrien Mazarguil CC: "dev@dpdk.org" Thread-Topic: [PATCH v3] net/mlx4: support hardware TSO Thread-Index: AQHUDt6Ihjvpgyieu0adN7JaQGghXqR1n7DQ Date: Thu, 28 Jun 2018 15:19:33 +0000 Message-ID: References: <1530187032-6489-1-git-send-email-motih@mellanox.com> <1530190137-17848-1-git-send-email-motih@mellanox.com> In-Reply-To: <1530190137-17848-1-git-send-email-motih@mellanox.com> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; VI1PR0501MB2845; 7:RfJLHdBG101ZUWuY2/rKWMoVPkFHS9BFr29JtYRerHHtOnEVTp2YsA0yaeNb/yRsTxeDR1NPBAyIqyBhpzAmf1VZnasvYAUeM/ZuCEYAENAFEZs+5JqM4ouKn7FtXsN0DL+kc3oyBnX+gR9fACurKhKI2o9q/0bGViHZoEgZVeftmt1uHJHT/0+uk2X5hujv4CFKFkfSoO56X50aXcgdZy/Japcn7convsJguNhbFgxZhi/2xIGl/BOnq2kPFD80 x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: 00f86b0c-ab40-4968-eb51-08d5dd0a8d40 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652034)(8989117)(5600026)(711020)(48565401081)(2017052603328)(7153060)(7193020); SRVR:VI1PR0501MB2845; x-ms-traffictypediagnostic: VI1PR0501MB2845: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(10201501046)(3231254)(944501410)(52105095)(93006095)(93001095)(3002001)(6055026)(149027)(150027)(6041310)(20161123564045)(20161123558120)(20161123562045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011)(7699016); SRVR:VI1PR0501MB2845; BCL:0; PCL:0; RULEID:; SRVR:VI1PR0501MB2845; x-forefront-prvs: 0717E25089 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(376002)(136003)(396003)(346002)(366004)(199004)(189003)(5250100002)(26005)(6506007)(8676002)(7736002)(55016002)(5660300001)(2906002)(33656002)(305945005)(229853002)(6436002)(74316002)(97736004)(25786009)(6246003)(66066001)(6116002)(3846002)(4326008)(9686003)(81166006)(106356001)(68736007)(478600001)(53936002)(8936002)(14454004)(81156014)(2900100001)(486006)(86362001)(76176011)(256004)(14444005)(105586002)(102836004)(99286004)(11346002)(446003)(110136005)(316002)(7696005)(476003); DIR:OUT; SFP:1101; SCL:1; SRVR:VI1PR0501MB2845; H:VI1PR0501MB2608.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: wGd7xvUkRAiOz08HaTuNAtA0N+be4G7LxhV9VSW+jAXKH0tnB/bpN9F43SRhg3qmJqMnv3JdCzQ+ecOr80M0d64xXSPjqUZ3Kp0wKO+kc5RIidNspmNp8F7El7M5bHsjnMkEK+vdvLRAU0yYPMlWbUq4Y4yfcXcPWWb4a4hoi07nfF3EOkmS16xYHvFuydtUbAJ6+L9x3tYgUJVa0h/c+9uyIXau+G6UCdvC1Ml9dnRqiMo7uSVPVMnw4wgaz/dtPoPFig+Z7FTUoagh8smPDbhOwNz9DMyzAj6k+AELFIk9j0HaArneAiQr5Dl53QSGFj6Cbpi1iYT+yz0kLajcaufAParPHilIMwoqa0YgeMo= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 00f86b0c-ab40-4968-eb51-08d5dd0a8d40 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jun 2018 15:19:33.0445 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2845 Subject: Re: [dpdk-dev] [PATCH v3] net/mlx4: support hardware TSO X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jun 2018 15:19:35 -0000 Hi Moti I started to review it but not finished all :) Please see some comments\questions, I will continue the review again in the next version, after addressing the = next comments and Adrien comments. From: Mordechay Haimovsky > + * No TSO SIZE is defined in DPDK, need to figure it out > + * in order to see if we can support it. > + */ > + mbuf.tso_segsz =3D size_test; > + priv->tso =3D > + ((device_attr_ex.tso_caps.max_tso >=3D mbuf.tso_segsz) > && > + (device_attr_ex.tso_caps.supported_qpts & > + (1 << IBV_QPT_RAW_PACKET))); > + if (priv->tso) > + priv->tso_max_payload_sz =3D > + device_attr_ex.tso_caps.max_tso; Are all the tso_caps fields exist in old rdma-core versions? > + DEBUG("TSO is %ssupported", > + priv->tso ? "" : "not "); > /* Configure the first MAC address by default. */ > err =3D mlx4_get_mac(priv, &mac.addr_bytes); > if (err) { ... > +mlx4_tx_burst_tso_get_params(struct rte_mbuf *buf, > + struct txq *txq, > + struct tso_info *tinfo) > +{ > + struct mlx4_sq *sq =3D &txq->msq; > + const uint8_t tunneled =3D txq->priv->hw_csum_l2tun && > + (buf->ol_flags & PKT_TX_TUNNEL_MASK); > + > + tinfo->tso_header_sz =3D buf->l2_len + buf->l3_len + buf->l4_len; > + if (tunneled) > + tinfo->tso_header_sz +=3D buf->outer_l2_len + buf- > >outer_l3_len; Are those tunnel sizes include outer and inner vlan sizes? > + if (unlikely(buf->tso_segsz =3D=3D 0 || tinfo->tso_header_sz =3D=3D 0))= { > + DEBUG("%p: Invalid TSO parameters", (void *)txq); > + return -EINVAL; > + } > + /* First segment must contain all TSO headers. */ > + if (unlikely(tinfo->tso_header_sz > MLX4_MAX_TSO_HEADER) || > + tinfo->tso_header_sz > buf->data_len) { > + DEBUG("%p: Invalid TSO header length", (void *)txq); > + return -EINVAL; > + } > + /* > + * Calculate the WQE TSO segment size > + * Note: > + * 1. An LSO segment must be padded such that the subsequent data > + * segment is 16-byte aligned. > + * 2. The start address of the TSO segment is always 16 Bytes aligned. > + */ ... > +static inline int > +mlx4_tx_burst_fill_tso_segs(struct rte_mbuf *buf, > + struct txq *txq, > + const struct tso_info *tinfo, > + volatile struct mlx4_wqe_data_seg *dseg, > + struct pv *pv, int *pv_counter) > +{ > + uint32_t lkey; > + int nb_segs =3D buf->nb_segs; > + int nb_segs_txbb; > + struct mlx4_sq *sq =3D &txq->msq; > + struct rte_mbuf *sbuf =3D buf; > + uint16_t sb_of =3D tinfo->tso_header_sz; > + uint16_t data_len; > + > + while (nb_segs > 0) { > + /* Wrap dseg if it points at the end of the queue. */ > + if ((volatile uint8_t *)dseg >=3D sq->eob) > + dseg =3D (volatile struct mlx4_wqe_data_seg *) > + (volatile uint8_t *)dseg - sq->size; I think we don't need this check in the first time, so maybe move it to the= end of the loop is better. > + /* how many dseg entries do we have in the current TXBB ? */ > + nb_segs_txbb =3D > + (MLX4_TXBB_SIZE / sizeof(struct mlx4_wqe_data_seg)) > - > + ((uintptr_t)dseg & (MLX4_TXBB_SIZE - 1)) / > + sizeof(struct mlx4_wqe_data_seg); > + switch (nb_segs_txbb) { I think this switch case is corrupted, what's happen If we have only 1\2\3 = segments but we are in the start of txbbb? You are going to write the first byte of the txbb firstly, no? > + case 4: > + /* Memory region key for this memory pool. */ > + lkey =3D mlx4_tx_mb2mr(txq, sbuf); > + if (unlikely(lkey =3D=3D (uint32_t)-1)) > + goto lkey_err; > + dseg->addr =3D > + rte_cpu_to_be_64(rte_pktmbuf_mtod_offset(sbuf, > + uintptr_t, > + sb_of)); > + dseg->lkey =3D lkey; > + /* > + * This data segment starts at the beginning of a new > + * TXBB, so we need to postpone its byte_count writing > + * for later. > + */ > + pv[*pv_counter].dseg =3D dseg; > + /* > + * Zero length segment is treated as inline segment > + * with zero data. > + */ > + data_len =3D sbuf->data_len - sb_of; > + pv[(*pv_counter)++].val =3D > + rte_cpu_to_be_32(data_len ? > + data_len : > + 0x80000000); > + sb_of =3D 0; > + sbuf =3D sbuf->next; > + dseg++; > + if (--nb_segs =3D=3D 0) > + break; > + /* fallthrough */ > + case 3: > + lkey =3D mlx4_tx_mb2mr(txq, sbuf); > + if (unlikely(lkey =3D=3D (uint32_t)-1)) > + goto lkey_err; > + data_len =3D sbuf->data_len - sb_of; > + mlx4_fill_tx_data_seg(dseg, > + lkey, > + rte_pktmbuf_mtod_offset(sbuf, > + uintptr_t, > + sb_of), > + rte_cpu_to_be_32(data_len ? > + data_len : > + 0x80000000)); > + sb_of =3D 0; > + sbuf =3D sbuf->next; > + dseg++; > + if (--nb_segs =3D=3D 0) > + break; > + /* fallthrough */ > + case 2: > + lkey =3D mlx4_tx_mb2mr(txq, sbuf); > + if (unlikely(lkey =3D=3D (uint32_t)-1)) > + goto lkey_err; > + data_len =3D sbuf->data_len - sb_of; > + mlx4_fill_tx_data_seg(dseg, > + lkey, > + rte_pktmbuf_mtod_offset(sbuf, > + uintptr_t, > + sb_of), > + rte_cpu_to_be_32(data_len ? > + data_len : > + 0x80000000)); > + sb_of =3D 0; > + sbuf =3D sbuf->next; > + dseg++; > + if (--nb_segs =3D=3D 0) > + break; > + /* fallthrough */ > + case 1: > + lkey =3D mlx4_tx_mb2mr(txq, sbuf); > + if (unlikely(lkey =3D=3D (uint32_t)-1)) > + goto lkey_err; > + data_len =3D sbuf->data_len - sb_of; > + mlx4_fill_tx_data_seg(dseg, > + lkey, > + rte_pktmbuf_mtod_offset(sbuf, > + uintptr_t, > + sb_of), > + rte_cpu_to_be_32(data_len ? > + data_len : > + 0x80000000)); > + sb_of =3D 0; > + sbuf =3D sbuf->next; > + dseg++; > + --nb_segs; > + break; > + default: > + /* Should never happen */ > + ERROR("%p: invalid number of txbb data segments > %d", > + (void *)txq, nb_segs_txbb); > + return -EINVAL; > + } > + } > + return 0; > +lkey_err: > + DEBUG("%p: unable to get MP <-> MR association", > + (void *)txq); > + return -EFAULT; > +} Matan