From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 6E1F9A046B for ; Mon, 22 Jul 2019 07:33:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 41AD51B942; Mon, 22 Jul 2019 07:33:49 +0200 (CEST) Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30083.outbound.protection.outlook.com [40.107.3.83]) by dpdk.org (Postfix) with ESMTP id 604864CA9 for ; Mon, 22 Jul 2019 07:33:47 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WvbtoWVC5DYHTh/URv+QS/yBvAAdmhQNnwNDr3Y8nWd6ufeB2i9QOGoTuhCCd9juDYTa+NuwffbzITUl5dPXpV7dpxS8Wf2adRmWUE2D8OCDQjtScyOp5LH7fAkwjaXskOcsahlcDBxvxfxGYildjt7U6NvKivTUdgxCAweELIFg4p/MtnBJLqNMUxm8QC/Plkmy2ebAygFf0TukTorTvKqzZiFXcFWfrlq62LgZrSFWLbRtVDJKItaIcGUWaKNIcnMtazM6LOYsxPrum6Xt8LclQh0IQvp8BBK9y5EBgOI0M/5vzeFRROWcYf6+ufincjVlG5/eYhqy9xzt6LTpxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y2aFZJOzBHCo9tvFQrTXaZjWrreQ9AEPS91cZ/DlnAw=; b=MczgERqy3/A75nt4c3+GfNJFkywiR6hOd6YcgMKsugrEYjTzKKtioxGfSDxhkyIKBYvzheuAKEMkvF2L1ZeIdIpBR+AKilr5iadvta66LptlYO4iGP/FmxU+6QBFVkBRBJvga4qxZW+aYlOI7GBgcDU5oTrEBxkwtoY1wbJSk3MFRtOkY8RP1SCttYyjQb5jtbeG4KRakcRNSyv13HJ3FopnSciDApRDtydpx1fZa1in8vuIy7iZLF8D+mP3VuiERbUft7fop/SQBL1Ko/yOfE2iiWZBnQfrE/Ui1U6RI/4Wm0SUGnlxvMvByxE+TPwrUCTMwnya5tnwqn1S/56Plg== ARC-Authentication-Results: i=1; mx.microsoft.com 1;spf=pass smtp.mailfrom=mellanox.com;dmarc=pass action=none header.from=mellanox.com;dkim=pass header.d=mellanox.com;arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y2aFZJOzBHCo9tvFQrTXaZjWrreQ9AEPS91cZ/DlnAw=; b=Q+zPXsdUm3X56WX/rF0nxDZ1iGwzxW9wK2LWtClc2vrBdtrmJqp8SqCji483Txu22c9f5tzEku7Vcw4gYrWSl9iOORsC9OzeYJhSr7W+k8jV+DfSj/OEdg3AdfVqhRE2gmRbZT8o3b42URscDV7knB+cTi+YOhXYd1KXnYpZBuc= Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com (52.134.72.27) by DB3PR0502MB3995.eurprd05.prod.outlook.com (52.134.72.30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2094.16; Mon, 22 Jul 2019 05:33:45 +0000 Received: from DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::69c1:c0d7:1fa1:f89f]) by DB3PR0502MB3980.eurprd05.prod.outlook.com ([fe80::69c1:c0d7:1fa1:f89f%6]) with mapi id 15.20.2094.013; Mon, 22 Jul 2019 05:33:45 +0000 From: Yongseok Koh To: Slava Ovsiienko CC: "dev@dpdk.org" Thread-Topic: [PATCH v4 5/8] net/mlx5: add Tx datapath configuration and setup Thread-Index: AQHVP9Axq85fX75a1kuqH3IXvbIvbabWHm6A Date: Mon, 22 Jul 2019 05:33:45 +0000 Message-ID: <70148E9C-FAD9-4EB5-99B0-92699CB8F5D5@mellanox.com> References: <1563346400-1762-1-git-send-email-viacheslavo@mellanox.com> <1563719100-368-1-git-send-email-viacheslavo@mellanox.com> <1563719100-368-6-git-send-email-viacheslavo@mellanox.com> In-Reply-To: <1563719100-368-6-git-send-email-viacheslavo@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=yskoh@mellanox.com; x-originating-ip: [69.181.245.183] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b6dc134e-e030-45f4-1c81-08d70e662a7a x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600148)(711020)(4605104)(1401327)(4618075)(2017052603328)(7193020); SRVR:DB3PR0502MB3995; x-ms-traffictypediagnostic: DB3PR0502MB3995: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3044; x-forefront-prvs: 01068D0A20 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(39860400002)(376002)(346002)(366004)(396003)(136003)(199004)(189003)(478600001)(6486002)(33656002)(53936002)(76176011)(53946003)(6436002)(316002)(37006003)(36756003)(6512007)(81156014)(81166006)(186003)(6862004)(25786009)(446003)(8676002)(26005)(30864003)(2616005)(11346002)(476003)(4326008)(229853002)(6636002)(8936002)(71190400001)(71200400001)(486006)(102836004)(7736002)(2906002)(14454004)(6506007)(53546011)(305945005)(6246003)(66946007)(66476007)(66556008)(64756008)(66446008)(66066001)(76116006)(91956017)(14444005)(256004)(6116002)(68736007)(86362001)(99286004)(5660300002)(3846002)(579004); DIR:OUT; SFP:1101; SCL:1; SRVR:DB3PR0502MB3995; H:DB3PR0502MB3980.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: qsMAWe493o3wqgWVbMOc2EuWK9hP9/TQYDxcM4gHDa01eycM4ac6a+cqXqQi63cL0WUaDsfLGM2gyrBpNNaSsg1a6rG3Gi5edRFAgq73vmW2sM7cTh7YsB08YVMQaRTO5+j2UTO60iotqPjOgyL3McbEsafH8oXdnJg0iKVatdd5nNaQ5SHgi2/MhemwPQWvM0XJTi4XmAK43ZNyTGRgi7vpr+3DNtbNsylSrGBA60fsNOEQ0bwX4dzoVfXNnV8RbTgeWSrDXRiRLVjFeksuvH6K72MiOdAZVx7YRlwf7pNSayxIHqTdATIa8N+FUdd01zWA2sUJgcaSDzOGInHCN+IwRT/Dq92Xo5fybhRkW/SGQdc0g3aop8fwm9JgF5DEBENLCCBLLXiQYZ3gweKQgiHhQx0SoHpQqgh6SvuKB54= Content-Type: text/plain; charset="us-ascii" Content-ID: <3840B9EDCEC084489E7431DEEB03DC12@eurprd05.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: b6dc134e-e030-45f4-1c81-08d70e662a7a X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Jul 2019 05:33:45.6564 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: yskoh@mellanox.com X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB3PR0502MB3995 Subject: Re: [dpdk-dev] [PATCH v4 5/8] net/mlx5: add Tx datapath configuration and setup X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > On Jul 21, 2019, at 7:24 AM, Viacheslav Ovsiienko wrote: >=20 > This patch updates the Tx datapath control and configuration > structures and code for mananging Tx datapath settings. >=20 > Signed-off-by: Viacheslav Ovsiienko > --- Acked-by: Yongseok Koh > drivers/net/mlx5/mlx5.c | 123 +++++++++++++++++++++++++ > drivers/net/mlx5/mlx5.h | 2 +- > drivers/net/mlx5/mlx5_rxtx.c | 4 +- > drivers/net/mlx5/mlx5_rxtx.h | 56 ++++++++--- > drivers/net/mlx5/mlx5_txq.c | 215 ++++++++++++++++++++++++++++++++++++++= ++--- > 5 files changed, 370 insertions(+), 30 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c > index bbf2583..37d3c08 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -1185,6 +1185,127 @@ struct mlx5_dev_spawn_data { > } >=20 > /** > + * Configures the minimal amount of data to inline into WQE > + * while sending packets. > + * > + * - the txq_inline_min has the maximal priority, if this > + * key is specified in devargs > + * - if DevX is enabled the inline mode is queried from the > + * device (HCA attributes and NIC vport context if needed). > + * - otherwise L2 mode (18 bytes) is assumed for ConnectX-4/4LX > + * and none (0 bytes) for other NICs > + * > + * @param spawn > + * Verbs device parameters (name, port, switch_info) to spawn. > + * @param config > + * Device configuration parameters. > + */ > +static void > +mlx5_set_min_inline(struct mlx5_dev_spawn_data *spawn, > + struct mlx5_dev_config *config) > +{ > + if (config->txq_inline_min !=3D MLX5_ARG_UNSET) { > + /* Application defines size of inlined data explicitly. */ > + switch (spawn->pci_dev->id.device_id) { > + case PCI_DEVICE_ID_MELLANOX_CONNECTX4: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LX: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: > + if (config->txq_inline_min < > + (int)MLX5_INLINE_HSIZE_L2) { > + DRV_LOG(DEBUG, > + "txq_inline_mix aligned to minimal" > + " ConnectX-4 required value %d", > + (int)MLX5_INLINE_HSIZE_L2); > + config->txq_inline_min =3D MLX5_INLINE_HSIZE_L2; > + } > + break; > + } > + goto exit; > + } > + if (config->hca_attr.eth_net_offloads) { > + /* We have DevX enabled, inline mode queried successfully. */ > + switch (config->hca_attr.wqe_inline_mode) { > + case MLX5_CAP_INLINE_MODE_L2: > + /* outer L2 header must be inlined. */ > + config->txq_inline_min =3D MLX5_INLINE_HSIZE_L2; > + goto exit; > + case MLX5_CAP_INLINE_MODE_NOT_REQUIRED: > + /* No inline data are required by NIC. */ > + config->txq_inline_min =3D MLX5_INLINE_HSIZE_NONE; > + config->hw_vlan_insert =3D > + config->hca_attr.wqe_vlan_insert; > + DRV_LOG(DEBUG, "Tx VLAN insertion is supported"); > + goto exit; > + case MLX5_CAP_INLINE_MODE_VPORT_CONTEXT: > + /* inline mode is defined by NIC vport context. */ > + if (!config->hca_attr.eth_virt) > + break; > + switch (config->hca_attr.vport_inline_mode) { > + case MLX5_INLINE_MODE_NONE: > + config->txq_inline_min =3D > + MLX5_INLINE_HSIZE_NONE; > + goto exit; > + case MLX5_INLINE_MODE_L2: > + config->txq_inline_min =3D > + MLX5_INLINE_HSIZE_L2; > + goto exit; > + case MLX5_INLINE_MODE_IP: > + config->txq_inline_min =3D > + MLX5_INLINE_HSIZE_L3; > + goto exit; > + case MLX5_INLINE_MODE_TCP_UDP: > + config->txq_inline_min =3D > + MLX5_INLINE_HSIZE_L4; > + goto exit; > + case MLX5_INLINE_MODE_INNER_L2: > + config->txq_inline_min =3D > + MLX5_INLINE_HSIZE_INNER_L2; > + goto exit; > + case MLX5_INLINE_MODE_INNER_IP: > + config->txq_inline_min =3D > + MLX5_INLINE_HSIZE_INNER_L3; > + goto exit; > + case MLX5_INLINE_MODE_INNER_TCP_UDP: > + config->txq_inline_min =3D > + MLX5_INLINE_HSIZE_INNER_L4; > + goto exit; > + } > + } > + } > + /* > + * We get here if we are unable to deduce > + * inline data size with DevX. Try PCI ID > + * to determine old NICs. > + */ > + switch (spawn->pci_dev->id.device_id) { > + case PCI_DEVICE_ID_MELLANOX_CONNECTX4: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LX: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF: > + config->txq_inline_min =3D MLX5_INLINE_HSIZE_L2; > + config->hw_vlan_insert =3D 0; > + break; > + case PCI_DEVICE_ID_MELLANOX_CONNECTX5: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX5VF: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX5EX: > + case PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF: > + /* > + * These NICs support VLAN insertion from WQE and > + * report the wqe_vlan_insert flag. But there is the bug > + * and PFC control may be broken, so disable feature. > + */ > + config->hw_vlan_insert =3D 0; > + break; > + default: > + config->txq_inline_min =3D MLX5_INLINE_HSIZE_NONE; > + break; > + } > +exit: > + DRV_LOG(DEBUG, "min tx inline configured: %d", config->txq_inline_min); > +} > + > +/** > * Spawn an Ethernet device from Verbs information. > * > * @param dpdk_dev > @@ -1677,6 +1798,8 @@ struct mlx5_dev_spawn_data { > #else > config.dv_esw_en =3D 0; > #endif > + /* Detect minimal data bytes to inline. */ > + mlx5_set_min_inline(spawn, &config); > /* Store device configuration on private structure. */ > priv->config =3D config; > if (config.dv_flow_en) { > diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h > index 6ea9d4e..7aad94d 100644 > --- a/drivers/net/mlx5/mlx5.h > +++ b/drivers/net/mlx5/mlx5.h > @@ -194,6 +194,7 @@ struct mlx5_hca_attr { > struct mlx5_dev_config { > unsigned int hw_csum:1; /* Checksum offload is supported. */ > unsigned int hw_vlan_strip:1; /* VLAN stripping is supported. */ > + unsigned int hw_vlan_insert:1; /* VLAN insertion in WQE is supported. *= / > unsigned int hw_fcs_strip:1; /* FCS stripping is supported. */ > unsigned int hw_padding:1; /* End alignment padding is supported. */ > unsigned int vf:1; /* This is a VF. */ > @@ -203,7 +204,6 @@ struct mlx5_dev_config { > unsigned int cqe_comp:1; /* CQE compression is enabled. */ > unsigned int cqe_pad:1; /* CQE padding is enabled. */ > unsigned int tso:1; /* Whether TSO is supported. */ > - unsigned int tx_inline:1; /* Engage TX data inlining. */ > unsigned int rx_vec_en:1; /* Rx vector is enabled. */ > unsigned int mr_ext_memseg_en:1; > /* Whether memseg should be extended for MR creation. */ > diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c > index f2d6918..13f9431 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.c > +++ b/drivers/net/mlx5/mlx5_rxtx.c > @@ -507,7 +507,7 @@ > MKSTR(err_str, "Unexpected CQE error syndrome " > "0x%02x CQN =3D %u SQN =3D %u wqe_counter =3D %u " > "wq_ci =3D %u cq_ci =3D %u", err_cqe->syndrome, > - txq_ctrl->cqn, txq->qp_num_8s >> 8, > + txq->cqe_s, txq->qp_num_8s >> 8, > rte_be_to_cpu_16(err_cqe->wqe_counter), > txq->wqe_ci, txq->cq_ci); > MKSTR(name, "dpdk_mlx5_port_%u_txq_%u_index_%u_%u", > @@ -516,7 +516,7 @@ > mlx5_dump_debug_information(name, NULL, err_str, 0); > mlx5_dump_debug_information(name, "MLX5 Error CQ:", > (const void *)((uintptr_t) > - &(*txq->cqes)[0]), > + txq->cqes), > sizeof(*err_cqe) * > (1 << txq->cqe_n)); > mlx5_dump_debug_information(name, "MLX5 Error SQ:", > diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h > index acde09d..03ddd9e 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.h > +++ b/drivers/net/mlx5/mlx5_rxtx.h > @@ -188,37 +188,61 @@ struct mlx5_hrxq { > uint8_t rss_key[]; /* Hash key. */ > }; >=20 > +/* TX queue send local data. */ > +__extension__ > +struct mlx5_txq_local { > + struct mlx5_wqe *wqe_last; /* last sent WQE pointer. */ > + struct rte_mbuf *mbuf; /* first mbuf to process. */ > + uint16_t pkts_copy; /* packets copied to elts. */ > + uint16_t pkts_sent; /* packets sent. */ > + uint16_t elts_free; /* available elts remain. */ > + uint16_t wqe_free; /* available wqe remain. */ > + uint16_t mbuf_off; /* data offset in current mbuf. */ > + uint16_t mbuf_nseg; /* number of remaining mbuf. */ > +}; > + > /* TX queue descriptor. */ > __extension__ > struct mlx5_txq_data { > uint16_t elts_head; /* Current counter in (*elts)[]. */ > uint16_t elts_tail; /* Counter of first element awaiting completion. */ > - uint16_t elts_comp; /* Counter since last completion request. */ > - uint16_t mpw_comp; /* WQ index since last completion request. */ > + uint16_t elts_comp; /* elts index since last completion request. */ > + uint16_t elts_s; /* Number of mbuf elements. */ > + uint16_t elts_m; /* Mask for mbuf elements indices. */ > + /* Fields related to elts mbuf storage. */ > + uint16_t wqe_ci; /* Consumer index for work queue. */ > + uint16_t wqe_pi; /* Producer index for work queue. */ > + uint16_t wqe_s; /* Number of WQ elements. */ > + uint16_t wqe_m; /* Mask Number for WQ elements. */ > + uint16_t wqe_comp; /* WQE index since last completion request. */ > + uint16_t wqe_thres; /* WQE threshold to request completion in CQ. */ > + /* WQ related fields. */ > uint16_t cq_ci; /* Consumer index for completion queue. */ > #ifndef NDEBUG > - uint16_t cq_pi; /* Producer index for completion queue. */ > + uint16_t cq_pi; /* Counter of issued CQE "always" requests. */ > #endif > - uint16_t wqe_ci; /* Consumer index for work queue. */ > - uint16_t wqe_pi; /* Producer index for work queue. */ > - uint16_t elts_n:4; /* (*elts)[] length (in log2). */ > + uint16_t cqe_s; /* Number of CQ elements. */ > + uint16_t cqe_m; /* Mask for CQ indices. */ > + /* CQ related fields. */ > + uint16_t elts_n:4; /* elts[] length (in log2). */ > uint16_t cqe_n:4; /* Number of CQ elements (in log2). */ > - uint16_t wqe_n:4; /* Number of of WQ elements (in log2). */ > + uint16_t wqe_n:4; /* Number of WQ elements (in log2). */ > uint16_t tso_en:1; /* When set hardware TSO is enabled. */ > uint16_t tunnel_en:1; > /* When set TX offload for tunneled packets are supported. */ > uint16_t swp_en:1; /* Whether SW parser is enabled. */ > - uint16_t mpw_hdr_dseg:1; /* Enable DSEGs in the title WQEBB. */ > - uint16_t max_inline; /* Multiple of RTE_CACHE_LINE_SIZE to inline. */ > - uint16_t inline_max_packet_sz; /* Max packet size for inlining. */ > + uint16_t vlan_en:1; /* VLAN insertion in WQE is supported. */ > + uint16_t inlen_send; /* Ordinary send data inline size. */ > + uint16_t inlen_empw; /* eMPW max packet size to inline. */ > + uint16_t inlen_mode; /* Minimal data length to inline. */ > uint32_t qp_num_8s; /* QP number shifted by 8. */ > uint64_t offloads; /* Offloads for Tx Queue. */ > struct mlx5_mr_ctrl mr_ctrl; /* MR control descriptor. */ > - volatile struct mlx5_cqe (*cqes)[]; /* Completion queue. */ > - volatile void *wqes; /* Work queue (use volatile to write into). */ > + struct mlx5_wqe *wqes; /* Work queue. */ > + struct mlx5_wqe *wqes_end; /* Work queue array limit. */ > + volatile struct mlx5_cqe *cqes; /* Completion queue. */ > volatile uint32_t *qp_db; /* Work queue doorbell. */ > volatile uint32_t *cq_db; /* Completion queue doorbell. */ > - struct rte_mbuf *(*elts)[]; /* TX elements. */ > uint16_t port_id; /* Port ID of device. */ > uint16_t idx; /* Queue index. */ > struct mlx5_txq_stats stats; /* TX queue counters. */ > @@ -226,6 +250,8 @@ struct mlx5_txq_data { > rte_spinlock_t *uar_lock; > /* UAR access lock required for 32bit implementations */ > #endif > + struct rte_mbuf *elts[0]; > + /* Storage for queued packets, must be the last field. */ > } __rte_cache_aligned; >=20 > /* Verbs Rx queue elements. */ > @@ -239,7 +265,6 @@ struct mlx5_txq_ibv { >=20 > /* TX queue control descriptor. */ > struct mlx5_txq_ctrl { > - struct mlx5_txq_data txq; /* Data path structure. */ > LIST_ENTRY(mlx5_txq_ctrl) next; /* Pointer to the next element. */ > rte_atomic32_t refcnt; /* Reference counter. */ > unsigned int socket; /* CPU socket ID for allocations. */ > @@ -249,8 +274,9 @@ struct mlx5_txq_ctrl { > struct mlx5_priv *priv; /* Back pointer to private data. */ > off_t uar_mmap_offset; /* UAR mmap offset for non-primary process. */ > void *bf_reg; /* BlueFlame register from Verbs. */ > - uint32_t cqn; /* CQ number. */ > uint16_t dump_file_n; /* Number of dump files. */ > + struct mlx5_txq_data txq; /* Data path structure. */ > + /* Must be the last field in the structure, contains elts[]. */ > }; >=20 > #define MLX5_TX_BFREG(txq) \ > diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c > index 55892e2..2f3aa5b 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -47,7 +47,7 @@ > unsigned int i; >=20 > for (i =3D 0; (i !=3D elts_n); ++i) > - (*txq_ctrl->txq.elts)[i] =3D NULL; > + txq_ctrl->txq.elts[i] =3D NULL; > DRV_LOG(DEBUG, "port %u Tx queue %u allocated and configured %u WRs", > PORT_ID(txq_ctrl->priv), txq_ctrl->txq.idx, elts_n); > txq_ctrl->txq.elts_head =3D 0; > @@ -68,7 +68,7 @@ > const uint16_t elts_m =3D elts_n - 1; > uint16_t elts_head =3D txq_ctrl->txq.elts_head; > uint16_t elts_tail =3D txq_ctrl->txq.elts_tail; > - struct rte_mbuf *(*elts)[elts_n] =3D txq_ctrl->txq.elts; > + struct rte_mbuf *(*elts)[elts_n] =3D &txq_ctrl->txq.elts; >=20 > DRV_LOG(DEBUG, "port %u Tx queue %u freeing WRs", > PORT_ID(txq_ctrl->priv), txq_ctrl->txq.idx); > @@ -411,7 +411,8 @@ struct mlx5_txq_ibv * > attr.cq =3D (struct ibv_cq_init_attr_ex){ > .comp_mask =3D 0, > }; > - cqe_n =3D desc / MLX5_TX_COMP_THRESH + 1; > + cqe_n =3D desc / MLX5_TX_COMP_THRESH + > + 1 + MLX5_TX_COMP_THRESH_INLINE_DIV; > tmpl.cq =3D mlx5_glue->create_cq(priv->sh->ctx, cqe_n, NULL, NULL, 0); > if (tmpl.cq =3D=3D NULL) { > DRV_LOG(ERR, "port %u Tx queue %u CQ creation failure", > @@ -449,7 +450,7 @@ struct mlx5_txq_ibv * > .pd =3D priv->sh->pd, > .comp_mask =3D IBV_QP_INIT_ATTR_PD, > }; > - if (txq_data->max_inline) > + if (txq_data->inlen_send) > attr.init.cap.max_inline_data =3D txq_ctrl->max_inline_data; > if (txq_data->tso_en) { > attr.init.max_tso_header =3D txq_ctrl->max_tso_header; > @@ -523,25 +524,29 @@ struct mlx5_txq_ibv * > goto error; > } > txq_data->cqe_n =3D log2above(cq_info.cqe_cnt); > + txq_data->cqe_s =3D 1 << txq_data->cqe_n; > + txq_data->cqe_m =3D txq_data->cqe_s - 1; > txq_data->qp_num_8s =3D tmpl.qp->qp_num << 8; > txq_data->wqes =3D qp.sq.buf; > txq_data->wqe_n =3D log2above(qp.sq.wqe_cnt); > + txq_data->wqe_s =3D 1 << txq_data->wqe_n; > + txq_data->wqe_m =3D txq_data->wqe_s - 1; > + txq_data->wqes_end =3D txq_data->wqes + txq_data->wqe_s; > txq_data->qp_db =3D &qp.dbrec[MLX5_SND_DBR]; > txq_data->cq_db =3D cq_info.dbrec; > - txq_data->cqes =3D > - (volatile struct mlx5_cqe (*)[]) > - (uintptr_t)cq_info.buf; > + txq_data->cqes =3D (volatile struct mlx5_cqe *)cq_info.buf; > txq_data->cq_ci =3D 0; > #ifndef NDEBUG > txq_data->cq_pi =3D 0; > #endif > txq_data->wqe_ci =3D 0; > txq_data->wqe_pi =3D 0; > + txq_data->wqe_comp =3D 0; > + txq_data->wqe_thres =3D txq_data->wqe_s / MLX5_TX_COMP_THRESH_INLINE_DI= V; > txq_ibv->qp =3D tmpl.qp; > txq_ibv->cq =3D tmpl.cq; > rte_atomic32_inc(&txq_ibv->refcnt); > txq_ctrl->bf_reg =3D qp.bf.reg; > - txq_ctrl->cqn =3D cq_info.cqn; > txq_uar_init(txq_ctrl); > if (qp.comp_mask & MLX5DV_QP_MASK_UAR_MMAP_OFFSET) { > txq_ctrl->uar_mmap_offset =3D qp.uar_mmap_offset; > @@ -663,7 +668,11 @@ struct mlx5_txq_ibv * > unsigned int wqe_size; > const unsigned int desc =3D 1 << txq_ctrl->txq.elts_n; >=20 > - wqe_size =3D MLX5_WQE_SIZE + txq_ctrl->max_inline_data; > + wqe_size =3D MLX5_WQE_CSEG_SIZE + > + MLX5_WQE_ESEG_SIZE + > + MLX5_WSEG_SIZE - > + MLX5_ESEG_MIN_INLINE_SIZE + > + txq_ctrl->max_inline_data; > return rte_align32pow2(wqe_size * desc) / MLX5_WQE_SIZE; > } >=20 > @@ -676,7 +685,189 @@ struct mlx5_txq_ibv * > static void > txq_set_params(struct mlx5_txq_ctrl *txq_ctrl) > { > - (void)txq_ctrl; > + struct mlx5_priv *priv =3D txq_ctrl->priv; > + struct mlx5_dev_config *config =3D &priv->config; > + unsigned int inlen_send; /* Inline data for ordinary SEND.*/ > + unsigned int inlen_empw; /* Inline data for enhanced MPW. */ > + unsigned int inlen_mode; /* Minimal required Inline data. */ > + unsigned int txqs_inline; /* Min Tx queues to enable inline. */ > + uint64_t dev_txoff =3D priv->dev_data->dev_conf.txmode.offloads; > + bool tso =3D txq_ctrl->txq.offloads & (DEV_TX_OFFLOAD_TCP_TSO | > + DEV_TX_OFFLOAD_VXLAN_TNL_TSO | > + DEV_TX_OFFLOAD_GRE_TNL_TSO | > + DEV_TX_OFFLOAD_IP_TNL_TSO | > + DEV_TX_OFFLOAD_UDP_TNL_TSO); > + bool vlan_inline; > + unsigned int temp; > + > + if (config->txqs_inline =3D=3D MLX5_ARG_UNSET) > + txqs_inline =3D > +#if defined(RTE_ARCH_ARM64) > + (priv->sh->pci_dev->id.device_id =3D=3D > + PCI_DEVICE_ID_MELLANOX_CONNECTX5BF) ? > + MLX5_INLINE_MAX_TXQS_BLUEFIELD : > +#endif > + MLX5_INLINE_MAX_TXQS; > + else > + txqs_inline =3D (unsigned int)config->txqs_inline; > + inlen_send =3D (config->txq_inline_max =3D=3D MLX5_ARG_UNSET) ? > + MLX5_SEND_DEF_INLINE_LEN : > + (unsigned int)config->txq_inline_max; > + inlen_empw =3D (config->txq_inline_mpw =3D=3D MLX5_ARG_UNSET) ? > + MLX5_EMPW_DEF_INLINE_LEN : > + (unsigned int)config->txq_inline_mpw; > + inlen_mode =3D (config->txq_inline_min =3D=3D MLX5_ARG_UNSET) ? > + 0 : (unsigned int)config->txq_inline_min; > + if (config->mps !=3D MLX5_MPW_ENHANCED) > + inlen_empw =3D 0; > + /* > + * If there is requested minimal amount of data to inline > + * we MUST enable inlining. This is a case for ConnectX-4 > + * which usually requires L2 inlined for correct operating > + * and ConnectX-4LX which requires L2-L4 inlined to > + * support E-Switch Flows. > + */ > + if (inlen_mode) { > + if (inlen_mode <=3D MLX5_ESEG_MIN_INLINE_SIZE) { > + /* > + * Optimize minimal inlining for single > + * segment packets to fill one WQEBB > + * without gaps. > + */ > + temp =3D MLX5_ESEG_MIN_INLINE_SIZE; > + } else { > + temp =3D inlen_mode - MLX5_ESEG_MIN_INLINE_SIZE; > + temp =3D RTE_ALIGN(temp, MLX5_WSEG_SIZE) + > + MLX5_ESEG_MIN_INLINE_SIZE; > + temp =3D RTE_MIN(temp, MLX5_SEND_MAX_INLINE_LEN); > + } > + if (temp !=3D inlen_mode) { > + DRV_LOG(INFO, > + "port %u minimal required inline setting" > + " aligned from %u to %u", > + PORT_ID(priv), inlen_mode, temp); > + inlen_mode =3D temp; > + } > + } > + /* > + * If port is configured to support VLAN insertion and device > + * does not support this feature by HW (for NICs before ConnectX-5 > + * or in case of wqe_vlan_insert flag is not set) we must enable > + * data inline on all queues because it is supported by single > + * tx_burst routine. > + */ > + txq_ctrl->txq.vlan_en =3D config->hw_vlan_insert; > + vlan_inline =3D (dev_txoff & DEV_TX_OFFLOAD_VLAN_INSERT) && > + !config->hw_vlan_insert; > + if (vlan_inline) > + inlen_send =3D RTE_MAX(inlen_send, MLX5_ESEG_MIN_INLINE_SIZE); > + /* > + * If there are few Tx queues it is prioritized > + * to save CPU cycles and disable data inlining at all. > + */ > + if ((inlen_send && priv->txqs_n >=3D txqs_inline) || vlan_inline) { > + /* > + * The data sent with ordinal MLX5_OPCODE_SEND > + * may be inlined in Ethernet Segment, align the > + * length accordingly to fit entire WQEBBs. > + */ > + temp =3D (inlen_send / MLX5_WQE_SIZE) * MLX5_WQE_SIZE + > + MLX5_ESEG_MIN_INLINE_SIZE + MLX5_WQE_DSEG_SIZE; > + temp =3D RTE_MIN(temp, MLX5_WQE_SIZE_MAX + > + MLX5_ESEG_MIN_INLINE_SIZE - > + MLX5_WQE_CSEG_SIZE - > + MLX5_WQE_ESEG_SIZE - > + MLX5_WQE_DSEG_SIZE * 2); > + temp =3D RTE_MIN(temp, MLX5_SEND_MAX_INLINE_LEN); > + temp =3D RTE_MAX(temp, inlen_mode); > + if (temp !=3D inlen_send) { > + DRV_LOG(INFO, > + "port %u ordinary send inline setting" > + " aligned from %u to %u", > + PORT_ID(priv), inlen_send, temp); > + inlen_send =3D temp; > + } > + /* > + * Not aligned to cache lines, but to WQEs. > + * First bytes of data (initial alignment) > + * is going to be copied explicitly at the > + * beginning of inlining buffer in Ethernet > + * Segment. > + */ > + assert(inlen_send >=3D MLX5_ESEG_MIN_INLINE_SIZE); > + assert(inlen_send <=3D MLX5_WQE_SIZE_MAX + > + MLX5_ESEG_MIN_INLINE_SIZE - > + MLX5_WQE_CSEG_SIZE - > + MLX5_WQE_ESEG_SIZE - > + MLX5_WQE_DSEG_SIZE * 2); > + txq_ctrl->txq.inlen_send =3D inlen_send; > + txq_ctrl->txq.inlen_mode =3D inlen_mode; > + txq_ctrl->txq.inlen_empw =3D 0; > + } else { > + /* > + * If minimal inlining is requested we must > + * enable inlining in general, despite the > + * number of configured queues. > + */ > + inlen_send =3D inlen_mode; > + if (inlen_mode) { > + /* > + * Extend space for inline data to allow > + * optional alignment of data buffer > + * start address, it may improve PCIe > + * performance. > + */ > + inlen_send =3D RTE_MIN(inlen_send + MLX5_WQE_SIZE, > + MLX5_SEND_MAX_INLINE_LEN); > + } > + txq_ctrl->txq.inlen_send =3D inlen_send; > + txq_ctrl->txq.inlen_mode =3D inlen_mode; > + txq_ctrl->txq.inlen_empw =3D 0; > + inlen_send =3D 0; > + inlen_empw =3D 0; > + } > + if (inlen_send && inlen_empw && priv->txqs_n >=3D txqs_inline) { > + /* > + * The data sent with MLX5_OPCODE_ENHANCED_MPSW > + * may be inlined in Data Segment, align the > + * length accordingly to fit entire WQEBBs. > + */ > + temp =3D (inlen_empw + MLX5_WQE_SIZE - 1) / MLX5_WQE_SIZE; > + temp =3D temp * MLX5_WQE_SIZE + > + MLX5_DSEG_MIN_INLINE_SIZE - MLX5_WQE_DSEG_SIZE; > + temp =3D RTE_MIN(temp, MLX5_WQE_SIZE_MAX + > + MLX5_DSEG_MIN_INLINE_SIZE - > + MLX5_WQE_CSEG_SIZE - > + MLX5_WQE_ESEG_SIZE - > + MLX5_WQE_DSEG_SIZE); > + temp =3D RTE_MIN(temp, MLX5_EMPW_MAX_INLINE_LEN); > + if (temp !=3D inlen_empw) { > + DRV_LOG(INFO, > + "port %u enhanced empw inline setting" > + " aligned from %u to %u", > + PORT_ID(priv), inlen_empw, temp); > + inlen_empw =3D temp; > + } > + assert(inlen_empw >=3D MLX5_ESEG_MIN_INLINE_SIZE); > + assert(inlen_empw <=3D MLX5_WQE_SIZE_MAX + > + MLX5_DSEG_MIN_INLINE_SIZE - > + MLX5_WQE_CSEG_SIZE - > + MLX5_WQE_ESEG_SIZE - > + MLX5_WQE_DSEG_SIZE); > + txq_ctrl->txq.inlen_empw =3D inlen_empw; > + } > + txq_ctrl->max_inline_data =3D RTE_MAX(inlen_send, inlen_empw); > + if (tso) { > + txq_ctrl->max_tso_header =3D MLX5_MAX_TSO_HEADER; > + txq_ctrl->max_inline_data =3D RTE_MAX(txq_ctrl->max_inline_data, > + MLX5_MAX_TSO_HEADER); > + txq_ctrl->txq.tso_en =3D 1; > + } > + txq_ctrl->txq.tunnel_en =3D config->tunnel_en | config->swp; > + txq_ctrl->txq.swp_en =3D ((DEV_TX_OFFLOAD_IP_TNL_TSO | > + DEV_TX_OFFLOAD_UDP_TNL_TSO | > + DEV_TX_OFFLOAD_OUTER_IPV4_CKSUM) & > + txq_ctrl->txq.offloads) && config->swp; > } >=20 > /** > @@ -724,6 +915,8 @@ struct mlx5_txq_ctrl * > tmpl->priv =3D priv; > tmpl->socket =3D socket; > tmpl->txq.elts_n =3D log2above(desc); > + tmpl->txq.elts_s =3D desc; > + tmpl->txq.elts_m =3D desc - 1; > tmpl->txq.port_id =3D dev->data->port_id; > tmpl->txq.idx =3D idx; > txq_set_params(tmpl); > @@ -737,8 +930,6 @@ struct mlx5_txq_ctrl * > rte_errno =3D ENOMEM; > goto error; > } > - tmpl->txq.elts =3D > - (struct rte_mbuf *(*)[1 << tmpl->txq.elts_n])(tmpl + 1); > rte_atomic32_inc(&tmpl->refcnt); > LIST_INSERT_HEAD(&priv->txqsctrl, tmpl, next); > return tmpl; > --=20 > 1.8.3.1 >=20