From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 88683A0562; Tue, 31 Mar 2020 10:20:51 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 4483F1C033; Tue, 31 Mar 2020 10:20:50 +0200 (CEST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140045.outbound.protection.outlook.com [40.107.14.45]) by dpdk.org (Postfix) with ESMTP id F39281BFC4 for ; Tue, 31 Mar 2020 10:20:48 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K/lFEuyy7DOvDPBy8/FpP1mMsQa3MY1zhWdsMYFUmLHus7161USVzqkvvFDftx3zuJHVL4k4QDsqsydjwpzDeyaq1Z3ZIKhmzRONgsr49IYlRdK7rHzUJBxj4bBqEM1MSSiXqnWevVjoHpnUMPktD15tnvrC5u+YVXk4O1rHQIoUg+dtp+tXsAbpag6OU7ifnj1hA4BhH5hHv66L4bYC2aGLiVqgYjXBIvaZBn4rWEfPx84ecrfAGEaUz3zlOanCUg6shxJrZsZRY/3IUBJgNLiObgmtVCzkm2PDG+0n5k4PvFkjDqHybwfhzSWchE0vBQcNHgfIFNmWxhppCOPOKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lbZS4XYeRecl2hUh2XHRUY74FLVWdnWF7hhjW0tl5PA=; b=b08c5eX8aEkU5yQzpG66tte4mBeHe8bNYG42fSU3jqjQoz8LpF8NfXPwLDowhA4Q70jS14GQsemzOuxIwUfp/FGOmqgkUKNcPBTti61Hrx1bAzOGlnMFbBi2lpjMcuYm2uVr65mUtQdAttLXVs6gl+QKZ8zdlEBkibeoMJ6muG9ChRJt/jGenggfQg6ZhyahhcxmRYVVykAn+noOAksJyB64AQohA7SsTnc7pQobAs48vtTejeAVC1lQgt6k0cLDuJWR6WgGcBn1W2Lht1VKKDWOZsyAN/enlrtMfxhK4KaK7eRT92/MugiCt97J5DAI/CQ6cyBEF1ikxIjz7t64iw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lbZS4XYeRecl2hUh2XHRUY74FLVWdnWF7hhjW0tl5PA=; b=b3ZH2OzLNpWQew/WSSuBTB/lKnov2NyLUXCPU3Dqn3qlt9cEzoPzI1+HAmG71liycqio6QAtFimwLal4ilXdQODaFjeY+nL9Gv/WLUi7Sk3SiSO4xxmZh3xq3X5rADFPD/GtX0+TvXUt1ohCZObn7KUYj6ijMLeg+6Anvmrtelo= Received: from AM4PR05MB3265.eurprd05.prod.outlook.com (10.171.188.154) by AM4PR05MB3362.eurprd05.prod.outlook.com (10.171.187.23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2856.20; Tue, 31 Mar 2020 08:20:23 +0000 Received: from AM4PR05MB3265.eurprd05.prod.outlook.com ([fe80::da5:7919:35c1:894]) by AM4PR05MB3265.eurprd05.prod.outlook.com ([fe80::da5:7919:35c1:894%6]) with mapi id 15.20.2856.019; Tue, 31 Mar 2020 08:20:23 +0000 From: Slava Ovsiienko To: Bing Zhao , Ori Kam , Matan Azrad CC: Raslan Darawsheh , "dev@dpdk.org" Thread-Topic: [PATCH] net/mlx5: introduce dev parameter for hairpin Thread-Index: AQHWAdwGLHXoHW4KcEuQmAC4/pyFdahiZqGw Date: Tue, 31 Mar 2020 08:20:23 +0000 Message-ID: References: <1585054741-233903-1-git-send-email-bingz@mellanox.com> In-Reply-To: <1585054741-233903-1-git-send-email-bingz@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=viacheslavo@mellanox.com; x-originating-ip: [95.164.10.10] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 2db47621-224c-4285-f9b7-08d7d54c5c23 x-ms-traffictypediagnostic: AM4PR05MB3362:|AM4PR05MB3362: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtFwd x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:2089; x-forefront-prvs: 0359162B6D x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM4PR05MB3265.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(136003)(396003)(366004)(39860400002)(346002)(376002)(66556008)(52536014)(316002)(53546011)(76116006)(66446008)(66946007)(64756008)(110136005)(66476007)(33656002)(478600001)(4326008)(8936002)(54906003)(26005)(55016002)(81156014)(6506007)(7696005)(9686003)(8676002)(5660300002)(186003)(2906002)(81166006)(86362001)(6636002)(71200400001); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: y37ECKw8mApN+xvEZpR0DLVAD7AmhIeZA3HzVjGnaJP+SzAwt5VDEH+ELwSRwxOyq2RYKoX6+zs2gdWGmIzcht35oORrWaAIixPaeKpl+JsWF9RXXSlDT08SUI2WKJWhHOV3Wi4vGEPiVmfiP6cNb/fEXVlyNxfkHMbUrEDn1FByLL5NDlX+11Czo47Xe4Lgtt9tTYVF2Fm1J03Hz7lQXX4UcFTlBubp4w0BE6yMBRfNuvdGzmb3J/7yMwLGJCOGXnuXvMBzKf21oSsFYY3lOEBHsRl2Lexi0V+4QG1bpFAslzwQ7Aiqo2GXaO4CjmFzF4xPJgjmC6HTwfqhS7WVgCoJCyIAeASKbh+igmYMYt8h1xVqdu+d4MvLXApO5O9PRplSAev4QM5AYW0X8CJAOuIYENtlZAlsSZQ9Xo/fmxoZdtPK+xoMyNxObDvUjW7N x-ms-exchange-antispam-messagedata: O0NttrMxZ7YGbLz2sQGZbeVUJE1UG5YxowTtGsGaeZbffgB2EGqVdFBEGR8P8kxO9l+x2nqpSMEni92V6mjTnQCf4MjShCdF1qBFCmM9a7j8V0hntPUTRR/500WGIFqLvoN21UKxETvqvVQjADCwKA== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2db47621-224c-4285-f9b7-08d7d54c5c23 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Mar 2020 08:20:23.4867 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 5sfcSklIeSnLis22a1vOepWju2wbIZVZv8+Mvgp1uxnmc1/ZpJsLmvhx+fkc+X78BQTI5r8aTmmmZhNaeEK7VM4VNis4qYrWGgcFHyXhBQg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR05MB3362 Subject: Re: [dpdk-dev] [PATCH] net/mlx5: introduce dev parameter for hairpin X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Bing Zhao > Sent: Tuesday, March 24, 2020 14:59 > To: Ori Kam ; Matan Azrad > Cc: Raslan Darawsheh ; Slava Ovsiienko > ; dev@dpdk.org > Subject: [PATCH] net/mlx5: introduce dev parameter for hairpin >=20 > When creating a hairpin queue, the total data size and the maximal number > of packets are interrelated. The differ is the stride size. > Larger buffer size means big packet like jumbo could be supported, but in= the > meanwhile, it will introduce more cache misses and have a side effect on = the > performance. > Now a new device parameter "hp_buf_log_sz" is introduced for applications > to set the total data buffer size (the logarithm value). > Then the maximal number of packets will also be calculated automaticlly b= y > this value. > Applications could also change this value to a larger one in order to sup= port > larger packets in hairpin case. A smaller value will be beneficial for me= mory > consumption. > If it is not set, the default value will be used. >=20 > Signed-off-by: Bing Zhao Acked-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/mlx5.c | 10 ++++++++++ > drivers/net/mlx5/mlx5.h | 1 + > drivers/net/mlx5/mlx5_defs.h | 2 +- > drivers/net/mlx5/mlx5_rxq.c | 17 ++++++++++++++--- > drivers/net/mlx5/mlx5_txq.c | 17 ++++++++++++++--- > 5 files changed, 40 insertions(+), 7 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > 94aaa60..3e6eaa0 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -150,6 +150,12 @@ > /* Configure timeout of LRO session (in microseconds). */ #define > MLX5_LRO_TIMEOUT_USEC "lro_timeout_usec" >=20 > +/* > + * Device parameter to configure the total data buffer size for a > +single > + * hairpin queue (logarithm value). > + */ > +#define MLX5_HP_BUF_SIZE "hp_buf_log_sz" > + > #ifndef HAVE_IBV_MLX5_MOD_MPW > #define MLX5DV_CONTEXT_FLAGS_MPW_ALLOWED (1 << 2) #define > MLX5DV_CONTEXT_FLAGS_ENHANCED_MPW (1 << 3) @@ -1580,6 +1586,8 > @@ struct mlx5_flow_id_pool * > config->lro.timeout =3D tmp; > } else if (strcmp(MLX5_CLASS_ARG_NAME, key) =3D=3D 0) { > DRV_LOG(DEBUG, "class argument is %s.", val); > + } else if (strcmp(MLX5_HP_BUF_SIZE, key) =3D=3D 0) { > + config->log_hp_size =3D tmp; > } else { > DRV_LOG(WARNING, "%s: unknown parameter", key); > rte_errno =3D EINVAL; > @@ -1632,6 +1640,7 @@ struct mlx5_flow_id_pool * > MLX5_MAX_DUMP_FILES_NUM, > MLX5_LRO_TIMEOUT_USEC, > MLX5_CLASS_ARG_NAME, > + MLX5_HP_BUF_SIZE, > NULL, > }; > struct rte_kvargs *kvlist; > @@ -3342,6 +3351,7 @@ struct mlx5_flow_id_pool * > }, > .dv_esw_en =3D 1, > .dv_flow_en =3D 1, > + .log_hp_size =3D MLX5_ARG_UNSET, > }; > /* Device specific configuration. */ > switch (pci_dev->id.device_id) { > diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index > d7c519b..53f35fd 100644 > --- a/drivers/net/mlx5/mlx5.h > +++ b/drivers/net/mlx5/mlx5.h > @@ -191,6 +191,7 @@ struct mlx5_dev_config { > unsigned int tso_max_payload_sz; /* Maximum TCP payload for TSO. > */ > unsigned int ind_table_max_size; /* Maximum indirection table size. > */ > unsigned int max_dump_files_num; /* Maximum dump files per > queue. */ > + unsigned int log_hp_size; /* Single hairpin queue data size in total. > +*/ > int txqs_inline; /* Queue number threshold for inlining. */ > int txq_inline_min; /* Minimal amount of data bytes to inline. */ > int txq_inline_max; /* Max packet size for inlining with SEND. */ diff = -- > git a/drivers/net/mlx5/mlx5_defs.h b/drivers/net/mlx5/mlx5_defs.h index > 83ca367..19e8253 100644 > --- a/drivers/net/mlx5/mlx5_defs.h > +++ b/drivers/net/mlx5/mlx5_defs.h > @@ -175,7 +175,7 @@ >=20 > /* Hairpin TX/RX queue configuration parameters. */ #define > MLX5_HAIRPIN_QUEUE_STRIDE 6 -#define MLX5_HAIRPIN_JUMBO_LOG_SIZE > (15 + 2) > +#define MLX5_HAIRPIN_JUMBO_LOG_SIZE (14 + 2) >=20 > /* Definition of static_assert found in /usr/include/assert.h */ #ifnde= f > HAVE_STATIC_ASSERT diff --git a/drivers/net/mlx5/mlx5_rxq.c > b/drivers/net/mlx5/mlx5_rxq.c index 8a6b410..3fdb632 100644 > --- a/drivers/net/mlx5/mlx5_rxq.c > +++ b/drivers/net/mlx5/mlx5_rxq.c > @@ -1286,9 +1286,20 @@ > attr.hairpin =3D 1; > max_wq_data =3D priv->config.hca_attr.log_max_hairpin_wq_data_sz; > /* Jumbo frames > 9KB should be supported, and more packets. */ > - attr.wq_attr.log_hairpin_data_sz =3D > - (max_wq_data < MLX5_HAIRPIN_JUMBO_LOG_SIZE) ? > - max_wq_data : MLX5_HAIRPIN_JUMBO_LOG_SIZE; > + if (priv->config.log_hp_size !=3D (uint32_t)MLX5_ARG_UNSET) { > + if (priv->config.log_hp_size > max_wq_data) { > + DRV_LOG(ERR, "total data size %u power of 2 is " > + "too large for hairpin", > + priv->config.log_hp_size); > + rte_errno =3D ERANGE; > + return NULL; > + } > + attr.wq_attr.log_hairpin_data_sz =3D priv->config.log_hp_size; > + } else { > + attr.wq_attr.log_hairpin_data_sz =3D > + (max_wq_data < > MLX5_HAIRPIN_JUMBO_LOG_SIZE) ? > + max_wq_data : > MLX5_HAIRPIN_JUMBO_LOG_SIZE; > + } > /* Set the packets number to the maximum value for performance. */ > attr.wq_attr.log_hairpin_num_packets =3D > attr.wq_attr.log_hairpin_data_sz - > diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c in= dex > 57bc116..0653f4c 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -512,9 +512,20 @@ > attr.tis_lst_sz =3D 1; > max_wq_data =3D priv->config.hca_attr.log_max_hairpin_wq_data_sz; > /* Jumbo frames > 9KB should be supported, and more packets. */ > - attr.wq_attr.log_hairpin_data_sz =3D > - (max_wq_data < MLX5_HAIRPIN_JUMBO_LOG_SIZE) ? > - max_wq_data : MLX5_HAIRPIN_JUMBO_LOG_SIZE; > + if (priv->config.log_hp_size !=3D (uint32_t)MLX5_ARG_UNSET) { > + if (priv->config.log_hp_size > max_wq_data) { > + DRV_LOG(ERR, "total data size %u power of 2 is " > + "too large for hairpin", > + priv->config.log_hp_size); > + rte_errno =3D ERANGE; > + return NULL; > + } > + attr.wq_attr.log_hairpin_data_sz =3D priv->config.log_hp_size; > + } else { > + attr.wq_attr.log_hairpin_data_sz =3D > + (max_wq_data < > MLX5_HAIRPIN_JUMBO_LOG_SIZE) ? > + max_wq_data : > MLX5_HAIRPIN_JUMBO_LOG_SIZE; > + } > /* Set the packets number to the maximum value for performance. */ > attr.wq_attr.log_hairpin_num_packets =3D > attr.wq_attr.log_hairpin_data_sz - > -- > 1.8.3.1