From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 19DA4A0C56; Mon, 23 Aug 2021 10:44:30 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 33C0E4114C; Mon, 23 Aug 2021 10:44:28 +0200 (CEST) Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by mails.dpdk.org (Postfix) with ESMTP id E9F0840143 for ; Mon, 23 Aug 2021 10:44:26 +0200 (CEST) Received: by mail-pg1-f171.google.com with SMTP id q2so16014327pgt.6 for ; Mon, 23 Aug 2021 01:44:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vvdntech-in.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9reN6twwZYkdoe0c4aWA+7ciYvEJ3dxbXY+hEswu3co=; b=L+ODMIgjlNTrjT46bprxvuR0akaetJ+9i2I7VZ9aUyba2eYnV9k0OB9aCRtMd/tZUM o9k6+IIoyeQxrh6CGGyChWpslxPPEu+876yl6Oq6RD/CgrlAvP8Nox08tJE4pHh4PrZU Ce6z2PAQboOehEsoehbUChpKVkn1t15lCh9RiP+FkwUR99bb+qeChjxDKaIkqJVQousg ti6vlfVnq74wMADIdM74o9WLCdG3LESbb7nA6950W0so1grpGeOeO++Q8kWyJdw8rpzA Czwrlvp2Qq91hAwusmioB3M7Gs5FBK3UD2homxdm+Ax9bl5uv2R1X09DnrfooGOYPTAS J8qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9reN6twwZYkdoe0c4aWA+7ciYvEJ3dxbXY+hEswu3co=; b=A4NwRgoTOhLTeohlZghnhBKMMqxowCGo+81KbI9Ud3VHjISK/vwxEojbu1EO14rjq5 vtYogMXO8I+3Z8UZkO1qrGKAJCowA5+Vk0ugRSZOnrDYtXBnYgCj8WfPPtI17lU0pkLK 6su8lURqyrFIgzsRQD5QAwVYMWZq73PO7y/XrmBVEtMARDewUaUvAQaGTt2aw7LaBLuC jwlnRJUJPrsm33HJ+BC+jX5guSE7AwkyhKRkpAJo1BaH9KFncCvm2uuCZxng2QOxtia9 YhmbQw30bv6ZHl9GOzrkpjvv7nhoGZIoqI1Ruvr+T5DUX0sz1z+ZH9p+lbmLrI1X3ybH +Mnw== X-Gm-Message-State: AOAM530P68hGWhtaF2cWU4QoKIHuNoVz61glWKWXjYB921HNHZ5XlXzV bX7irkwZ+DYrJgLaDAuc/WRfTS20xc64lxV9 X-Google-Smtp-Source: ABdhPJw78cSTtYJ9EfX0qs69Hzlbk8bp1N16+ReHp/cyTihGbKI3YAS5rahEI1OdF0u187FlUuW22w== X-Received: by 2002:a63:6447:: with SMTP id y68mr31577984pgb.154.1629708265700; Mon, 23 Aug 2021 01:44:25 -0700 (PDT) Received: from 470--5GDC--BLR.blr5glab.net ([106.51.39.131]) by smtp.gmail.com with ESMTPSA id j4sm17566832pgi.6.2021.08.23.01.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Aug 2021 01:44:25 -0700 (PDT) From: Aman Kumar To: dev@dpdk.org Cc: rasland@nvidia.com, asafp@nvidia.com, shys@nvidia.com, viacheslavo@nvidia.com, akozyrev@nvidia.com, matan@nvidia.com, anatoly.burakov@intel.com, keesang.song@amd.com, aman.kumar@vvdntech.in Date: Mon, 23 Aug 2021 14:14:11 +0530 Message-Id: <20210823084411.29592-2-aman.kumar@vvdntech.in> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210823084411.29592-1-aman.kumar@vvdntech.in> References: <20210823084411.29592-1-aman.kumar@vvdntech.in> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH 2/2] net/mlx5: optimize mprq memcpy for AMD EPYC2 platforms X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" add non temporal load and temporal store for mprq memcpy. define mlx5_ntload_tstore in meson build configuration to enable this optimization. This utilizes AMD EPYC2 optimized rte_memcpy* routines. Signed-off-by: Aman Kumar --- drivers/net/mlx5/meson.build | 1 + drivers/net/mlx5/mlx5.c | 12 ++++++++++++ drivers/net/mlx5/mlx5.h | 3 +++ drivers/net/mlx5/mlx5_rx.h | 24 ++++++++++++++++++++++++ drivers/net/mlx5/mlx5_rxq.c | 4 ++++ meson_options.txt | 2 ++ 6 files changed, 46 insertions(+) diff --git a/drivers/net/mlx5/meson.build b/drivers/net/mlx5/meson.build index dac7f1fabf..0d2888742c 100644 --- a/drivers/net/mlx5/meson.build +++ b/drivers/net/mlx5/meson.build @@ -61,6 +61,7 @@ foreach option:cflags_options cflags += option endif endforeach +dpdk_conf.set('RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY', get_option('mlx5_ntload_tstore')) if get_option('buildtype').contains('debug') cflags += [ '-pedantic', '-DPEDANTIC' ] else diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index f84e061fe7..cf57867c25 100644 --- a/drivers/net/mlx5/mlx5.c +++ b/drivers/net/mlx5/mlx5.c @@ -161,6 +161,11 @@ /* Configure timeout of LRO session (in microseconds). */ #define MLX5_LRO_TIMEOUT_USEC "lro_timeout_usec" +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY +/* mprq_tstore_memcpy */ +#define MLX5_MPRQ_TSTORE_MEMCPY "mprq_tstore_memcpy" +#endif + /* * Device parameter to configure the total data buffer size for a single * hairpin queue (logarithm value). @@ -1991,6 +1996,10 @@ mlx5_args_check(const char *key, const char *val, void *opaque) config->decap_en = !!tmp; } else if (strcmp(MLX5_ALLOW_DUPLICATE_PATTERN, key) == 0) { config->allow_duplicate_pattern = !!tmp; +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + } else if (strcmp(MLX5_MPRQ_TSTORE_MEMCPY, key) == 0) { + config->mprq_tstore_memcpy = tmp; +#endif } else { DRV_LOG(WARNING, "%s: unknown parameter", key); rte_errno = EINVAL; @@ -2051,6 +2060,9 @@ mlx5_args(struct mlx5_dev_config *config, struct rte_devargs *devargs) MLX5_SYS_MEM_EN, MLX5_DECAP_EN, MLX5_ALLOW_DUPLICATE_PATTERN, +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + MLX5_MPRQ_TSTORE_MEMCPY, +#endif NULL, }; struct rte_kvargs *kvlist; diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index e02714e231..7d5617f5ca 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -298,6 +298,9 @@ struct mlx5_dev_config { int tx_skew; /* Tx scheduling skew between WQE and data on wire. */ struct mlx5_hca_attr hca_attr; /* HCA attributes. */ struct mlx5_lro_config lro; /* LRO configuration. */ +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + unsigned int mprq_tstore_memcpy:1; +#endif }; diff --git a/drivers/net/mlx5/mlx5_rx.h b/drivers/net/mlx5/mlx5_rx.h index 3f2b99fb65..19318bdd1b 100644 --- a/drivers/net/mlx5/mlx5_rx.h +++ b/drivers/net/mlx5/mlx5_rx.h @@ -148,6 +148,9 @@ struct mlx5_rxq_data { uint32_t rxseg_n; /* Number of split segment descriptions. */ struct mlx5_eth_rxseg rxseg[MLX5_MAX_RXQ_NSEG]; /* Buffer split segment descriptions - sizes, offsets, pools. */ +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + unsigned int mprq_tstore_memcpy:1; +#endif } __rte_cache_aligned; enum mlx5_rxq_type { @@ -422,6 +425,15 @@ mprq_buf_to_pkt(struct mlx5_rxq_data *rxq, struct rte_mbuf *pkt, uint32_t len, const uint32_t offset = strd_idx * strd_sz + strd_shift; void *addr = RTE_PTR_ADD(mlx5_mprq_buf_addr(buf, strd_n), offset); +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + if (unlikely(!rxq->mprq_tstore_memcpy) && + len <= rxq->mprq_max_memcpy_len) { + rte_prefetch1(addr); + if (len > RTE_CACHE_LINE_SIZE) + rte_prefetch2((void *)((uintptr_t)addr + + RTE_CACHE_LINE_SIZE)); + } +#endif /* * Memcpy packets to the target mbuf if: * - The size of packet is smaller than mprq_max_memcpy_len. @@ -433,8 +445,20 @@ mprq_buf_to_pkt(struct mlx5_rxq_data *rxq, struct rte_mbuf *pkt, uint32_t len, (hdrm_overlap > 0 && !rxq->strd_scatter_en)) { if (likely(len <= (uint32_t)(pkt->buf_len - RTE_PKTMBUF_HEADROOM))) { +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + uintptr_t data_addr; + + data_addr = (uintptr_t)rte_pktmbuf_mtod(pkt, void *); + if (!((data_addr | (uintptr_t)addr) & ALIGNMENT_MASK) && + rxq->mprq_tstore_memcpy) + rte_memcpy_aligned_tstore16((void *)data_addr, + addr, len); + else + rte_memcpy((void *)data_addr, addr, len); +#else rte_memcpy(rte_pktmbuf_mtod(pkt, void *), addr, len); +#endif DATA_LEN(pkt) = len; } else if (rxq->strd_scatter_en) { struct rte_mbuf *prev = pkt; diff --git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index abd8ce7989..a1b0fa6455 100644 --- a/drivers/net/mlx5/mlx5_rxq.c +++ b/drivers/net/mlx5/mlx5_rxq.c @@ -1449,6 +1449,10 @@ mlx5_rxq_new(struct rte_eth_dev *dev, uint16_t idx, uint16_t desc, tmpl->socket = socket; if (dev->data->dev_conf.intr_conf.rxq) tmpl->irq = 1; +#ifdef RTE_LIBRTE_MLX5_NTLOAD_TSTORE_ALIGN_COPY + tmpl->rxq.mprq_tstore_memcpy = config->mprq_tstore_memcpy; +#endif + /* * This Rx queue can be configured as a Multi-Packet RQ if all of the * following conditions are met: diff --git a/meson_options.txt b/meson_options.txt index e232c9c340..cc7b629d17 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -38,6 +38,8 @@ option('max_lcores', type: 'integer', value: 128, description: 'maximum number of cores/threads supported by EAL') option('max_numa_nodes', type: 'integer', value: 32, description: 'maximum number of NUMA nodes supported by EAL') +option('mlx5_ntload_tstore', type: 'boolean', value: false, description: + 'to enable optimized MPRQ in RX datapath') option('platform', type: 'string', value: 'native', description: 'Platform to build, either "native", "generic" or a SoC. Please refer to the Linux build guide for more information.') option('enable_trace_fp', type: 'boolean', value: false, description: -- 2.25.1