From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E91F3A0C46; Tue, 28 Sep 2021 01:53:56 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 66F2140E3C; Tue, 28 Sep 2021 01:53:56 +0200 (CEST) Received: from mail-io1-f52.google.com (mail-io1-f52.google.com [209.85.166.52]) by mails.dpdk.org (Postfix) with ESMTP id 5022C40DF6 for ; Tue, 28 Sep 2021 01:53:55 +0200 (CEST) Received: by mail-io1-f52.google.com with SMTP id q3so24834565iot.3 for ; Mon, 27 Sep 2021 16:53:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=VfvYElKmOz7cL7Sx6fR1tZqzHc0PIo/K2hjvJBE+/Oo=; b=Y9ynBvHwBfGsjMckNf9insfUlxDW9n9dqfyATD75h6jZviAQd0WxKzEEWjC8ildd6r +5Dx1yG5E+4ywnVsuuAwFALTfkQlnykwBE2kTj4mZMNKMe8j+zrb2c6JB9OoA5qB8nKQ QThWr2Ps20pV3FN7zy6doRq3cIlxztARMMlP0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=VfvYElKmOz7cL7Sx6fR1tZqzHc0PIo/K2hjvJBE+/Oo=; b=2UIIqWhstO82GmlGHt5NMPAhXiX2Vgc/KssoSMRLevUHZs+HeaaSsK0hKap6vlH6Fe qvgF3dAzLizUr/bVTPMQw/hNosZ9xXuV+CihajEGPsWrgoN81f+9RlQCVfWsrpN5x/TI EdAqQPUExT37U0G5iWtKXjLtVenRgC9NvCDR85R06qFG2t5i/sFQiASvpcyq62aggzPX DqIYoNgb0kpoI+yWNfTlKDN6k97lUKLUyWzFVn2jP63qBLYYBNeWwQzvs3WA1nsgtQ9f uu02yvAPDQT/WE7Xai/Y+opfsTLPBSwOCmFoIFhKun7qVSdYdy9aFx8lZbY/empgXQPR 1ZXw== X-Gm-Message-State: AOAM532z7MDfffC7XGNTHqVyCWTSBjSZ4KD0AicfKXOZG8RRePp8660o N3+kKHiHsVu7/zTfhNX0ylLi/pMMvPUIr9hgYHgqQw== X-Google-Smtp-Source: ABdhPJxzyknFojPt1bNRRVu2gD29OofBsfeUvslO5a0m0AiH1t1ptn2M4v7bmhpOs7Ii6/qZKV8qhD3Jeqpblkzk3M4= X-Received: by 2002:a02:1081:: with SMTP id 123mr1984223jay.83.1632786834534; Mon, 27 Sep 2021 16:53:54 -0700 (PDT) MIME-Version: 1.0 References: <20210727034204.20649-1-xuemingl@nvidia.com> <20210917080121.329373-1-xuemingl@nvidia.com> <20210917080121.329373-2-xuemingl@nvidia.com> In-Reply-To: <20210917080121.329373-2-xuemingl@nvidia.com> From: Ajit Khaparde Date: Mon, 27 Sep 2021 16:53:38 -0700 Message-ID: To: Xueming Li Cc: dpdk-dev , Jerin Jacob , Ferruh Yigit , Andrew Rybchenko , Viacheslav Ovsiienko , Thomas Monjalon , Lior Margalit Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="00000000000049454805cd02cf2d" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 Subject: Re: [dpdk-dev] [PATCH v3 1/8] ethdev: introduce shared Rx queue X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" --00000000000049454805cd02cf2d Content-Type: text/plain; charset="UTF-8" On Fri, Sep 17, 2021 at 1:02 AM Xueming Li wrote: > > In current DPDK framework, each RX queue is pre-loaded with mbufs for > incoming packets. When number of representors scale out in a switch > domain, the memory consumption became significant. Most important, > polling all ports leads to high cache miss, high latency and low > throughput. > > This patch introduces shared RX queue. Ports with same configuration in > a switch domain could share RX queue set by specifying sharing group. > Polling any queue using same shared RX queue receives packets from all > member ports. Source port is identified by mbuf->port. > > Port queue number in a shared group should be identical. Queue index is > 1:1 mapped in shared group. > > Share RX queue must be polled on single thread or core. > > Multiple groups is supported by group ID. Can you clarify this a little more? Apologies if this was already covered: * Can't we do this for Tx also? Couple of nits inline. Thanks > > Signed-off-by: Xueming Li > Cc: Jerin Jacob > --- > Rx queue object could be used as shared Rx queue object, it's important > to clear all queue control callback api that using queue object: > https://mails.dpdk.org/archives/dev/2021-July/215574.html > --- > doc/guides/nics/features.rst | 11 +++++++++++ > doc/guides/nics/features/default.ini | 1 + > doc/guides/prog_guide/switch_representation.rst | 10 ++++++++++ > lib/ethdev/rte_ethdev.c | 1 + > lib/ethdev/rte_ethdev.h | 7 +++++++ > 5 files changed, 30 insertions(+) > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst > index a96e12d155..2e2a9b1554 100644 > --- a/doc/guides/nics/features.rst > +++ b/doc/guides/nics/features.rst > @@ -624,6 +624,17 @@ Supports inner packet L4 checksum. > ``tx_offload_capa,tx_queue_offload_capa:DEV_TX_OFFLOAD_OUTER_UDP_CKSUM``. > > > +.. _nic_features_shared_rx_queue: > + > +Shared Rx queue > +--------------- > + > +Supports shared Rx queue for ports in same switch domain. > + > +* **[uses] rte_eth_rxconf,rte_eth_rxmode**: ``offloads:RTE_ETH_RX_OFFLOAD_SHARED_RXQ``. > +* **[provides] mbuf**: ``mbuf.port``. > + > + > .. _nic_features_packet_type_parsing: > > Packet type parsing > diff --git a/doc/guides/nics/features/default.ini b/doc/guides/nics/features/default.ini > index 754184ddd4..ebeb4c1851 100644 > --- a/doc/guides/nics/features/default.ini > +++ b/doc/guides/nics/features/default.ini > @@ -19,6 +19,7 @@ Free Tx mbuf on demand = > Queue start/stop = > Runtime Rx queue setup = > Runtime Tx queue setup = > +Shared Rx queue = > Burst mode info = > Power mgmt address monitor = > MTU update = > diff --git a/doc/guides/prog_guide/switch_representation.rst b/doc/guides/prog_guide/switch_representation.rst > index ff6aa91c80..45bf5a3a10 100644 > --- a/doc/guides/prog_guide/switch_representation.rst > +++ b/doc/guides/prog_guide/switch_representation.rst > @@ -123,6 +123,16 @@ thought as a software "patch panel" front-end for applications. > .. [1] `Ethernet switch device driver model (switchdev) > `_ > > +- Memory usage of representors is huge when number of representor grows, > + because PMD always allocate mbuf for each descriptor of Rx queue. > + Polling the large number of ports brings more CPU load, cache miss and > + latency. Shared Rx queue can be used to share Rx queue between PF and > + representors in same switch domain. ``RTE_ETH_RX_OFFLOAD_SHARED_RXQ`` "in the same switch" > + is present in Rx offloading capability of device info. Setting the > + offloading flag in device Rx mode or Rx queue configuration to enable > + shared Rx queue. Polling any member port of shared Rx queue can return "of the shared Rx queue.." > + packets of all ports in group, port ID is saved in ``mbuf.port``. "ports in the group, " > + > Basic SR-IOV > ------------ > > diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c > index a7c090ce79..b3a58d5e65 100644 > --- a/lib/ethdev/rte_ethdev.c > +++ b/lib/ethdev/rte_ethdev.c > @@ -127,6 +127,7 @@ static const struct { > RTE_RX_OFFLOAD_BIT2STR(OUTER_UDP_CKSUM), > RTE_RX_OFFLOAD_BIT2STR(RSS_HASH), > RTE_ETH_RX_OFFLOAD_BIT2STR(BUFFER_SPLIT), > + RTE_ETH_RX_OFFLOAD_BIT2STR(SHARED_RXQ), > }; > > #undef RTE_RX_OFFLOAD_BIT2STR > diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h > index d2b27c351f..a578c9db9d 100644 > --- a/lib/ethdev/rte_ethdev.h > +++ b/lib/ethdev/rte_ethdev.h > @@ -1047,6 +1047,7 @@ struct rte_eth_rxconf { > uint8_t rx_drop_en; /**< Drop packets if no descriptors are available. */ > uint8_t rx_deferred_start; /**< Do not start queue with rte_eth_dev_start(). */ > uint16_t rx_nseg; /**< Number of descriptions in rx_seg array. */ > + uint32_t shared_group; /**< Shared port group index in switch domain. */ > /** > * Per-queue Rx offloads to be set using DEV_RX_OFFLOAD_* flags. > * Only offloads set on rx_queue_offload_capa or rx_offload_capa > @@ -1373,6 +1374,12 @@ struct rte_eth_conf { > #define DEV_RX_OFFLOAD_OUTER_UDP_CKSUM 0x00040000 > #define DEV_RX_OFFLOAD_RSS_HASH 0x00080000 > #define RTE_ETH_RX_OFFLOAD_BUFFER_SPLIT 0x00100000 > +/** > + * Rx queue is shared among ports in same switch domain to save memory, > + * avoid polling each port. Any port in group can be used to receive packets. "Any port in the group can" > + * Real source port number saved in mbuf->port field. > + */ > +#define RTE_ETH_RX_OFFLOAD_SHARED_RXQ 0x00200000 > > #define DEV_RX_OFFLOAD_CHECKSUM (DEV_RX_OFFLOAD_IPV4_CKSUM | \ > DEV_RX_OFFLOAD_UDP_CKSUM | \ > -- > 2.33.0 > --00000000000049454805cd02cf2d--