From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ADFA6A034F; Mon, 11 Oct 2021 10:25:55 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 35D0240E01; Mon, 11 Oct 2021 10:25:55 +0200 (CEST) Received: from shelob.oktetlabs.ru (shelob.oktetlabs.ru [91.220.146.113]) by mails.dpdk.org (Postfix) with ESMTP id 2885440142 for ; Mon, 11 Oct 2021 10:25:54 +0200 (CEST) Received: from [192.168.38.17] (aros.oktetlabs.ru [192.168.38.17]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by shelob.oktetlabs.ru (Postfix) with ESMTPSA id 6DDE37F502; Mon, 11 Oct 2021 11:25:53 +0300 (MSK) DKIM-Filter: OpenDKIM Filter v2.11.0 shelob.oktetlabs.ru 6DDE37F502 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=oktetlabs.ru; s=default; t=1633940753; bh=LGj2JZblaRpuUlRC7uK+r20NorEexFw2rZ0CK3OAMCg=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=dSEn978fwXzU9o3bw6F3RYP1kECOrjgrmteKmqeiaMnuHUFa+d/JOU31fleKh+uEy ZInxJ5z4T8mmiQQ/XJBSvZ93VEFUsw1ML2kVf8b3ZS4pMzYSU5QAb670KStv0iQrKM cxpmUuyAnU+ZSe8hFdMN3LhxF4YANfsbiCyIiGnk= To: Konstantin Ananyev , dev@dpdk.org Cc: xiaoyun.li@intel.com, anoobj@marvell.com, jerinj@marvell.com, ndabilpuram@marvell.com, adwivedi@marvell.com, shepard.siegel@atomicrules.com, ed.czeck@atomicrules.com, john.miller@atomicrules.com, irusskikh@marvell.com, ajit.khaparde@broadcom.com, somnath.kotur@broadcom.com, rahul.lakkireddy@chelsio.com, hemant.agrawal@nxp.com, sachin.saxena@oss.nxp.com, haiyue.wang@intel.com, johndale@cisco.com, hyonkim@cisco.com, qi.z.zhang@intel.com, xiao.w.wang@intel.com, humin29@huawei.com, yisen.zhuang@huawei.com, oulijun@huawei.com, beilei.xing@intel.com, jingjing.wu@intel.com, qiming.yang@intel.com, matan@nvidia.com, viacheslavo@nvidia.com, sthemmin@microsoft.com, longli@microsoft.com, heinrich.kuhn@corigine.com, kirankumark@marvell.com, mczekaj@marvell.com, jiawenwu@trustnetic.com, jianwang@trustnetic.com, maxime.coquelin@redhat.com, chenbo.xia@intel.com, thomas@monjalon.net, ferruh.yigit@intel.com, mdr@ashroe.eu, jay.jayatheerthan@intel.com References: <20211004135603.20593-1-konstantin.ananyev@intel.com> <20211007112750.25526-1-konstantin.ananyev@intel.com> <20211007112750.25526-5-konstantin.ananyev@intel.com> From: Andrew Rybchenko Organization: OKTET Labs Message-ID: <37c19f16-ce0d-533f-2861-176ddf416df7@oktetlabs.ru> Date: Mon, 11 Oct 2021 11:25:53 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <20211007112750.25526-5-konstantin.ananyev@intel.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v5 4/7] ethdev: copy fast-path API into separate structure X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 10/7/21 2:27 PM, Konstantin Ananyev wrote: > Copy public function pointers (rx_pkt_burst(), etc.) and related > pointers to internal data from rte_eth_dev structure into a > separate flat array. That array will remain in a public header. > The intention here is to make rte_eth_dev and related structures internal. > That should allow future possible changes to core eth_dev structures > to be transparent to the user and help to avoid ABI/API breakages. > The plan is to keep minimal part of data from rte_eth_dev public, > so we still can use inline functions for fast-path calls > (like rte_eth_rx_burst(), etc.) to avoid/minimize slowdown. > The whole idea beyond this new schema: > 1. PMDs keep to setup fast-path function pointers and related data > inside rte_eth_dev struct in the same way they did it before. > 2. Inside rte_eth_dev_start() and inside rte_eth_dev_probing_finish() > (for secondary process) we call eth_dev_fp_ops_setup, which > copies these function and data pointers into rte_eth_fp_ops[port_id]. > 3. Inside rte_eth_dev_stop() and inside rte_eth_dev_release_port() > we call eth_dev_fp_ops_reset(), which resets rte_eth_fp_ops[port_id] > into some dummy values. > 4. fast-path ethdev API (rte_eth_rx_burst(), etc.) will use that new > flat array to call PMD specific functions. > That approach should allow us to make rte_eth_devices[] private > without introducing regression and help to avoid changes in drivers code. > > Signed-off-by: Konstantin Ananyev Overall LGTM, few nits below. > --- > lib/ethdev/ethdev_private.c | 52 ++++++++++++++++++++++++++++++++++ > lib/ethdev/ethdev_private.h | 7 +++++ > lib/ethdev/rte_ethdev.c | 27 ++++++++++++++++++ > lib/ethdev/rte_ethdev_core.h | 55 ++++++++++++++++++++++++++++++++++++ > 4 files changed, 141 insertions(+) > > diff --git a/lib/ethdev/ethdev_private.c b/lib/ethdev/ethdev_private.c > index 012cf73ca2..3eeda6e9f9 100644 > --- a/lib/ethdev/ethdev_private.c > +++ b/lib/ethdev/ethdev_private.c > @@ -174,3 +174,55 @@ rte_eth_devargs_parse_representor_ports(char *str, void *data) > RTE_LOG(ERR, EAL, "wrong representor format: %s\n", str); > return str == NULL ? -1 : 0; > } > + > +static uint16_t > +dummy_eth_rx_burst(__rte_unused void *rxq, > + __rte_unused struct rte_mbuf **rx_pkts, > + __rte_unused uint16_t nb_pkts) > +{ > + RTE_ETHDEV_LOG(ERR, "rx_pkt_burst for unconfigured port\n"); May be "unconfigured" -> "stopped" ? Or "non-started" ? > + rte_errno = ENOTSUP; > + return 0; > +} > + > +static uint16_t > +dummy_eth_tx_burst(__rte_unused void *txq, > + __rte_unused struct rte_mbuf **tx_pkts, > + __rte_unused uint16_t nb_pkts) > +{ > + RTE_ETHDEV_LOG(ERR, "tx_pkt_burst for unconfigured port\n"); May be "unconfigured" -> "stopped" ? > + rte_errno = ENOTSUP; > + return 0; > +} > + > +void > +eth_dev_fp_ops_reset(struct rte_eth_fp_ops *fpo) > +{ > + static void *dummy_data[RTE_MAX_QUEUES_PER_PORT]; > + static const struct rte_eth_fp_ops dummy_ops = { > + .rx_pkt_burst = dummy_eth_rx_burst, > + .tx_pkt_burst = dummy_eth_tx_burst, > + .rxq = {.data = dummy_data, .clbk = dummy_data,}, > + .txq = {.data = dummy_data, .clbk = dummy_data,}, > + }; > + > + *fpo = dummy_ops; > +} > + > +void > +eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo, > + const struct rte_eth_dev *dev) > +{ > + fpo->rx_pkt_burst = dev->rx_pkt_burst; > + fpo->tx_pkt_burst = dev->tx_pkt_burst; > + fpo->tx_pkt_prepare = dev->tx_pkt_prepare; > + fpo->rx_queue_count = dev->rx_queue_count; > + fpo->rx_descriptor_status = dev->rx_descriptor_status; > + fpo->tx_descriptor_status = dev->tx_descriptor_status; > + > + fpo->rxq.data = dev->data->rx_queues; > + fpo->rxq.clbk = (void **)(uintptr_t)dev->post_rx_burst_cbs; > + > + fpo->txq.data = dev->data->tx_queues; > + fpo->txq.clbk = (void **)(uintptr_t)dev->pre_tx_burst_cbs; > +} > diff --git a/lib/ethdev/ethdev_private.h b/lib/ethdev/ethdev_private.h > index 3724429577..5721be7bdc 100644 > --- a/lib/ethdev/ethdev_private.h > +++ b/lib/ethdev/ethdev_private.h > @@ -26,4 +26,11 @@ eth_find_device(const struct rte_eth_dev *_start, rte_eth_cmp_t cmp, > /* Parse devargs value for representor parameter. */ > int rte_eth_devargs_parse_representor_ports(char *str, void *data); > > +/* reset eth fast-path API to dummy values */ > +void eth_dev_fp_ops_reset(struct rte_eth_fp_ops *fpo); > + > +/* setup eth fast-path API to ethdev values */ > +void eth_dev_fp_ops_setup(struct rte_eth_fp_ops *fpo, > + const struct rte_eth_dev *dev); > + > #endif /* _ETH_PRIVATE_H_ */ > diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c > index c8abda6dd7..9f7a0cbb8c 100644 > --- a/lib/ethdev/rte_ethdev.c > +++ b/lib/ethdev/rte_ethdev.c > @@ -44,6 +44,9 @@ > static const char *MZ_RTE_ETH_DEV_DATA = "rte_eth_dev_data"; > struct rte_eth_dev rte_eth_devices[RTE_MAX_ETHPORTS]; > > +/* public fast-path API */ Shoudn't it be a doxygen style comment? > +struct rte_eth_fp_ops rte_eth_fp_ops[RTE_MAX_ETHPORTS]; > + > /* spinlock for eth device callbacks */ > static rte_spinlock_t eth_dev_cb_lock = RTE_SPINLOCK_INITIALIZER; > [snip] > diff --git a/lib/ethdev/rte_ethdev_core.h b/lib/ethdev/rte_ethdev_core.h > index 51cd68de94..d5853dff86 100644 > --- a/lib/ethdev/rte_ethdev_core.h > +++ b/lib/ethdev/rte_ethdev_core.h > @@ -50,6 +50,61 @@ typedef int (*eth_rx_descriptor_status_t)(void *rxq, uint16_t offset); > typedef int (*eth_tx_descriptor_status_t)(void *txq, uint16_t offset); > /**< @internal Check the status of a Tx descriptor */ > > +/** > + * @internal > + * Structure used to hold opaque pointers to internal ethdev Rx/Tx > + * queues data. > + * The main purpose to expose these pointers at all - allow compiler > + * to fetch this data for fast-path ethdev inline functions in advance. > + */ > +struct rte_ethdev_qdata { > + void **data; > + /**< points to array of internal queue data pointers */ Please, put the documentation on the same like or just put documentation before the documented member. > + void **clbk; > + /**< points to array of queue callback data pointers */ > +}; > + > +/** > + * @internal > + * fast-path ethdev functions and related data are hold in a flat array. > + * One entry per ethdev. > + * On 64-bit systems contents of this structure occupy exactly two 64B lines. > + * On 32-bit systems contents of this structure fits into one 64B line. > + */ > +struct rte_eth_fp_ops { > + > + /** > + * Rx fast-path functions and related data. > + * 64-bit systems: occupies first 64B line > + */ As I understand the above comment is for a group of below fields. If so, Doxygen annocation for member groups should be used. > + eth_rx_burst_t rx_pkt_burst; > + /**< PMD receive function. */ May I ask to avoid usage of documentation after member in a separate line. It makes sense to if it is located in the same line, but otherwise, it should be simply put before the documented member. [snip]