From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dispatch1-us1.ppe-hosted.com (dispatch1-us1.ppe-hosted.com [67.231.154.164]) by dpdk.org (Postfix) with ESMTP id 0AC107CE1 for ; Tue, 14 Nov 2017 07:24:37 +0100 (CET) Received: from pure.maildistiller.com (unknown [10.110.50.29]) by dispatch1-us1.ppe-hosted.com (Proofpoint Essentials ESMTP Server) with ESMTP id C28B22005C; Tue, 14 Nov 2017 06:24:37 +0000 (UTC) X-Virus-Scanned: Proofpoint Essentials engine Received: from mx1-us1.ppe-hosted.com (unknown [10.110.49.251]) by pure.maildistiller.com (Proofpoint Essentials ESMTP Server) with ESMTPS id 0C77922004D; Tue, 14 Nov 2017 06:24:36 +0000 (UTC) Received: from webmail.solarflare.com (uk.solarflare.com [193.34.186.16]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1-us1.ppe-hosted.com (Proofpoint Essentials ESMTP Server) with ESMTPS id 63414400070; Tue, 14 Nov 2017 06:24:36 +0000 (UTC) Received: from [192.168.38.17] (84.52.114.114) by ukex01.SolarFlarecom.com (10.17.10.4) with Microsoft SMTP Server (TLS) id 15.0.1044.25; Tue, 14 Nov 2017 06:24:30 +0000 To: Ilya Matveychikov , Adrien Mazarguil CC: References: <2259E047-80C0-40AC-AAF4-F21617605508@gmail.com> <20171113103927.GP24849@6wind.com> <5F2502C3-99CF-4BE1-9DEC-364C5E636061@gmail.com> <20171113171512.GV24849@6wind.com> <2FF46D73-66D4-4E6B-8509-DD0CEEFF12D3@gmail.com> From: Andrew Rybchenko Message-ID: <70e9ecfe-2f65-980c-7e47-eb8734ad3256@solarflare.com> Date: Tue, 14 Nov 2017 09:24:27 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <2FF46D73-66D4-4E6B-8509-DD0CEEFF12D3@gmail.com> Content-Language: en-GB X-Originating-IP: [84.52.114.114] X-ClientProxiedBy: ocex03.SolarFlarecom.com (10.20.40.36) To ukex01.SolarFlarecom.com (10.17.10.4) X-TM-AS-Product-Ver: SMEX-11.0.0.1191-8.100.1062-23464.003 X-TM-AS-Result: No--15.636300-0.000000-31 X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-MDID: 1510640677-RCQuE5R1VX7H Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] A question about (poor) rte_ethdev internal rx/tx callbacks design X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Nov 2017 06:24:38 -0000 On 11/13/2017 10:33 PM, Ilya Matveychikov wrote: >> On Nov 13, 2017, at 9:15 PM, Adrien Mazarguil wrote: >> >> On Mon, Nov 13, 2017 at 02:56:23PM +0400, Ilya Matveychikov wrote: >>>> On Nov 13, 2017, at 2:39 PM, Adrien Mazarguil wrote: >>>> >>>> On Sat, Nov 11, 2017 at 09:18:45PM +0400, Ilya Matveychikov wrote: >>>>> Folks, >>>>> >>>>> Are you serious with it: >>>>> >>>>> typedef uint16_t (*eth_rx_burst_t)(void *rxq, >>>>> struct rte_mbuf **rx_pkts, >>>>> uint16_t nb_pkts); >>>>> typedef uint16_t (*eth_tx_burst_t)(void *txq, >>>>> struct rte_mbuf **tx_pkts, >>>>> uint16_t nb_pkts); >>>>> >>>>> I’m not surprised that every PMD stores port_id in every and each queue as having just the queue as an argument doesn’t allow to get the device. So the question is - why not to use something like: >>>>> >>>>> typedef uint16_t (*eth_rx_burst_t)(void *dev, uint16_t queue_id, >>>>> struct rte_mbuf **rx_pkts, >>>>> uint16_t nb_pkts); >>>>> typedef uint16_t (*eth_tx_burst_t)(void *dev, uint16_t queue_id, >>>>> struct rte_mbuf **tx_pkts, >>>>> uint16_t nb_pkts); >>>> I assume it's since the rte_eth_[rt]x_burst() wrappers already pay the price >>>> for that indirection, doing it twice would be redundant. >>> No need to do it twice, agree. We can pass dev pointer as well as queue, not just the queue’s >>> index. >>> >>>> Basically the cost of storing a back-pointer to dev or a queue index in each >>>> Rx/Tx queue structure is minor compared to saving a couple of CPU cycles >>>> wherever we can. >>> Not sure about it. More data to store - more cache space to occupy. Note that every queue has >>> at least 4 bytes more than it actually needs. And RTE_MAX_QUEUES_PER_PORT is defined >>> by it’s default to 1024. So we may have 4k extra for each port.... >> Note that queues are only allocated if requested by application, there's >> really not much overhead involved. > Yeah, mostly you are right here. > >> Also to echo Konstantin's reply and clarify mine, PMDs normally do not >> access this structure from their data plane. This pointer, if needed, is >> normally stored away from hot regions accessed during TX/RX, usually at the >> end of TX/RX structures and only for the convenience of management >> operations. It therefore has no measurable impact on the CPU cache. >> > I did a research of how drivers implements rx/tx queues and now I want to share the information > and some thoughts about it (see the info at the end): > > 1) All drivers have tx/rx queues defined as structures > 2) Current design implies that it’s enough to pass opaque rx/tx queue to the driver and frankly > speaking it is. But.. > 3) Most of drivers wants to get not only the queue’s pointer but at least queue_id and port_id and > most of them wants to have the pointer to internal devices’ data also. > > The way each driver solves (3) issue is data duplication. In other words, every queue used to have > such the information (queue_id, port_id and dev_priv pointer) inside. > > My question was and still about such the design. Not sure that it’s the best way to do it keeping in > mind that queue_id may be calculated using pointer difference and port_id may be stored just only > once per device. But it’ll require to change internal interface, sure. > > And as I promised here is the result of the research on rx/tx queues: <...> > drivers/net/sfq: > struct sfc_ef10_rxq { dp { queue_id, port_id } } > struct sfc_ef10_txq { dp { queue_id, port_id } } Which are not used on data/fast path (as Adrien state above). So, it does not affect cache usage efficient etc.