From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <arybchenko@solarflare.com>
Received: from dispatch1-us1.ppe-hosted.com (dispatch1-us1.ppe-hosted.com
[67.231.154.164]) by dpdk.org (Postfix) with ESMTP id 0AC107CE1
for <dev@dpdk.org>; Tue, 14 Nov 2017 07:24:37 +0100 (CET)
Received: from pure.maildistiller.com (unknown [10.110.50.29])
by dispatch1-us1.ppe-hosted.com (Proofpoint Essentials ESMTP Server) with
ESMTP id C28B22005C; Tue, 14 Nov 2017 06:24:37 +0000 (UTC)
X-Virus-Scanned: Proofpoint Essentials engine
Received: from mx1-us1.ppe-hosted.com (unknown [10.110.49.251])
by pure.maildistiller.com (Proofpoint Essentials ESMTP Server) with ESMTPS id
0C77922004D; Tue, 14 Nov 2017 06:24:36 +0000 (UTC)
Received: from webmail.solarflare.com (uk.solarflare.com [193.34.186.16])
(using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
(No client certificate requested)
by mx1-us1.ppe-hosted.com (Proofpoint Essentials ESMTP Server) with ESMTPS id
63414400070; Tue, 14 Nov 2017 06:24:36 +0000 (UTC)
Received: from [192.168.38.17] (84.52.114.114) by ukex01.SolarFlarecom.com
(10.17.10.4) with Microsoft SMTP Server (TLS) id 15.0.1044.25; Tue, 14 Nov
2017 06:24:30 +0000
To: Ilya Matveychikov <matvejchikov@gmail.com>, Adrien Mazarguil
<adrien.mazarguil@6wind.com>
CC: <dev@dpdk.org>
References: <2259E047-80C0-40AC-AAF4-F21617605508@gmail.com>
<20171113103927.GP24849@6wind.com>
<5F2502C3-99CF-4BE1-9DEC-364C5E636061@gmail.com>
<20171113171512.GV24849@6wind.com>
<2FF46D73-66D4-4E6B-8509-DD0CEEFF12D3@gmail.com>
From: Andrew Rybchenko <arybchenko@solarflare.com>
Message-ID: <70e9ecfe-2f65-980c-7e47-eb8734ad3256@solarflare.com>
Date: Tue, 14 Nov 2017 09:24:27 +0300
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <2FF46D73-66D4-4E6B-8509-DD0CEEFF12D3@gmail.com>
Content-Language: en-GB
X-Originating-IP: [84.52.114.114]
X-ClientProxiedBy: ocex03.SolarFlarecom.com (10.20.40.36) To
ukex01.SolarFlarecom.com (10.17.10.4)
X-TM-AS-Product-Ver: SMEX-11.0.0.1191-8.100.1062-23464.003
X-TM-AS-Result: No--15.636300-0.000000-31
X-TM-AS-User-Approved-Sender: Yes
X-TM-AS-User-Blocked-Sender: No
X-MDID: 1510640677-RCQuE5R1VX7H
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 8bit
X-Content-Filtered-By: Mailman/MimeDel 2.1.15
Subject: Re: [dpdk-dev] A question about (poor) rte_ethdev internal rx/tx
callbacks design
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 14 Nov 2017 06:24:38 -0000
On 11/13/2017 10:33 PM, Ilya Matveychikov wrote:
>> On Nov 13, 2017, at 9:15 PM, Adrien Mazarguil <adrien.mazarguil@6wind.com> wrote:
>>
>> On Mon, Nov 13, 2017 at 02:56:23PM +0400, Ilya Matveychikov wrote:
>>>> On Nov 13, 2017, at 2:39 PM, Adrien Mazarguil <adrien.mazarguil@6wind.com> wrote:
>>>>
>>>> On Sat, Nov 11, 2017 at 09:18:45PM +0400, Ilya Matveychikov wrote:
>>>>> Folks,
>>>>>
>>>>> Are you serious with it:
>>>>>
>>>>> typedef uint16_t (*eth_rx_burst_t)(void *rxq,
>>>>> struct rte_mbuf **rx_pkts,
>>>>> uint16_t nb_pkts);
>>>>> typedef uint16_t (*eth_tx_burst_t)(void *txq,
>>>>> struct rte_mbuf **tx_pkts,
>>>>> uint16_t nb_pkts);
>>>>>
>>>>> I’m not surprised that every PMD stores port_id in every and each queue as having just the queue as an argument doesn’t allow to get the device. So the question is - why not to use something like:
>>>>>
>>>>> typedef uint16_t (*eth_rx_burst_t)(void *dev, uint16_t queue_id,
>>>>> struct rte_mbuf **rx_pkts,
>>>>> uint16_t nb_pkts);
>>>>> typedef uint16_t (*eth_tx_burst_t)(void *dev, uint16_t queue_id,
>>>>> struct rte_mbuf **tx_pkts,
>>>>> uint16_t nb_pkts);
>>>> I assume it's since the rte_eth_[rt]x_burst() wrappers already pay the price
>>>> for that indirection, doing it twice would be redundant.
>>> No need to do it twice, agree. We can pass dev pointer as well as queue, not just the queue’s
>>> index.
>>>
>>>> Basically the cost of storing a back-pointer to dev or a queue index in each
>>>> Rx/Tx queue structure is minor compared to saving a couple of CPU cycles
>>>> wherever we can.
>>> Not sure about it. More data to store - more cache space to occupy. Note that every queue has
>>> at least 4 bytes more than it actually needs. And RTE_MAX_QUEUES_PER_PORT is defined
>>> by it’s default to 1024. So we may have 4k extra for each port....
>> Note that queues are only allocated if requested by application, there's
>> really not much overhead involved.
> Yeah, mostly you are right here.
>
>> Also to echo Konstantin's reply and clarify mine, PMDs normally do not
>> access this structure from their data plane. This pointer, if needed, is
>> normally stored away from hot regions accessed during TX/RX, usually at the
>> end of TX/RX structures and only for the convenience of management
>> operations. It therefore has no measurable impact on the CPU cache.
>>
> I did a research of how drivers implements rx/tx queues and now I want to share the information
> and some thoughts about it (see the info at the end):
>
> 1) All drivers have tx/rx queues defined as structures
> 2) Current design implies that it’s enough to pass opaque rx/tx queue to the driver and frankly
> speaking it is. But..
> 3) Most of drivers wants to get not only the queue’s pointer but at least queue_id and port_id and
> most of them wants to have the pointer to internal devices’ data also.
>
> The way each driver solves (3) issue is data duplication. In other words, every queue used to have
> such the information (queue_id, port_id and dev_priv pointer) inside.
>
> My question was and still about such the design. Not sure that it’s the best way to do it keeping in
> mind that queue_id may be calculated using pointer difference and port_id may be stored just only
> once per device. But it’ll require to change internal interface, sure.
>
> And as I promised here is the result of the research on rx/tx queues:
<...>
> drivers/net/sfq:
> struct sfc_ef10_rxq { dp { queue_id, port_id } }
> struct sfc_ef10_txq { dp { queue_id, port_id } }
Which are not used on data/fast path (as Adrien state above). So, it
does not affect cache usage efficient etc.