From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id A74FAA0C4B for ; Wed, 14 Jul 2021 12:25:46 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 77D544014E; Wed, 14 Jul 2021 12:25:46 +0200 (CEST) Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by mails.dpdk.org (Postfix) with ESMTP id 7B01B4014E for ; Wed, 14 Jul 2021 12:25:44 +0200 (CEST) Received: by mail-wr1-f42.google.com with SMTP id v5so2574617wrt.3 for ; Wed, 14 Jul 2021 03:25:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=1fsbFBuUyiAkcNCOI1kS+b7df06XuCyoY/mFsW4HjUg=; b=bMQVEso47C2Ig0Mdsu4w5uEl8TGtZVBXUgoT8NltAnt2DzXxLcOHr9lgJGtjeorrKs qYqK3sbeuRMYM5VDlOcQDNn+8N4wKLj8bMB9Qbp3hc8LA9p8iKfEQyiykHtaObo2spVW zubvR4D5XiL+S7gjlmBRoS48w+OW01kb2zQi874LLG6JUrXi0+kn6Di3GgUGuDv1srVi 58AgOMkq4KmA617DSupKG8KTKOpXsc8HmwF8L+fcjwgaBPbL6WaRGGDe1POlb71qW1zG PKugIIVwu5jBmEkMrXieSYKnPB3Ichbo5rakYzbIGc0xYdM0vZ9ca8Sqou7/iTu3PWxS FpJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=1fsbFBuUyiAkcNCOI1kS+b7df06XuCyoY/mFsW4HjUg=; b=sm8rgTBQKFzqNb4W5uQrk9pPXDZj6vrUSQHeASBul31nY+pDiiJNaq/U3kaLXiBeqM VZVmo1vWFrvDR7rySZvrXUvEOrIEMVCRZPymuAdfi/m9mDqvligZebKEX3l5ZxaHNYaw czym6wetRUW/xW+enPucFlRx3qo2AFDt2GGKtR8Zvb/J90+9dvrHULS1b11+Qg7GBamT QyVxAlig8nRGWpSsJV1rg87qiki9oEJP1tgCWk2w4r5A1bU7TGNbZcdBG/2rt6NPS/WI ora5oeukOUEZCbLxE4NaHuHZuyPdlxfAykiI95LICNJDy88uyzeIYufwuQbCL/1adqyb nPHg== X-Gm-Message-State: AOAM531HL/UoXtgPk6DLA1sbgDGw2kvm4kQRa60OIu+B0mdnK2B4NIQC RdaZwFKIHyG5hAeTzxE3GnVhXXNQ8d7/RwZN X-Google-Smtp-Source: ABdhPJyp9PIJWeVJWQ3mYcab0tiksSpkq6l6inRW3bpcP8LQ+ft3EBMh88vFvMxxYc6XoKD+qtB6/g== X-Received: by 2002:a5d:4c52:: with SMTP id n18mr12058482wrt.295.1626258343655; Wed, 14 Jul 2021 03:25:43 -0700 (PDT) Received: from ?IPv6:2a01:4b00:9d54:2900:366c:50bf:ae31:61bd? ([2a01:4b00:9d54:2900:366c:50bf:ae31:61bd]) by smtp.gmail.com with ESMTPSA id x9sm2079571wrm.82.2021.07.14.03.25.42 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 14 Jul 2021 03:25:43 -0700 (PDT) To: Stephen Hemminger , "Van Haaren, Harry" Cc: Filip Janiszewski , "users@dpdk.org" References: <01e2ccbe-a2c8-be7c-7f9d-af43f609e75f@filipjaniszewski.com> <20210519090650.5023ce00@hermes.local> From: Alireza Sanaee Message-ID: <596823fd-17b8-eb64-cafc-58bf9065115b@gmail.com> Date: Wed, 14 Jul 2021 11:25:41 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <20210519090650.5023ce00@hermes.local> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] Performance of rte_eth_stats_get X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On 19/05/2021 17:06, Stephen Hemminger wrote: > On Wed, 19 May 2021 15:14:38 +0000 > "Van Haaren, Harry" wrote: > >>> -----Original Message----- >>> From: users On Behalf Of Filip Janiszewski >>> Sent: Wednesday, May 19, 2021 2:10 PM >>> To: users@dpdk.org >>> Subject: [dpdk-users] Performance of rte_eth_stats_get >>> >>> Hi, >>> >>> Is it safe to call rte_eth_stats_get while capturing from the port? >>> >>> I'm mostly concerned about performance, if rte_eth_stats_get will in any >>> way impact the port performance, in the application I plan to call the >>> function from a thread that is not directly involved in the capture, >>> there's another worker responsible for rx bursting, but I wonder if the >>> NIC might get upset if I call it too frequently (say 10 times per >>> second) and potentially cause some performance issues. >>> >>> The question is really Nic agnostic, but if the Nic vendor is actually >>> relevant then I'm running Intel 700 series nic and Mellanox ConnectX-4/5. >> >> To understand what really goes on when getting stats, it might help to list the >> steps involved in getting statistics from the NIC HW. >> >> 1) CPU sends an MMIO read (Memory Mapped I/O, aka, sometimes referred >> to as a "pci read") to the NIC. >> 2) The PCI bus has to handle extra TLPs (pci transactions) to satisfy read >> 3) NIC has to send a reply based on accessing its internal counters >> 4) CPU gets a result from the PCI read. >> >> Notice how elegantly this whole process is abstracted from SW? In code, reading >> a stat value is just dereferencing a pointer that is mapped to the NIC HW address. >> In practice from a CPU performance point of view, doing an MMIO-read is one of >> the slowest things you can do. You say the stats-reads are occurring from a thread >> that is not handling rx/datapath, so perhaps the CPU cycle cost itself isn't a concern. >> >> Do note however, that when reading a full set of extended stats from the NIC, there >> could be many 10's to 100's of MMIO reads (depending on the statistics requested, >> and how the PMD itself is implemented to handle stats updates). >> >> The PCI bus does become more busy with reads to the NIC HW when doing lots of >> statistic updates, so there is some more contention/activity to be expected there. >> The PCM tool can be very useful to see MMIO traffic, you could measure how many >> extra PCI transactions are occurring due to reading stats every X ms? >> https://github.com/opcm/pcm >> >> I can recommend measuring pkt latency/jitter as a histogram, as then outliers in performance >> can be identified. If you specifically want to identify if these are due stats reads, compare >> with a "no stats reads" latency/jitter histogram, and graphically see the impact. >> In the end if it doesn't affect packet latency/jitter, then it has no impact right? >> >> Ultimately, I can't give a generic answer - best steps are to measure carefully and find out! >> >>> Thanks >> >> Hope the above helps and doesn't add confusion :) Regards, -Harry > > Many drivers require transactions with the firmware via mailbox. > And that transaction needs a spin wait for the shared area. > Thank you for explaining the steps quite nicely. I also noticed this problem too. Calling `rte_eth_stats_get` in the PMDport per batch almost halves the throughput in a 10G setup IIRC, the cost is prohibitively HIGH. This, however, doesn't show up when DPDK connects a vhost-pmdport, since all of port statistics are probably somewhere in the shared memory.