From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E2DDCA034E for ; Wed, 22 Dec 2021 17:30:36 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6CB0740040; Wed, 22 Dec 2021 17:30:36 +0100 (CET) Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) by mails.dpdk.org (Postfix) with ESMTP id C25244003C for ; Wed, 22 Dec 2021 17:30:35 +0100 (CET) Received: by mail-ot1-f49.google.com with SMTP id x3-20020a05683000c300b0057a5318c517so3468139oto.13 for ; Wed, 22 Dec 2021 08:30:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=codilime.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=qo5xAlU0Ok66M9iI3XemD5oeg6FcHyE6FGkB36PKk5k=; b=i6xlluBvgoOG8RGvLjlRdYL9AbqJ2whkh/8rJlTTLDXqCZq3DxKvy44+6EaqCV+dae JdiSb/WmEzMTeOky8qntY9vRQyrvkdENM3JtFOocHY9DeCULp4PbS0lMJKOH5Qc7/zM3 oX8B8I10hTZd3Lz/tCmwETNRmpg4TnsZQ+l4iV2LVQW8yXqZRwgeCSlqJXL7HDzmtUWO XiD00PyrUIn32arhevQDppGh+kQzleBhfMqbSZqBYBdyctt0RHc6ceeEZUl0ApGleCQZ tx51UMixTSBIaEoTBufxThBYsFqoh259h+2Hkd07BBWpazdpoZFTojeX9ccBt0yxuIra zDxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=qo5xAlU0Ok66M9iI3XemD5oeg6FcHyE6FGkB36PKk5k=; b=c03McjZ3w6YBln5V7THBvGoglSkAJ6u/kSnKo0whTEsWm+sECtEU4QJt0GEVBGldxF M4nNrkTUEznPJgm29843Q5wufzvz/T2t/DyNXDTgZwpFRJz/t9v4ul4ZZrpqfu3ajIXd OZHV5r0jxbjSe00RW0TxiwnyTEuItSn/6AKPKDBDufSPhS0tiAYMy1vjp9Bk9nM3tc9n C0nYQ6HDeveN6Iaz5W8rZFd6uyq06VGyaxqO+nodau4xHSvpXsOfkE1aTMY/aeIDXItW csVgYrRmjnXWnBeHAS+LeuNJIYgA/VIX9MoMnfnAuLHLdHZ5e1vXvQ4mCE1wQXdVUe1e /ZUg== X-Gm-Message-State: AOAM533JJcxJUQJanM35w2zUJqHVeuBbn3ywLvRFL7qXpFR6EIgjoHUe GffZ8Q5YKuVZH5IfVWmwUmIMF/qo191f8mtdGKfHF38cN3G6U4jy6i04AqjK/Bt1HuQMCz9rWSP UoWP3ja9BcJhCUJOZE3pFDB0= X-Google-Smtp-Source: ABdhPJwON7nPd/IjNTJLTSkx+w2DgiogD1fDxvzAfodLI2OcKwAFb0kfgFyffUzpmvXk5JbMIomHXDgi3gURUOownjE= X-Received: by 2002:a05:6830:18b:: with SMTP id q11mr2631733ota.113.1640190634872; Wed, 22 Dec 2021 08:30:34 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Micha=C5=82_Niciejewski?= Date: Wed, 22 Dec 2021 17:30:24 +0100 Message-ID: Subject: Re: Unexpected behavior when using mbuf pool with external buffers To: "Van Haaren, Harry" Cc: "users@dpdk.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Thank you for the replay, On Wed, Dec 22, 2021 at 11:24 AM Van Haaren, Harry wrote: > I'll "top post" on this reply as the content is in HTML format below. In = future, please try to send plain-text emails to DPDK mailing lists. I hope it's better now. > Estimating and talking is never conclusive =E2=80=93 lets measure using L= inux "Perf" tool. Run this command 3x, just like you posted the drop stats = below. > > I expect to see lower dTLB-load-misses on the first run (no drops, 10 mpp= s), and that the dTLB misses are higher for 15 mpps *and* for 10 mpps again= afterwards. > > perf stat -e cycles,dTLB-load-misses -C -- sleep 1 extbuf, aligned_alloc, 10mpps, first run Performance counter stats for 'CPU(s) 0': 2404553948 cycles 461 dTLB-load-misses 1.001938861 seconds time elapsed extbuf, aligned_alloc, 15mpps Performance counter stats for 'CPU(s) 0': 2404518710 cycles 466 dTLB-load-misses 1.001920171 seconds time elapsed extbuf, aligned_alloc, 10mpps, second run Performance counter stats for 'CPU(s) 0': 2402586106 cycles 449 dTLB-load-misses 1.001114692 seconds time elapsed I also checked what happens when there is no traffic at all and the results are similar: Performance counter stats for 'CPU(s) 0': 2949935339 cycles 465 dTLB-load-misses 1.002236168 seconds time elapsed Also, I checked how the application behaves when adding --no-huge option and using a normal mbuf pool. The results are very different compared to aligned_alloc + extbuf mbuf pool: 10mpps, --no-huge Performance counter stats for 'CPU(s) 0': 2402616160 cycles 17980033 dTLB-load-misses 1.001125954 seconds time elapsed Application logs: Queue: 0 Number of all rx burst calls: 5757205 Number of non-zero rx burst calls: 1073081 Avg pkt nb received per rx burst: 1.7364 All received pkts: 9996804 All sent pkts: 8074460 All dropped pkts: 1922344 --=20 Micha=C5=82 Niciejewski Junior Software Engineer michal.niciejewski@codilime.com CodiLime Sp. z o.o. - Ltd. company with its registered office in Poland, 02-493 Warsaw, ul. Krancowa 5. Registered by The District Court for the Capital City of Warsaw, XII Commercial Department of the National Court Register. Entered into National Court Register under No. KRS 0000388871. Tax identification number (NIP) 5272657478. Statistical number (REGON) 142974628. --=20 ------------------------------- This document contains material that is=20 confidential in CodiLime Sp. z o.o. DO NOT PRINT. DO NOT COPY. DO NOT=20 DISTRIBUTE. If you are not the intended recipient of this document, be=20 aware that any use, review, retransmission, distribution, reproduction or= =20 any action taken in reliance upon this message is strictly prohibited. If= =20 you received this in error, please contact the sender and help@codilime.com= =20 . Return the paper copy, delete the material from= =20 all computers and storage media.