From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5582BA034C for ; Tue, 18 Jan 2022 14:41:32 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 01B1242720; Tue, 18 Jan 2022 14:41:32 +0100 (CET) Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) by mails.dpdk.org (Postfix) with ESMTP id 5C9294068E for ; Tue, 18 Jan 2022 14:41:30 +0100 (CET) Received: by mail-ot1-f49.google.com with SMTP id a12-20020a0568301dcc00b005919e149b4cso24186998otj.8 for ; Tue, 18 Jan 2022 05:41:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=codilime.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=0EKVlSsfU7uFb5itT6y047JhPYxp9maClqrEgZLzo7I=; b=e/2/W/qRfEzMy7+J252cb2HOI4IkRAp1ty2wJysEtP5kRmc9WG2naLyJiVl8pmdrM4 ZbaAthBDsgZfpS8Vs9D/od44dOWrUdmDJwl4jr1FjN1NwcDULP4qN7nShlEShdCikBoy GgJpxjLVwCLfdRYNHp3Gzsq9Pno4jjtIlO4boinsP1yO/Un0Ao7Eui6qTvvem7muh9UV gVvY45f5PCUPpY6sqx+ZrsUCE7JB5MYW9Eehho6cyCwPXCf3nNKPt1VGNqyza7X0F80U r3Ivu+GMuLzzaWXDxbm+LNFrj+0fYAZv8mr7DPUSQxoYT/StOMVYiZz8Iwf5nKw+IL5L bjNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=0EKVlSsfU7uFb5itT6y047JhPYxp9maClqrEgZLzo7I=; b=xQyOi+WVc4Ybkb/n7frFX/n+gc4bETH5+r7dsjflTKEiDQ+fAAcv7xNdJKoD+1GJa+ zbTE0ghSWnYb+iL2IWgJo/2hNElI+Nx22hqAjS3sjimnBW/sVvAL18rwsTUtMDSolWFV jq3gwu8KzaQ+l2TgaLhorUufCXrHH2JI84Bm66CidRWRIONgLpetUDDBtXPHd3G0qwWc Sbk4rl8+8EJ0g4ORJJf0/GhEecu602VVywoILCfPSkPDJEd8vcq+EPYkNRV9szLI1MJO znSxx3fCBSdD00JILcyiSLDMClWzc/TWTdutyCefiZgChG8fT81/MjdLGcMrCrDjdiOs 48MQ== X-Gm-Message-State: AOAM532B5VAV2k48APVotZw/H1G115Z9yfosx5NSs4Z83jA0krNXb4Lk gm9ISMLLpaapRn+W7JFk4ZtBYxeFPizzXXbEnByoUJGGoNbZjg+liatOltouxrO/rk/++M0iYB0 1yW3q806kPu7zp1TR48d6wCs= X-Google-Smtp-Source: ABdhPJwXK7t7fMdqwxPGPbRGX+d6r6iSCrUmbIBMCx/mvpFmlPyJEEh2fPMjNkix9u3HjtQEXkFf8Wn1fHz4w0gX6yc= X-Received: by 2002:a9d:1d3:: with SMTP id e77mr18948602ote.283.1642513289445; Tue, 18 Jan 2022 05:41:29 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Micha=C5=82_Niciejewski?= Date: Tue, 18 Jan 2022 14:41:18 +0100 Message-ID: Subject: Re: Unexpected behavior when using mbuf pool with external buffers To: "Van Haaren, Harry" Cc: "users@dpdk.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Hi, based on the materials you provided I found that IOTLB is the bottleneck (not the TLB). I am DMA mapping the memory so if I understand correctly, only IOMMU is involved here. Below I post some outputs from pcm tool: 10mpps, first run IOTLB Hit - 25 M IOTLB Miss - 10 M 15mpps IOTLB Hit - 28 M IOTLB Miss - 20 M 10mpps, second run IOTLB Hit - 23 M IOTLB Miss - 18 M I also tested the same scenario on another, more powerful server, and the results differ greatly: 10mpps, first run IOTLB Hit - 36 M IOTLB Miss - 644 K 25mpps (here I had to send more packets before drops appeared) IOTLB Hit - 71 M IOTLB Miss - 3860 K 10mpps, second run IOTLB Hit - 36 M IOTLB Miss - 1047 K So the problems with mempool fragmentation are visible but it is not that painful as in the first server. It looks like the first server is much worse in terms of IOMMU than the second one. I disabled IOMMU and used physical addresses to create an external buffer mempool (https://gist.github.com/tropuq/55c334bf3a2ab86b89a0b59e42b8af08) and it solved the performance issues. I have some questions about these results: 1. Is there something wrong with IOMMU in the first server - could it be that it's missing some additional configuration? 2. Is it normal to see that big differences? 3. Is there any way to find some info about IOMMU, like the size of IOTLB, = etc.? Thanks, Micha=C5=82 Niciejewski On Wed, Dec 22, 2021 at 5:30 PM Micha=C5=82 Niciejewski wrote: > > Thank you for the replay, > > On Wed, Dec 22, 2021 at 11:24 AM Van Haaren, Harry > wrote: > > I'll "top post" on this reply as the content is in HTML format below. I= n future, please try to send plain-text emails to DPDK mailing lists. > > I hope it's better now. > > > Estimating and talking is never conclusive =E2=80=93 lets measure using= Linux "Perf" tool. Run this command 3x, just like you posted the drop stat= s below. > > > > I expect to see lower dTLB-load-misses on the first run (no drops, 10 m= pps), and that the dTLB misses are higher for 15 mpps *and* for 10 mpps aga= in afterwards. > > > > perf stat -e cycles,dTLB-load-misses -C -- sleep = 1 > > extbuf, aligned_alloc, 10mpps, first run > Performance counter stats for 'CPU(s) 0': > 2404553948 cycles > 461 dTLB-load-misses > 1.001938861 seconds time elapsed > > extbuf, aligned_alloc, 15mpps > Performance counter stats for 'CPU(s) 0': > 2404518710 cycles > 466 dTLB-load-misses > 1.001920171 seconds time elapsed > > extbuf, aligned_alloc, 10mpps, second run > Performance counter stats for 'CPU(s) 0': > 2402586106 cycles > 449 dTLB-load-misses > 1.001114692 seconds time elapsed > > I also checked what happens when there is no traffic at all and the > results are similar: > > Performance counter stats for 'CPU(s) 0': > 2949935339 cycles > 465 dTLB-load-misses > 1.002236168 seconds time elapsed > > Also, I checked how the application behaves when adding --no-huge > option and using a normal mbuf pool. The results are very different > compared to aligned_alloc + extbuf mbuf pool: > > 10mpps, --no-huge > Performance counter stats for 'CPU(s) 0': > 2402616160 cycles > 17980033 dTLB-load-misses > 1.001125954 seconds time elapsed > > Application logs: > Queue: 0 > Number of all rx burst calls: 5757205 > Number of non-zero rx burst calls: 1073081 > Avg pkt nb received per rx burst: 1.7364 > All received pkts: 9996804 > All sent pkts: 8074460 > All dropped pkts: 1922344 --=20 Micha=C5=82 Niciejewski Junior Software Engineer michal.niciejewski@codilime.com CodiLime Sp. z o.o. - Ltd. company with its registered office in Poland, 02-493 Warsaw, ul. Krancowa 5. Registered by The District Court for the Capital City of Warsaw, XII Commercial Department of the National Court Register. Entered into National Court Register under No. KRS 0000388871. Tax identification number (NIP) 5272657478. Statistical number (REGON) 142974628. --=20 ------------------------------- This document contains material that is=20 confidential in CodiLime Sp. z o.o. DO NOT PRINT. DO NOT COPY. DO NOT=20 DISTRIBUTE. If you are not the intended recipient of this document, be=20 aware that any use, review, retransmission, distribution, reproduction or= =20 any action taken in reliance upon this message is strictly prohibited. If= =20 you received this in error, please contact the sender and help@codilime.com= =20 . Return the paper copy, delete the material from= =20 all computers and storage media.