From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id BDDBFA0093 for ; Sat, 21 May 2022 11:42:16 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 56E0F40156; Sat, 21 May 2022 11:42:16 +0200 (CEST) Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by mails.dpdk.org (Postfix) with ESMTP id 120C840040 for ; Sat, 21 May 2022 11:42:15 +0200 (CEST) Received: by mail-oi1-f177.google.com with SMTP id y66so6466055oia.1 for ; Sat, 21 May 2022 02:42:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vSgLfyUWcTdwhHb49MtNPB94iiTtYMWOyiep+EVSaeg=; b=n75Kq4mDM+FDQIHLl4Dzk97mOD9zLs8cu+Xh6E+z79iXbalg7qS2mqcIaMQ60Jxgh5 HgrvChs6xZTyNOjeIptbHMsYnuBz02sK6ekrvcexzpVbw+zuMSdlM240KLg/zSdhwQ3c BleqQDQsUWNFjh6+3Fdbmh6oulJryEle1avYYvrCC3KzVQ6+1WxH68ai3WmyHmX2oZM6 iBrVh3oJqq8sIMg8/gEUlJz0/r4rS6T+Ho6a9m8uwH4TfqXG0IoeWMhIj3rPTygnMCfQ IKnMaXEIe5mPYiqJXodqUbWs7fn2XBphuMyRoWopkNSKRCZ6h2OGyi8sCMeL4NnRInEi p06g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vSgLfyUWcTdwhHb49MtNPB94iiTtYMWOyiep+EVSaeg=; b=Or8dRNnbiXz49MNZUzV46JDECFpBNRREVpymEau7FnI3ImNePkGWuCR+K3MuRu5+GR fLkwFy/MpNwCH8eS2Lu2J/l2pniK/4ylecj+CQ2VBiBdO4SY3Nd5dImE4OjHDjBh0mJz xBy9mQAAMiW1vRLUd6YCiGgA4ztNylOdMu3paU8t/JpcqcJRVpG9JoiTk/tEZ/1KrbpI pR7ao12z2U35FV4GdH8Amx21ByqxU1nqez1rzmLzLuftxfQee2flDR276jz+jORAnGOZ 44aE7WcrAx2UDkSNLFCkRSaMRWJssrXS11DIofDFG2x6doauOKT7uEbt4F94qmlqzb/V fd4Q== X-Gm-Message-State: AOAM531rFtnVg+fu25FQHZdvLobaxplBHmsiil8DIj6o0qwM4iAefYu6 BRMA5qiyEPiRcpmtnFaR+01PqCXss1RYcImaE3DbAZu0N/I1Bw== X-Google-Smtp-Source: ABdhPJznAwFn0BA/5wesXipOdYdelvglw6aqmGgdL2BZmqOhcXTX0+lJ/n6Vvf3nZy4Fw616RIeoFR59whY2QBVc9rA= X-Received: by 2002:aca:a8d7:0:b0:328:b19f:dbac with SMTP id r206-20020acaa8d7000000b00328b19fdbacmr7772814oie.243.1653126134326; Sat, 21 May 2022 02:42:14 -0700 (PDT) MIME-Version: 1.0 References: <20220520084843.698f04ee@hermes.local> In-Reply-To: <20220520084843.698f04ee@hermes.local> From: Antonio Di Bacco Date: Sat, 21 May 2022 11:42:06 +0200 Message-ID: Subject: Re: Optimizing memory access with DPDK allocated memory To: Stephen Hemminger Cc: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org I read a couple of articles (https://www.thomas-krenn.com/en/wiki/Optimize_memory_performance_of_Intel_Xeon_Scalable_systems?xtxsearchselecthit=1 and this https://www.exxactcorp.com/blog/HPC/balance-memory-guidelines-for-intel-xeon-scalable-family-processors) and I understood a little bit more. If the XEON memory controller is able to spread contiguous memory accesses onto different channels in hardware (as Stepphen correctly stated), then, how DPDK with option -n can benefit an application? I also coded a test application to write a 1GB hugepage and calculate time needed but, equipping an additional two DIMM on two unused channels of my available six channels motherboard (X11DPi-NT) , I didn't observe any improvement. This is strange because adding two channels to the 4 already equipped should make a noticeable difference. For reference this is the small program for allocating and writing memory. https://github.com/adibacco/simple_mp_mem_2 and the results with 4 memory channels: https://docs.google.com/spreadsheets/d/1mDoKYLMhMMKDaOS3RuGEnpPgRNKuZOy4lMIhG-1N7B8/edit?usp=sharing On Fri, May 20, 2022 at 5:48 PM Stephen Hemminger wrote: > > On Fri, 20 May 2022 10:34:46 +0200 > Antonio Di Bacco wrote: > > > Let us say I have two memory channels each one with its own 16GB memory > > module, I suppose the first memory channel will be used when addressing > > physical memory in the range 0 to 0x4 0000 0000 and the second when > > addressing physical memory in the range 0x4 0000 0000 to 0x7 ffff ffff. > > Correct? > > Now, I need to have a 2GB buffer with one "writer" and one "reader", the > > writer writes on half of the buffer (call it A) and, in the meantime, the > > reader reads on the other half (B). When the writer finishes writing its > > half buffer (A), signal it to the reader and they swap, the reader starts > > to read from A and writer starts to write to B. > > If I allocate the whole buffer (on two 1GB hugepages) across the two memory > > channels, one half of the buffer is allocated on the end of first channel > > while the other half is allocated on the start of the second memory > > channel, would this increase performances compared to the whole buffer > > allocated within the same memory channel? > > Most systems just interleave memory chips based on number of filled slots. > This is handled by BIOS before kernel even starts. > The DPDK has a number of memory channels parameter and what it does > is try and optimize memory allocation by spreading. > > Looks like you are inventing your own limited version of what memif does.