From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1862241C58 for ; Fri, 10 Feb 2023 03:38:45 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DD3EC40EE6; Fri, 10 Feb 2023 03:38:44 +0100 (CET) Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by mails.dpdk.org (Postfix) with ESMTP id 1DE3440EE3 for ; Fri, 10 Feb 2023 03:38:43 +0100 (CET) Received: by mail-pl1-f179.google.com with SMTP id w5so5058564plg.8 for ; Thu, 09 Feb 2023 18:38:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=WT64EiMjjwX3A7LoQBetwdCpIaC69A9INJ3pvcf6M30=; b=JxxL75liZbYrztPabX/jfcWad2pBkte8fe3oV1shUUM4LZkiHNrfyFkwAWHE3mdltO 4Swl2j/tMKSGs9K4y7B8yDoMcYbL44Xh8Xc1nwIe8Se/tAjyk7+uSPSxntXh/Lv4YUhe 919I7ooRSHf+aAJnAN3/F48ZYJ745gmmQu8nVqUPHimXl7Jszb2MH1Sx/WM2NRhXRKSO 2Ad4fGOk6PfHBrTZhsE6o9DlWA3m2gvHIGIhTfdsCjx20DRDe6/T2n9eVyHG2D+uSrVZ nOtckA7zKCh1qf1pGcLbavXc+d3gY2iIe0zbeBmLxBKhzG64emxLZCaPecyeH16wTTyp /mqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WT64EiMjjwX3A7LoQBetwdCpIaC69A9INJ3pvcf6M30=; b=Bo7FF+/hEuZ8ss8O8KFtwyZ8Plft1VlJb1ZzkWLEg0c7Uw4XvfT9QFy/71B9YFhr0U 7/lCFBjE77p/xRrZz0G/3rlBhA1y/lBX/ECf34xIx9vXpXtWCOMsIXYYsSOTfuUgkPcJ 2D2vVOiSElyuNWLIHOyYV2uQ+7PAcYNpuiBpwsTNoB0lhax1ypvRm+k6KsKzCPc7TyCK 4RJbTwYodXa3Jy6Y6PyEjyMQPekObafqTPtg8i70slYgUp9IZ4ycmJFtob+oWjIS6jfP zdI5VVq+Orb2kooDepPi+CxD0KXD4uJVFc6B7/Tfn8u1vT1mPxoRkk+tH66VFDCDslYY TD8w== X-Gm-Message-State: AO0yUKXOrapJ8/lEWXEsr1dRjsC8fAxyV+YmGqkbh3NlYkSZLeAW0atz oyZh88GRvvN9nz93MYHJGzirPb/fh+BPmL2ZuVk= X-Google-Smtp-Source: AK7set97tZh7I9pcn7JOi1//Cu0+n4j6ZhNrTxj44mKB6NIJHFjI9oGgoV8tXxFim/Gnd33bBjQ3Vw== X-Received: by 2002:a17:902:ebcf:b0:19a:7341:bf0a with SMTP id p15-20020a170902ebcf00b0019a7341bf0amr395403plg.49.1675996723197; Thu, 09 Feb 2023 18:38:43 -0800 (PST) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id p6-20020a170903248600b0019949fd956bsm2193392plw.178.2023.02.09.18.38.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Feb 2023 18:38:43 -0800 (PST) Date: Thu, 9 Feb 2023 18:38:41 -0800 From: Stephen Hemminger To: "Xiaoping Yan (NSB)" Cc: "users@dpdk.org" Subject: Re: cache miss increases when change rx descriptor from 512 to 2048 Message-ID: <20230209183841.4e30b2d0@hermes.local> In-Reply-To: <56bfdfd02a484084aceb37699b8fc043@nokia-sbell.com> References: <4b132ffd05594663b5abb71f42e6f97f@nokia-sbell.com> <20230209083815.753f41ea@hermes.local> <56bfdfd02a484084aceb37699b8fc043@nokia-sbell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Fri, 10 Feb 2023 01:59:02 +0000 "Xiaoping Yan (NSB)" wrote: > Hi Stephen, >=20 > Can you help to elaborate a bit on " since descriptors are used LIFO the = rx descriptor and mbuf will not be in cache."? > Last in means the mbuf just used and freed? And FO means such mbuf (LI) w= ill be used first, would it mean that it has good chance it is still in cac= he? >=20 > Thank you a lot. >=20 > Br, Xiaoping >=20 > -----Original Message----- > From: Stephen Hemminger =20 > Sent: 2023=E5=B9=B42=E6=9C=8810=E6=97=A5 0:38 > To: Xiaoping Yan (NSB) > Cc: users@dpdk.org > Subject: Re: cache miss increases when change rx descriptor from 512 to 2= 048 >=20 > On Thu, 9 Feb 2023 03:58:56 +0000 > "Xiaoping Yan (NSB)" wrote: >=20 > > Hi experts, > >=20 > > I had a traffic throughput test for my dpdk application, with same soft= ware and test case, only difference is the number of rx/tx descriptor: > > Rx/tx descriptor 512, test result 3.2mpps Rx/tx descriptor 2048, test=20 > > result 3mpp From perf data, rx descriptor 2048 case has more cache=20 > > miss, and lower instruction per cycle Perf for 512 rx descriptor > > 114289237792 cpu-cycles > > 365408402395 instructions # 3.20 insn per c= ycle > > 74186289932 branches > > 36020793 branch-misses # 0.05% of all bra= nches > > 1298741388 bus-cycles > > 3413460 cache-misses # 0.723 % of all c= ache refs > > 472363654 cache-references > > Perf for 2048 rx descriptor: > > 57038451185 cpu-cycles > > 173805485573 instructions # 3.05 insn per c= ycle > > 35289607389 branches > > 15418885 branch-misses # 0.04% of all bra= nches > > 648164239 bus-cycles > > 13170596 cache-misses # 1.702 % of all c= ache refs > > 773765263 cache-references > >=20 > > I understand it means more rx descriptor somehow causes more cache miss= and then less instruction per cycle, so lower performance. > >=20 > > Any one observe similar results? > > Any idea to mitigate (or investigate further) the impact? (we want to=20 > > use 2048 to better tolerate some jitter/burst) Any comment? > >=20 > > Thank you. > >=20 > > Br, Xiaoping > > =20 >=20 > If number of RX descriptors is small, there is a higher chance that when = the device driver walks the descriptor table or is using the resulting mbuf= that the data is still in cache. >=20 > With large number of descriptors, since descriptors are used LIFO the rx = descriptor and mbuf will not be in cache. The receive descriptors are a ring, so the mbuf returned by the hardware is= the oldest one that was queued. My mistake, that would be FIFO. So the oldest mbuf (and associated ring data) will be the one the driver us= es.