From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ADAF141C53 for ; Thu, 9 Feb 2023 17:38:20 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 77A93410EA; Thu, 9 Feb 2023 17:38:20 +0100 (CET) Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by mails.dpdk.org (Postfix) with ESMTP id 634D140EDC for ; Thu, 9 Feb 2023 17:38:18 +0100 (CET) Received: by mail-pj1-f54.google.com with SMTP id bx22so2611957pjb.3 for ; Thu, 09 Feb 2023 08:38:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=kK1U7EeAGsoWiKxaY1r1lvUb6H3SWqG1fPNDVAAOLwQ=; b=vJ6DQ6/BaFhls0QAq3RjeSEJAN15rUfNE1vcyry9StISYeNKmgYvftfCLnTmlIossn hGagEomlSNbC8+1bWsoOZxvy4HB0Nd/PboDjxikLzvw48lxaKDFOzF+dWYyF4tvn+ZO2 pwcKTBLX7R/QYYTLb9kehPENtFz9M9jX/LsIiEnOBQFgMfW85sRPWkX7ZIYuogtW9HMo SDEzMuw+vCjq2IHd31wEbxQAFcHZ+H5Hpm5MtY3aZCUTtF3qZy9wqg2OWnfst42A/Mq8 EyvlaWXI1+hZDMqvpyLFrA/7CFrieg8DSa6vyWnr8/201BoTov1n40WcEB/BWPMANoWT SdEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kK1U7EeAGsoWiKxaY1r1lvUb6H3SWqG1fPNDVAAOLwQ=; b=f5nXpW0wr3CMffXm1ArNPYqR2VQme92SZjjsBqNqXRnQdz9srNQawK+uaDU9UzuFKL 4F7584PIKBcJX+kq+9mWrqeO2xCE6v3rCw/Jit+dl4TCIIZDr/JnJfxAvBkM2Zw0/rEG UlRAPpzjqeWAQIqxW+rvdUaQLuQJ9/OWLdSlgRCgghxtJXT8tNy+q9jFV+EweoRmIIFb u7TPoMOEY6Let4E1xsT3j2aMDfJRwNApuykBKr1FUDps6tE+FOd1p4QY7H/zLb3E9oMC bZDg6JIA3uTHQoBLR+uXR5sZzRzZTg+bQ33qtV52N3I/e+h5jcO+e4FXem4Z5jVPLzMy KRuA== X-Gm-Message-State: AO0yUKX9NdiiOpJb3qmTAD/Mlg+7vx9f2JHoj1MTjNFXC46kgROSdnNd PiO7E985NWrG425k9iVm6crzZw== X-Google-Smtp-Source: AK7set8q7qVPQ60P38/vo4KWHx6aupJqKLQZVS5sTLQLxhor2MYsfCY+9fodmaq08/Aj0mn9HZT8Jw== X-Received: by 2002:a17:903:182:b0:196:37db:b4b7 with SMTP id z2-20020a170903018200b0019637dbb4b7mr14248875plg.62.1675960697493; Thu, 09 Feb 2023 08:38:17 -0800 (PST) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id t3-20020a1709027fc300b0019949a3be88sm1695977plb.138.2023.02.09.08.38.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Feb 2023 08:38:17 -0800 (PST) Date: Thu, 9 Feb 2023 08:38:15 -0800 From: Stephen Hemminger To: "Xiaoping Yan (NSB)" Cc: "users@dpdk.org" Subject: Re: cache miss increases when change rx descriptor from 512 to 2048 Message-ID: <20230209083815.753f41ea@hermes.local> In-Reply-To: <4b132ffd05594663b5abb71f42e6f97f@nokia-sbell.com> References: <4b132ffd05594663b5abb71f42e6f97f@nokia-sbell.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org On Thu, 9 Feb 2023 03:58:56 +0000 "Xiaoping Yan (NSB)" wrote: > Hi experts, > > I had a traffic throughput test for my dpdk application, with same software and test case, only difference is the number of rx/tx descriptor: > Rx/tx descriptor 512, test result 3.2mpps > Rx/tx descriptor 2048, test result 3mpp > From perf data, rx descriptor 2048 case has more cache miss, and lower instruction per cycle > Perf for 512 rx descriptor > 114289237792 cpu-cycles > 365408402395 instructions # 3.20 insn per cycle > 74186289932 branches > 36020793 branch-misses # 0.05% of all branches > 1298741388 bus-cycles > 3413460 cache-misses # 0.723 % of all cache refs > 472363654 cache-references > Perf for 2048 rx descriptor: > 57038451185 cpu-cycles > 173805485573 instructions # 3.05 insn per cycle > 35289607389 branches > 15418885 branch-misses # 0.04% of all branches > 648164239 bus-cycles > 13170596 cache-misses # 1.702 % of all cache refs > 773765263 cache-references > > I understand it means more rx descriptor somehow causes more cache miss and then less instruction per cycle, so lower performance. > > Any one observe similar results? > Any idea to mitigate (or investigate further) the impact? (we want to use 2048 to better tolerate some jitter/burst) > Any comment? > > Thank you. > > Br, Xiaoping > If number of RX descriptors is small, there is a higher chance that when the device driver walks the descriptor table or is using the resulting mbuf that the data is still in cache. With large number of descriptors, since descriptors are used LIFO the rx descriptor and mbuf will not be in cache.