From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EB3B242B90; Wed, 24 May 2023 20:29:24 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 73F5940156; Wed, 24 May 2023 20:29:24 +0200 (CEST) Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by mails.dpdk.org (Postfix) with ESMTP id EA9E1400EF for ; Wed, 24 May 2023 20:29:22 +0200 (CEST) Received: by mail-pg1-f173.google.com with SMTP id 41be03b00d2f7-53404873a19so474960a12.3 for ; Wed, 24 May 2023 11:29:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20221208.gappssmtp.com; s=20221208; t=1684952962; x=1687544962; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=sC9rw981ELHRFKzON+TckhVENsXYRYZMWgP2YxGw8wU=; b=wyGXNlsIO+JksJIp7P9ISKgmofpUEm35qIqSx779+VYM5Dxsq9sJqZNgTGFDN7LDiP FGYuKII7arndNw8fCKHbBmiXNPIGVS7hRX1CUmZ3RVWqtJq2UQS9fhMIUiHdWwQTmCNr 1yfDGVQ3KLsmu1pmiFw/K0D29SF9hIFhva0klFl2goxYjPOtNSaX63ct+iHyhNte5k12 kbWyR027YzEySKJGSzeOjbQqpY9393aOrwT5FHk//MbudM0UOTwBesCTXtLvKg/6gEo2 3/LKtyigv8BidGlbVTta3x+krVRW568etRSE+WZQQma23XlRrc5LoxpzV3o/ndV0mG7J JYVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684952962; x=1687544962; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sC9rw981ELHRFKzON+TckhVENsXYRYZMWgP2YxGw8wU=; b=ad9ywWwRGpZlWOJz42HRdq6p/9gi+Ibs709JFYcGcFRfB7Ri/gFeaNgDnMw9/qb/f9 HxXAkgCiAKc8RCiRa+v6XnrqJKfIBbvfN8EZvVeK0rZoO2JcqVEepGxMCpFF1hZ8pQdP nz9VJMAqXEiPiD0sbhGhTMWkuNSwS1iyCSlcbKH2VQyFgGWVh9MAkTSQi8zv1EJVUjqx 1XNh8xcNcje5N/+nM6ySaTwo2xwq2YGqQcjmp8vnTCAWH/WWrF+sLPczo0Py6n/vK6ol 1ACupOqRJYFJPk8b90TgEh+CpFbv8wU3ZkcNxctmTta63/Vl8HmMlrFQAmpy2Rwu8Wz3 kxnA== X-Gm-Message-State: AC+VfDyW3IsMYkK3om8/sfVlvjViGRggnOT5HSMxc0oq0kJ+nD3jMsBL dTT3fLZvqR9xOG2InK4mTHRerA== X-Google-Smtp-Source: ACHHUZ4bKTlM6lreV2mRMlwUv4Gfs+sGx9i5BOO/NWN4vOFwO2GCGsN6gOPmvay/A3gka0ctU1ZyqA== X-Received: by 2002:a17:902:ee41:b0:1ae:626b:4771 with SMTP id 1-20020a170902ee4100b001ae626b4771mr17484561plo.36.1684952961895; Wed, 24 May 2023 11:29:21 -0700 (PDT) Received: from hermes.local (204-195-120-218.wavecable.com. [204.195.120.218]) by smtp.gmail.com with ESMTPSA id z5-20020a170903018500b001a045f45d49sm8979437plg.281.2023.05.24.11.29.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 May 2023 11:29:21 -0700 (PDT) Date: Wed, 24 May 2023 11:29:19 -0700 From: Stephen Hemminger To: Bruce Richardson Cc: Alireza Sanaee , Subject: Re: AF_XDP performance Message-ID: <20230524112919.64fc806c@hermes.local> In-Reply-To: References: <0fba676d-a49b-76fe-6714-0700d5b61749@qmul.ac.uk> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, 24 May 2023 17:36:32 +0100 Bruce Richardson wrote: > On Wed, May 24, 2023 at 01:32:17PM +0100, Alireza Sanaee wrote: > > Hi everyone, > > > > I was looking at this deck of slides > > https://www.dpdk.org/wp-content/uploads/sites/35/2020/11/XDP_ZC_PMD-1.pdf > > > > I tried to reproduce the results with the testpmd application. I am > > working with BlueField 2 NIC and I could sustain ~10Mpps with testpmd > > with AF_XDP, and about 20Mpps without AF_XDP on the RX drop experiment. I > > was wondering why AF_XDP so lower compared to PCI-e scenario given the > > fact that both cases are zero-cpy. Is it because of the frame size? > > > While I can't claim to explain all the differences, in short I believe the > AF_XDP version is just doing more work. With a native DPDK driver, the > driver takes the packet descriptors directly from the NIC RX ring and uses > the metadata to construct a packet mbuf, which is returned to the > application. > > With AF_XDP, however, the NIC descriptor ring is not directly accessible by > the app. Therefore the processing is (AFAIK): > * kernel reads NIC descriptor ring and processes descriptor > * kernel calls BPF program for the received packets to determine what > action to take, e.g. forward to socket > * kernel writes an AF_XDP descriptor to the AF_XDP socket RX ring > * application reads the AF_XDP ring entry written by the kernel and then > creates a DPDK mbuf to return to the application. > > There are also other considerations around potential cache locality of > descriptors too that could affect things, but I would expect the extra > descriptor processing work outlined above probably explains most of the > difference. > > Regards, > /Bruce There is also a context switch from kernel polling thread to the DPDK polling thread to consider. Plus the overhead of running the BPF program. The context switches mean that both the instruction and data cache is likely to have lots of misses. Remember on modern processes the limiting factor is usually memory performance from caching, not number of instructions.