From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 7820145C16; Wed, 30 Oct 2024 17:06:48 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 672D940299; Wed, 30 Oct 2024 17:06:48 +0100 (CET) Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by mails.dpdk.org (Postfix) with ESMTP id 1827140281 for ; Wed, 30 Oct 2024 17:06:46 +0100 (CET) Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-71e983487a1so5015143b3a.2 for ; Wed, 30 Oct 2024 09:06:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20230601.gappssmtp.com; s=20230601; t=1730304406; x=1730909206; darn=dpdk.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=CFoO9/mZ4LjlZhZlxyvgf4o8oqa8gd5U+ip5hrMShLI=; b=gBwGUl1u3uxiWdL8dfgEf4I00QjUeyLPWcewCex+K+vk3uTN+3kYxD7R8ggG9D+UTI ZwvMqSJsv3aDNCzDUl+yBXhb8u2GJeuXThUexf9ElvXUOewDw7MPqmGlAReA9G6ZLhHz SFntA4P/7bA1B/jw+0LXLIKy1Ez7DxJiy3XAnFX9YZ6wg12vgV4C3JWCWxrwRfFUHqjd 7u3pbaxTJWOCZa5CYu9pO4HalHlCXHTnioINo/imcszvBJFhl7sPXgeivlG/gGurkTfl zC8DeKOfW6SpZtUiq8UGs1C8olvwH/bZK5xHdgNrGwQzkBpSXAJqn3K+sQYm62rgN8y5 nGgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730304406; x=1730909206; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CFoO9/mZ4LjlZhZlxyvgf4o8oqa8gd5U+ip5hrMShLI=; b=ul4KTyJY/SYQe5hZ/4z23WfkzootsUekntBCq9/fh1TeZbh1dPoi/cC+YStJ8NPFeH wIrunaBCEnsDu5CMocKgxTUADlNzwejfyE/4m+S7rtPJNTCk3TcoBTDkt3LMELtrOfTu 5LHIxBK6xYHaaxCLWQP/hSjm7G8kZyPzqUk2Q+CB+dToFcBx157dOuOZFJnGq7Jf9fos 0I59cbOx1Vrc6YGaQoLlJqH4fplyWRzdTJNdxwHIJovDXr6rgMAu5DG89+bVZih3qZU/ hWpEOjy08nsBX3wPGJTu+u5yKoJrMTENC8jt8dCerk9uSnsC1GYYkzxo7AMKHkpnebEJ q3IA== X-Forwarded-Encrypted: i=1; AJvYcCXndOWadoD7rd1OikamfLSspk/S1z+NV06KwRJ2Rv1tQ0poQYOXMN4WVtfx4h933sj7IoY=@dpdk.org X-Gm-Message-State: AOJu0YxIBtsHpHIYBzSkumgI9dcUwDNeDv8dgz9tz1dMdfjEX9fJfaSL 6qKi1y3ArsQNnkNtclDeoTyt49uhs+5EdrrVqjBwHWa+bAA5FLavUWc1FJvWPkw= X-Google-Smtp-Source: AGHT+IHUG8Nycu2S+ooQVsxjfz70Qo+/Gy7yVLyvy57w35N7hRfQk3MyHw9XIAP4udqqL8mEt7DIHA== X-Received: by 2002:a05:6a00:1488:b0:71e:7c25:8217 with SMTP id d2e1a72fcca58-720b9de1487mr203059b3a.25.1730304405624; Wed, 30 Oct 2024 09:06:45 -0700 (PDT) Received: from hermes.local (204-195-96-226.wavecable.com. [204.195.96.226]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72057939b82sm9475770b3a.81.2024.10.30.09.06.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 30 Oct 2024 09:06:45 -0700 (PDT) Date: Wed, 30 Oct 2024 09:06:43 -0700 From: Stephen Hemminger To: =?UTF-8?B?THVrw6HFoSDFoGnFoW1pxaE=?= Cc: Morten =?UTF-8?B?QnLDuHJ1cA==?= , anatoly.burakov@intel.com, ian.stokes@intel.com, dev@dpdk.org Subject: Re: [PATCH] net: increase the maximum of RX/TX descriptors Message-ID: <20241030090643.66af553f@hermes.local> In-Reply-To: <75463f4f-4139-4a53-9e63-05fe4cccb74f@cesnet.cz> References: <20241029124832.224112-1-sismis@cesnet.cz> <98CBD80474FA8B44BF855DF32C47DC35E9F845@smartserver.smartshare.dk> <20241030082020.2fe8eadb@hermes.local> <75463f4f-4139-4a53-9e63-05fe4cccb74f@cesnet.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org On Wed, 30 Oct 2024 16:40:10 +0100 Luk=C3=A1=C5=A1 =C5=A0i=C5=A1mi=C5=A1 wrote: > On 30. 10. 24 16:20, Stephen Hemminger wrote: > > On Wed, 30 Oct 2024 14:58:40 +0100 > > Luk=C3=A1=C5=A1 =C5=A0i=C5=A1mi=C5=A1 wrote: > > =20 > >> On 29. 10. 24 15:37, Morten Br=C3=B8rup wrote: =20 > >>>> From: Lukas Sismis [mailto:sismis@cesnet.cz] > >>>> Sent: Tuesday, 29 October 2024 13.49 > >>>> > >>>> Intel PMDs are capped by default to only 4096 RX/TX descriptors. > >>>> This can be limiting for applications requiring a bigger buffer > >>>> capabilities. The cap prevented the applications to configure > >>>> more descriptors. By bufferring more packets with RX/TX > >>>> descriptors, the applications can better handle the processing > >>>> peaks. > >>>> > >>>> Signed-off-by: Lukas Sismis > >>>> --- =20 > >>> Seems like a good idea. > >>> > >>> Have the max number of descriptors been checked with the datasheets f= or all the affected NIC chips? > >>> =20 > >> I was hoping to get some feedback on this from the Intel folks. > >> > >> But it seems like I can change it only for ixgbe (82599) to 32k > >> (possibly to 64k - 8), others - ice (E810) and i40e (X710) are capped = at > >> 8k - 32. > >> > >> I neither have any experience with other drivers nor I have them > >> available to test so I will let it be in the follow-up version of this > >> patch. > >> > >> Lukas > >> =20 > > Having large number of descriptors especially at lower speeds will > > increase buffer bloat. For real life applications, do not want increase > > latency more than 1ms. > > > > 10 Gbps has 7.62Gbps of effective bandwidth due to overhead. > > Rate for 1500 MTU is 7.62Gbs / (1500 * 8) =3D 635 K pps (i.e 1.5 us per= packet) > > A ring of 4096 descriptors can take 6 ms for full size packets. > > > > Be careful, optimizing for 64 byte benchmarks can be disaster in real w= orld. > > =20 > Thanks for the info Stephen, however I am not trying to optimize for 64=20 > byte benchmarks. The work has been initiated by an IO problem and Intel=20 > NICs. Suricata IDS worker (1 core per queue) received a burst of packets= =20 > and then sequentially processes them one by one. Well it seems like=20 > having a 4k buffers it seems to not be enough. NVIDIA NICs allow e.g.=20 > 32k descriptors and it works fine. In the end it worked fine when ixgbe=20 > descriptors were increased as well. I am not sure why AF-Packet can=20 > handle this much better than DPDK, AFP doesn't have crazy high number of= =20 > descriptors configured <=3D 4096, yet it works better. At the moment I=20 > assume there is an internal buffering in the kernel which allows to=20 > handle processing spikes. >=20 > To give more context here is the forum discussion -=20 > https://forum.suricata.io/t/high-packet-drop-rate-with-dpdk-compared-to-a= f-packet-in-suricata-7-0-7/4896 >=20 >=20 >=20 I suspect AF_PACKET provides an intermediate step which can buffer more or spread out the work.