From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <stephen@networkplumber.org> Received: from mail-pf0-f177.google.com (mail-pf0-f177.google.com [209.85.192.177]) by dpdk.org (Postfix) with ESMTP id 622792BFF for <users@dpdk.org>; Thu, 7 Sep 2017 03:47:54 +0200 (CEST) Received: by mail-pf0-f177.google.com with SMTP id g13so15494739pfm.2 for <users@dpdk.org>; Wed, 06 Sep 2017 18:47:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZnkWsBQfKCm0HR64RKg0Po2I2ASaZz7zMxlAkWMgPyM=; b=CCE1rF4b+ggk5dj0v4ctN9v8ZubcYBssqjJiQ8jZbkH1gIVuQDsJyT8wU3XEC0fAiZ VMVnm+nPz4c5uUkzSMrtDKBBRoDqPwmvRItIXpjIxV6MQgQIi0udi4M2EOJT7Mo0G7fn p512/rFAPGZbUwFwNVYUwkQH/xMD6bzCjHZG+5pmkKUt6VjW2tJ069k+wNMd89E6jwRn TdMu/fctnZIEwBR9dAi+3tUWV0I1fHe4cIHlGTKUwWjckQxZwPKXCCS2+iiC+5KZC1UJ Cp3xSpuiK0TPMJtNDARWOJ0VFgsAzE8Uc5fN6ui4TqTL7+4QT3Nn6zDebOBB4vsCJRST D/1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZnkWsBQfKCm0HR64RKg0Po2I2ASaZz7zMxlAkWMgPyM=; b=j+CRm54NyUrQAzjiNYFIBgkjhba6GXkb5HAcZDV9XuIqtaWXc2pR3nh3ipfMYtFMQl sDmlziRV22Ov0Ndevkw4J72bCM3sjHA+oCGqo+w+MSXGe7ERBMXHxpvGxHOg3icB7h/t P+55aXWYnP+Z23q/Ih/+5bycBiKHoL86pXmYVPbWlaGZ/JZ6SE2HlHtlqGesZVguZx8U f3gpQDBSHo2T6bOZ2xA6lmwINHHb0IklJK//VvcwoeT6eJhi7Q7Mi5al5P41G9PUQWAs Ec+kHIkgrmS1voUyZ8uruCjlG5/M/JjKtH/OYRSPxarYJnkWRb3b/HGSCrW738HzSGqi KuNA== X-Gm-Message-State: AHPjjUi6lk4CRuG11AuqOITr1d8TeyHVvCB2FIMMf45d48p5VkFZrDfM qO1X0eAw5uWoGgmV X-Google-Smtp-Source: ADKCNb7/WeqiSD6QWwEPVCNngxA9JS8PhsfEPEA6Pd3pQkql3RysJ4p+aBwqt2kCUdnBp2pvhTFPBA== X-Received: by 10.99.147.75 with SMTP id w11mr1133668pgm.308.1504748872937; Wed, 06 Sep 2017 18:47:52 -0700 (PDT) Received: from xeon-e3 (76-14-207-240.or.wavecable.com. [76.14.207.240]) by smtp.gmail.com with ESMTPSA id b68sm1353777pfk.23.2017.09.06.18.47.52 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 06 Sep 2017 18:47:52 -0700 (PDT) Date: Wed, 6 Sep 2017 18:47:45 -0700 From: Stephen Hemminger <stephen@networkplumber.org> To: Andrew Bainbridge <andbain@microsoft.com> Cc: "users@dpdk.org" <users@dpdk.org> Message-ID: <20170906184745.711bc096@xeon-e3> In-Reply-To: <HE1PR83MB0220FF34AE789E0245EA01ADAE910@HE1PR83MB0220.EURPRD83.prod.outlook.com> References: <HE1PR83MB0220FF34AE789E0245EA01ADAE910@HE1PR83MB0220.EURPRD83.prod.outlook.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-users] Low level understanding of mbufs needed. X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions <users.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/users>, <mailto:users-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/users/> List-Post: <mailto:users@dpdk.org> List-Help: <mailto:users-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/users>, <mailto:users-request@dpdk.org?subject=subscribe> X-List-Received-Date: Thu, 07 Sep 2017 01:47:54 -0000 On Mon, 4 Sep 2017 10:44:02 +0000 Andrew Bainbridge <andbain@microsoft.com> wrote: > I'm a newbie. I want to learn more about to use mbufs to achieve the best throughput. My application is something like a VPN server. In pseudo code: > > while 1: > pkt = recv() > if pkt.ip.daddr == CLIENT: > new_pkt = encap(pkt) > else: > new_pkt = decap(pkt) > send(new_pkt) > > Where encap() prepends an IP and UDP header, and decap() does the opposite. > > Most of each packet I send is the same as one I just received. Is it possible to do the send without having to allocate a new mbuf and memcpy into it? > > I want to learn more about how the system works at the low level. > > My guess of how it works is that the NIC reads in a packet from the Ethernet cable and writes it into its on-chip SRAM. Once it has enough data buffered, or enough time has elapsed it does a PCIe write request to copy the data into system RAM. The simplest scheme would be to have a single large circular buffer in system RAM and for the packets to be written nose-to-tail into that buffer. > Does DPDK do that? I guess not. I guess the supported cards all support scatter/gather, which AFAICT means the NICs are smart enough to understand an array of pointers to buffers. > > So what then? I have many 1500 byte buffers allocated, and I give the NIC an array of pointers to those buffers. The NIC then "scatters" the input stream into these buffers, one packet per buffer. > > I guess the best scheme for my application would be if I could tell the NIC to always leave 30 bytes or so of headroom on each packet, so that I can prepend the extra headers in the encap case. Can I request that when I configure the mbufs? > > If you can point me to some kind of tutorial or blog post that covers this area, that would also be helpful. > > Thanks, > Andrew You are heading off on a tangent. That is not how DPDK works. DPDK works like Linux and FreeBSD kernel. Either read the documentation or look at how examples work.