From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa0-f49.google.com (mail-oa0-f49.google.com [209.85.219.49]) by dpdk.org (Postfix) with ESMTP id 521F5333 for ; Tue, 1 Jul 2014 09:39:21 +0200 (CEST) Received: by mail-oa0-f49.google.com with SMTP id i7so9907917oag.8 for ; Tue, 01 Jul 2014 00:39:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=2ulVzocmyMCtG6PsXsw0FUdPDTkCR5arjN9Lwrun8QI=; b=GIUn9VLtxVTLsM13i/NC05LjWkVaXW6sa08b+RF+qOJI+AYUEd7YACQJtLviyhwG/O qiGVM2/ImKDXSWSjL7lI7nwv1pd+4WrtIse0eIsgskfQleY2M+BHTOdb/JrGtxKGhmm+ Tr/tMdlQo4IM6Hzehp/8wxQJkDUUmMVjsvRLkPsFML5/zxQgpntoS/m6ANyhZlwkw2dq m0vqC5mza1sXo66FGbPLWzOG4BqB7fM09N7LGm9fhUDwHaeGH1j5pxl/FJJQqTr+P1vJ L3Qt2IXTH3K85CNVBs/nLYLVtk6xw96JmSGi8zgfKei2FhzmZawsrnGpQy6fN+I/ERVL Vr6g== X-Gm-Message-State: ALoCoQmh6muq4onGZOpIa1zXDz1EB5VWWRJw4wAb9CTXc5xA6gcWeeaGaPvxqn06TMeO7TpX98sJ MIME-Version: 1.0 X-Received: by 10.182.66.79 with SMTP id d15mr47192843obt.49.1404200380557; Tue, 01 Jul 2014 00:39:40 -0700 (PDT) Received: by 10.202.202.137 with HTTP; Tue, 1 Jul 2014 00:39:40 -0700 (PDT) In-Reply-To: <59AF69C657FD0841A61C55336867B5B02CF117C1@IRSMSX103.ger.corp.intel.com> References: <59AF69C657FD0841A61C55336867B5B02CF117C1@IRSMSX103.ger.corp.intel.com> Date: Tue, 1 Jul 2014 10:39:40 +0300 Message-ID: From: Alex Markuze To: "Richardson, Bruce" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Memory Pinning. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2014 07:39:21 -0000 On Mon, Jun 30, 2014 at 7:55 PM, Richardson, Bruce < bruce.richardson@intel.com> wrote: > > -----Original Message----- > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Alex Markuze > > Sent: Monday, June 30, 2014 3:01 AM > > To: dev@dpdk.org > > Subject: [dpdk-dev] Memory Pinning. > > > > Hi, Guys. > > I have several newbie questions about the DPDK design I was hoping some > one > > could answer. > > > > Both in the RX and TX flow, the Buffer Memory must be pinned and not > > swappable. > > In RDMA, memory is explicitly registered and made pinned (to the limit > > defined @ /etc/security/limits.conf) .With regular sockets/kernel driver > > the NIC DMA's the buffer from/to the kernel which are by definition un > > swappable. > > > > So I'm guessing that at least the TX/RX buffers are mapped to kernel > space. > > > > My questions are 1. How are the buffers made unswappable ? Are they > shared > > with the kernel 2. When and Which buffers are mapped/unmapped to the > kernel > > space. 3. When are the buffers DMA mapped and by whom? > > The memory used is all hugepage memory and as such is not swappable by the > kernel, so remains in place for the duration of the application. At > initialization time, we query from the kernel via /proc the physical > address of the pages being used, and when sending buffers to the NIC we use > those physical addresses directly. > > Thanks for the clarification, the actual physical memory can be used in the write descriptor only when the iova is the same as the physical address. When IOMMU is enabled which AFAIK is enabled with deferred protection by default (intel_iommu=on) , each device will have its own notion of the iova (which can actually used for the DMA op) for the same physical address. So how does DPDK handle IOMMU currently? > > > > And another "bonus" Question. On TX flow I didn't find a way to receive a > > send completion. > > So how Can I know when its safe to modify the sent buffers (besides of > > waiting for the ring buffer to complete a full circle)? > > This will depend upon the configuration of the NIC on TX. By default when > using the fast-path we have the NIC only write-back confirmation of a > packet being sent every 32 packets. You can poll the ring for this > notification and which point you know all previous packets have been sent. > If you want to know on a per-packet basis as soon as the packet is sent, > you'll need to change the write-back threshold to write back every packet. > That will impact performance, though. Note, too, that there are no APIs > right now to query if a particular packet is sent, so you will have to > write the code to scan the TX rings directly yourself. > > /Bruce >