From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wg0-f44.google.com (mail-wg0-f44.google.com [74.125.82.44]) by dpdk.org (Postfix) with ESMTP id 764F05A52 for ; Tue, 21 Apr 2015 11:47:25 +0200 (CEST) Received: by wgyo15 with SMTP id o15so207475396wgy.2 for ; Tue, 21 Apr 2015 02:47:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=i5pPdhiqUNbddAHyyTSxRS6xnmfrgDaSkm42qho/BOo=; b=csMBl2Usvw9YwKWr6+Mh/y+FXaB+McA/iGZQdfNfNBpo7TzsU1KhNeVtQlempH2wct +Q9hdPtdw7ruehquZz+zThpeLYqi8z0CHSbUPhIvOLRXqmWYQ6iY8a/sHqX6LGY0lqxB VXpwIKKZ5lV3n8RgHNt5m5RIOKXAcKjbXFj0o7UXj7sagKmXCR6oYXqORnf0JMywAntY iESTha33l+zCGvIEEwzHQfghzG5qNKYtlw6wzregLgn86zUhUnW8G4Q/PD7eGfu6VPv/ Y6aG527yH+vKNRNDX9aN39gzRU1Djkq3+lTysTYJq2qHFzMJTGsJvl/hIJHX2fV928Xb T/XQ== X-Gm-Message-State: ALoCoQm4/otwiPgOcBOtCzGOgaaI8SP0CD+y9cO93uDmZHdGm6tzfXFBlSHhXFzU5DXv0yUwHL9R X-Received: by 10.180.83.193 with SMTP id s1mr4117889wiy.22.1429609645209; Tue, 21 Apr 2015 02:47:25 -0700 (PDT) Received: from [10.0.0.166] ([212.143.139.214]) by mx.google.com with ESMTPSA id ln8sm1835353wjc.18.2015.04.21.02.47.24 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 21 Apr 2015 02:47:24 -0700 (PDT) Message-ID: <55361CAA.7000202@cloudius-systems.com> Date: Tue, 21 Apr 2015 12:47:22 +0300 From: Vlad Zolotarov User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Bruce Richardson References: <5534CFFF.7000404@cloudius-systems.com> <20150420105020.GB9280@bricha3-MOBL3> <55360F9C.7070601@cloudius-systems.com> <20150421092748.GB5360@bricha3-MOBL3> In-Reply-To: <20150421092748.GB5360@bricha3-MOBL3> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] DCA X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2015 09:47:25 -0000 On 04/21/15 12:27, Bruce Richardson wrote: > On Tue, Apr 21, 2015 at 11:51:40AM +0300, Vlad Zolotarov wrote: >> >> On 04/20/15 13:50, Bruce Richardson wrote: >>> On Mon, Apr 20, 2015 at 01:07:59PM +0300, Vlad Zolotarov wrote: >>>> Hi, >>>> I would like to ask if there is any reason why DPDK doesn't have support for >>>> DCA feature? >>>> >>>> thanks, >>>> vlad >>> With modern platforms with DDIO the data written by the NIC automatically goes >>> into the cache of the CPU without us needing to use DCA. >> Thanks for a reply, Bruce. >> One question though. According to DDIO documentation it only affects the >> CPUs "local" relatively to the NIC. DCA, on the other hand may be configured >> to work with any CPU. Modern platforms usually have a few NUMA nodes and >> requirement of binding network handling threads only to CPUs "local" to the >> NIC is very limiting. >> >> Could u, pls., comment on this? >> >> thanks in advance, >> vlad >> > My main comment is that yes, you are correct. DDIO only works with the local > socket, while DCA can be made to work with remote sockets. If you need to do > polling on a device from a remote socket you may need to look at DCA. > > Can you perhaps comment on the use-case where you find this binding limiting? Modern > platforms have multiple NUMA nodes, but they also generally have PCI slots > connected to those multiple NUMA nodes also, so that you can have your NIC ports > similarly NUMA partitionned? The immediate example where this could be problematic is an AWS Guest with Enhanced Netowrking case: in c3.8xlarge instance u get a 2 NUMA nodes, 32 CPU cores and u can bind as many 82599 Intel VFs as u need, each providing 4 Rx and 4 Tx queues. AFAIR nothing is promised about the locality of PFs VFs belong to. To utilize all CPUs we'll need 4 or 8 VFs depending on the queues layout we decide (a separate CPU for each queue or a separate CPU for each Rx + Tx queue pair). In this case u may get absolutely different NUMA layouts: - all VFs reside on the same PF: half of the queues will be remote to one of the NUMA node or all of them are remote to all CPUs. - VFs come from two PFs which may reside in the same NUMA nodes as CPUs or not... - VFs come from more than two different PFs... So, in the above example DCA would cover all our needs while DDIO won't be able to cover them in most of the cases. > > /Bruce