From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f176.google.com (mail-wi0-f176.google.com [209.85.212.176]) by dpdk.org (Postfix) with ESMTP id DAB21CF9 for ; Wed, 26 Aug 2015 19:07:54 +0200 (CEST) Received: by wicja10 with SMTP id ja10so50771171wic.1 for ; Wed, 26 Aug 2015 10:07:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=mPj+9N2BtRfo5eujCTrfVqAh3WivuObeD1i+R25o5wM=; b=I35SlcR7DYeUJsOOjM3rtqpZR95hrne3uURZNYxQjjtJOY4qVdLfqLYcb21qoVOPXz sABeqVyypyzKTJL8dgmlOV84s8f03m8WXgj3wrOF1uAVZsURckiJ9RgKBiaIb7jAeri1 EaRKb2kb+mtcYPB0gTzDTtwz9Ry0epuGFxD+EQhLGc953aBH9/tF/S/h55M6Nh2GDEDB sBr3kv6J/9RPkuzRiIX1be4Agl9xjp3XYaZaMD7x/TU6qHp1YbKejp7KI6rOCwToZfsK /f9QjcEuZyp5ddNVzTmIlLF8GTJRYnVIZP8zvzk9IUashLUfZfLiFaHOX2ALcz+SUFk5 +F7A== X-Gm-Message-State: ALoCoQmGJ+9Uevzq+Hv/sNvc4iBRIWVuwOdykxLVq2lRxnr5QkPeHuBgpiU5VmUJIK+YEbgGU4wY X-Received: by 10.194.63.42 with SMTP id d10mr63918079wjs.92.1440608874641; Wed, 26 Aug 2015 10:07:54 -0700 (PDT) Received: from [192.168.0.101] ([90.152.119.35]) by smtp.googlemail.com with ESMTPSA id v9sm4747477wjq.41.2015.08.26.10.07.53 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 26 Aug 2015 10:07:54 -0700 (PDT) Message-ID: <55DDF26A.3000804@linaro.org> Date: Wed, 26 Aug 2015 18:07:54 +0100 From: Zoltan Kiss User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.8.0 MIME-Version: 1.0 To: "Traynor, Kevin" , "dev@dpdk.org" , "dev@openvswitch.org" References: <55D76854.5010306@linaro.org> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [ovs-dev] OVS-DPDK performance problem on ixgbe vector PMD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 17:07:55 -0000 Hi, On 24/08/15 12:43, Traynor, Kevin wrote: > >> -----Original Message----- >> From: dev [mailto:dev-bounces@openvswitch.org] On Behalf Of Zoltan Kiss >> Sent: Friday, August 21, 2015 7:05 PM >> To: dev@dpdk.org; dev@openvswitch.org >> Cc: Richardson, Bruce; Ananyev, Konstantin >> Subject: [ovs-dev] OVS-DPDK performance problem on ixgbe vector PMD >> >> Hi, >> >> I've set up a simple packet forwarding perf test on a dual-port 10G >> 82599ES: one port receives 64 byte UDP packets, the other sends it out, >> one core used. I've used latest OVS with DPDK 2.1, and the first result >> was only 13.2 Mpps, which was a bit far from the 13.9 I've seen last >> year with the same test. The first thing I've changed was to revert back >> to the old behaviour about this issue: >> >> http://permalink.gmane.org/gmane.comp.networking.dpdk.devel/22731 >> >> So instead of the new default I've passed 2048 + RTE_PKTMBUF_HEADROOM. >> That increased the performance to 13.5, but to figure out what's wrong >> started to play with the receive functions. First I've disabled vector >> PMD, but ixgbe_recv_pkts_bulk_alloc() was even worse, only 12.5 Mpps. So >> then I've enabled scattered RX, and with >> ixgbe_recv_pkts_lro_bulk_alloc() I could manage to get 13.98 Mpps, which >> is I guess as close as possible to the 14.2 line rate (on my HW at >> least, with one core) >> Does anyone has a good explanation about why the vector PMD performs so >> significantly worse? I would expect that on a 3.2 GHz i5-4570 one core >> should be able to reach ~14 Mpps, SG and vector PMD shouldn't make a >> difference. > > I've previously turned on/off vectorisation and found that for tx it makes > a significant difference. For Rx it didn't make a much of a difference but > rx bulk allocation which gets enabled with it did improve performance. > > Is there is something else also running on the current pmd core? did you > try moving it to another? I've tied the pmd to the second core, as far as I can see from top and profiling outputs hardly anything else runs there. Also, did you compile OVS with -O3/-Ofast, they > tend to give a performance boost. Yes > > Are you hitting 3.2 GHz for the core with the pmd? I think that is only > with turbo boost, so it may not be achievable all the time. The turbo boost freq is 3.6 GHz. > >> I've tried to look into it with oprofile, but the results were quite >> strange: 35% of the samples were from miniflow_extract, the part where >> parse_vlan calls data_pull to jump after the MAC addresses. The oprofile >> snippet (1M samples): >> >> 511454 19 0.0037 flow.c:511 >> 511458 149 0.0292 dp-packet.h:266 >> 51145f 4264 0.8357 dp-packet.h:267 >> 511466 18 0.0035 dp-packet.h:268 >> 51146d 43 0.0084 dp-packet.h:269 >> 511474 172 0.0337 flow.c:511 >> 51147a 4320 0.8467 string3.h:51 >> 51147e 358763 70.3176 flow.c:99 >> 511482 2 3.9e-04 string3.h:51 >> 511485 3060 0.5998 string3.h:51 >> 511488 1693 0.3318 string3.h:51 >> 51148c 2933 0.5749 flow.c:326 >> 511491 47 0.0092 flow.c:326 >> >> And the corresponding disassembled code: >> >> 511454: 49 83 f9 0d cmp r9,0xd >> 511458: c6 83 81 00 00 00 00 mov BYTE PTR [rbx+0x81],0x0 >> 51145f: 66 89 83 82 00 00 00 mov WORD PTR [rbx+0x82],ax >> 511466: 66 89 93 84 00 00 00 mov WORD PTR [rbx+0x84],dx >> 51146d: 66 89 8b 86 00 00 00 mov WORD PTR [rbx+0x86],cx >> 511474: 0f 86 af 01 00 00 jbe 511629 >> >> 51147a: 48 8b 45 00 mov rax,QWORD PTR [rbp+0x0] >> 51147e: 4c 8d 5d 0c lea r11,[rbp+0xc] >> 511482: 49 89 00 mov QWORD PTR [r8],rax >> 511485: 8b 45 08 mov eax,DWORD PTR [rbp+0x8] >> 511488: 41 89 40 08 mov DWORD PTR [r8+0x8],eax >> 51148c: 44 0f b7 55 0c movzx r10d,WORD PTR [rbp+0xc] >> 511491: 66 41 81 fa 81 00 cmp r10w,0x81 >> >> My only explanation to this so far is that I misunderstand something >> about the oprofile results. >> >> Regards, >> >> Zoltan >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> http://openvswitch.org/mailman/listinfo/dev