From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by dpdk.org (Postfix) with ESMTP id E77BD2BA1 for ; Fri, 24 Feb 2017 22:09:35 +0100 (CET) Received: by mail-wm0-f45.google.com with SMTP id v77so23562325wmv.1 for ; Fri, 24 Feb 2017 13:09:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=schaman-hu.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=BGohuRnlsBUFCU5Y4rrKBCVNT+lK7S3IAKVhrgEgOEE=; b=lBwGSq+uQlehkMfiOwVeGU0pHmAEuAwdbcsN0HMQL9BtkqOnJhKBtUGq5gDemxc6Fv rzT+WPHtYKoA7sfmDM/50LQBiKF5MGk3PyopaT4K85cRVRcCIN+6kG8HvmJcsm2BZIts fT2hVwXwoQDDNsibzlngT6Wdepw/5MUfyJAZzR69VfYoTGqUtjFyytG9Ks9UvLxUJj6z 76BxdaFuYTWOsqd1ALGf2Ob1qBZeh/bJfEFLzUYPH6p0Pe+ySpuiErnXv67npsJ69Un7 j4LJh3scgXUUMWF5ThQ300k/WaaFGpRSYnHp8Nbi00yIlhYBf8gPeruLGitJ+jC4/rNL 0mIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=BGohuRnlsBUFCU5Y4rrKBCVNT+lK7S3IAKVhrgEgOEE=; b=iZv7ozMaByhlWzwquXYed78XFXau1xVKhkG1iIDkSQV4zm9VTRGQAnTt45I2kYpDTQ 57lyxMOTVBS+zbawcGaeksBP3dghFhR6tZ1Eu2nn9iNXPitamjr1HpN3iYimff3k1pc4 KNLhpY8x3/6/HWuNQc9iYfHFsEoGeGE8lc0RpOkCDuG9GJO7OFiQ+ynfkOORopdKuYe2 2Ac1k59b90b/5NKb1k6TDcpxiyIWJBEDdMeAwa7H63iw9qg893ouWbMmLGo8HqVfoTQM kwjz08cOeqBLIeF1xjDWhAuuOMgwhPHGkSoB5GAzdbyOsEKzlw4MosqN+qyqA97ti54D t6ww== X-Gm-Message-State: AMke39lAlyG9QaL3/s9Iz612nUUf+D/ovj6GZWU3rLJ7pH/0hgMm1sgxiSH9hbwrAErg9Q== X-Received: by 10.28.93.68 with SMTP id r65mr4687975wmb.133.1487970575289; Fri, 24 Feb 2017 13:09:35 -0800 (PST) Received: from [192.168.0.100] (catv-213-222-155-91.catv.broadband.hu. [213.222.155.91]) by smtp.googlemail.com with ESMTPSA id w17sm11788718wra.28.2017.02.24.13.09.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Feb 2017 13:09:34 -0800 (PST) To: "Dumitrescu, Cristian" , "dev@dpdk.org" References: <3EB4FA525960D640B5BDFFD6A3D891265274E7BB@IRSMSX108.ger.corp.intel.com> From: Zoltan Kiss Message-ID: Date: Fri, 24 Feb 2017 22:09:33 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0 MIME-Version: 1.0 In-Reply-To: <3EB4FA525960D640B5BDFFD6A3D891265274E7BB@IRSMSX108.ger.corp.intel.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] rte_sched library performance question X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Feb 2017 21:09:36 -0000 On 16/02/17 20:08, Dumitrescu, Cristian wrote: > Hi Zoltan, > >> -----Original Message----- >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss >> Sent: Thursday, February 16, 2017 3:14 PM >> To: dev@dpdk.org >> Subject: [dpdk-dev] rte_sched library performance question >> >> Hi, >> >> I'm experimenting a little bit with the scheduler library, and I got some >> performance numbers which seems to be worse than what I've expected. >> I'm sending 64 bytes packets on a 10G interface to a separate thread, and >> my simple test program (based on the qos_sched example) does the >> following: >> >> while (1) { >> uint16_t ret = rte_ring_sc_dequeue_burst(it.ring, >> (void**)flushbatch, FLUSH_SIZE); >> rte_mbuf** t = flushbatch; >> >> if (!ret) { >> /* This call is necessary to make sure the TX completed >> mbuf's >> * are returned to the pool even if there is nothing to >> * transmit */ >> rte_eth_tx_burst(it.portid, lcore, t, 0); >> continue; >> } >> rte_sched_port_enqueue(it.port, flushbatch, ret); >> ret = rte_sched_port_dequeue(it.port, flushbatch, FLUSH_SIZE); > Looks to me like the scheduler dequeue burst is equal to the enqueue burst size of FLUSH_SIZE, right? > In this case, you are always dequeueuing the exact packets that you just enqueued, and the scheduler dequeue needs to work really hard to find exactly those FLUSH_SIZE queues that each one have a single packet at this point. > > This is wht the enqueue burst size should be bigger than the dequeue burst size. Basically, you add some water into the reservoir up to a reasonable fill level before you start pouring it in your glass if you want to fill the glass quickly. > > Typical values used: > -for vector PMD: (enqueue = 32, dequeue = 24), (32, 28), (32, 16), etc > -for scalar PMD: (64, 48), (64, 32), ... We used (256, 248) for VPP Thanks, it helped my case too. Btw. it would be good do link this document somewhere in the DPDK docs, as it contains a lot of good information about the scheduler: https://networkbuilders.intel.com/docs/Network_Builders_RA_NFV_QoS_Aug2014.pdf > >> while (ret) { >> uint16_t n = rte_eth_tx_burst(it.portid, lcore, t, ret); >> /* we cannot drop the packets, so re-send */ >> /* update number of packets to be sent */ >> ret -= n; >> t = &t[n]; >> }; >> } >> >> I run this on a separate thread, another one doing rx and feeding the >> packets to the ring. When I comment out the enqueue and dequeue part in >> the >> code (reducing it to simple l2fwd), I can forward the entire ~14 Mpps >> traffic, whilst with the scheduler enabled I can only reach ~5.4 Mpps at >> best. I've tried with a single pipe or with 4k (used rand() to randomly >> distribute between pipe, everything else (class etc) was set to 0), didn't >> make a difference. Is this expected? I'm running this on a Xeon E5-2630 0 @ >> 2.30GHz >> >> I've used the following configuration: >> >> ; port configuration [port] >> >> [port] >> frame overhead = 24 >> number of subports per port = 1 >> number of pipes per subport = 1024 >> queue sizes = 64 64 64 64 >> >> ; Subport configuration >> >> [subport 0] >> tb rate = 1250000000; Bytes per second >> tb size = 1000000000; Bytes >> tc 0 rate = 1250000000; Bytes per second >> tc 1 rate = 1250000000; Bytes per second >> tc 2 rate = 1250000000; Bytes per second >> tc 3 rate = 1250000000; Bytes per second >> tc period = 10; Milliseconds >> tc oversubscription period = 1000; Milliseconds >> >> pipe 0-1024 = 0; These pipes are configured with pipe profile 0 >> >> ; Pipe configuration >> >> [pipe profile 0] >> tb rate = 1250000000; Bytes per second >> tb size = 1000000000; Bytes >> >> tc 0 rate = 1250000000; Bytes per second >> tc 1 rate = 1250000000; Bytes per second >> tc 2 rate = 1250000000; Bytes per second >> tc 3 rate = 1250000000; Bytes per second >> tc period = 10; Milliseconds >> >> tc 0 oversubscription weight = 1 >> tc 1 oversubscription weight = 1 >> tc 2 oversubscription weight = 1 >> tc 3 oversubscription weight = 1 >> >> tc 0 wrr weights = 1 1 1 1 >> tc 1 wrr weights = 1 1 1 1 >> tc 2 wrr weights = 1 1 1 1 >> tc 3 wrr weights = 1 1 1 1 >> >> Regards, >> >> Zoltan > Regards, > Cristian