From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id C219E6A67 for ; Tue, 17 May 2016 19:13:42 +0200 (CEST) Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga102.fm.intel.com with ESMTP; 17 May 2016 10:13:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.26,324,1459839600"; d="scan'208";a="808784069" Received: from irsmsx108.ger.corp.intel.com ([163.33.3.3]) by orsmga003.jf.intel.com with ESMTP; 17 May 2016 10:13:40 -0700 Received: from irsmsx111.ger.corp.intel.com (10.108.20.4) by IRSMSX108.ger.corp.intel.com (163.33.3.3) with Microsoft SMTP Server (TLS) id 14.3.248.2; Tue, 17 May 2016 18:13:28 +0100 Received: from irsmsx108.ger.corp.intel.com ([169.254.11.241]) by irsmsx111.ger.corp.intel.com ([169.254.2.233]) with mapi id 14.03.0248.002; Tue, 17 May 2016 18:13:28 +0100 From: "Dumitrescu, Cristian" To: "gayathri.manepalli@wipro.com" , "dev@dpdk.org" CC: "Stokes, Ian" Thread-Topic: [ovs-dev] Traffic scheduling by qos_sched library in DPDK Thread-Index: AdGs4hmnewDtaf61Q/uyh7SFOd6pHQDe0VDA Date: Tue, 17 May 2016 17:13:28 +0000 Message-ID: <3EB4FA525960D640B5BDFFD6A3D89126479BEDA8@IRSMSX108.ger.corp.intel.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMmZmNTg3ZjMtZGIzNS00OTQwLTgzZjQtMjBkMGUzMWU2YWIxIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE1LjkuNi42IiwiVHJ1c3RlZExhYmVsSGFzaCI6IkJocEliTXI1Z1wvNytiVFJcL0d5QXRZR1ExU2REbWVuN05VaHVqXC9abUswNXc9In0= x-ctpclassification: CTP_IC x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [ovs-dev] Traffic scheduling by qos_sched library in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 May 2016 17:13:43 -0000 Hi Gayathri, > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of > gayathri.manepalli@wipro.com > Sent: Friday, May 13, 2016 7:44 AM > To: dev@dpdk.org > Cc: Stokes, Ian > Subject: [dpdk-dev] [ovs-dev] Traffic scheduling by qos_sched library in > DPDK >=20 > Hi Team, >=20 > I started working on implementing the QoS Shaping in OVS+DPDK by making > use of rte_sched library provided in DPDK.=20 Great news, thank you! Meanwhile to compare the > performance, started performance test with DPDK sample scheduling > application. Below are the configuration details of system which I am usi= ng, >=20 > Server Type : Intel ATOM >=20 > Huge page configuration: (Each page of 2M size) >=20 > [root@ATOM qos_sched]# grep -i huge /proc/meminfo > AnonHugePages: 90112 kB > HugePages_Total: 7168 > HugePages_Free: 7168 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB >=20 > Port Capacity : 1G (All four ports) > I am able to successfully run the qos_sched with three pfc's as below, >=20 > ./build/qos_sched -c 0x3e -n 1 --socket-mem 14336 -- --pfc "0,1,2,3" --pf= c > "1,0,3,2" --pfc "2,3,4,5" --cfg ./profile.cfg >=20 > Issue : >=20 > When I am trying to add one more profile configuration flow(4th one) , I = am > getting below error >=20 > Command : ./build/qos_sched -c 0x3e -n 1 --socket-mem 14336 -- --pfc > "0,1,2,3" --pfc "1,0,3,2" --pfc "2,3,4,5" --pfc "3,2,5,4" --cfg ./profil= e.cfg >=20 > Error: >=20 > done: Link Up - speed 1000 Mbps - full-duplex > SCHED: Low level config for pipe profile 0: > Token bucket: period =3D 1, credits per period =3D 1, size =3D 10= 0000 > Traffic classes: period =3D 5000000, credits per period =3D [5000= 000, 5000000, > 5000000, 5000000] > Traffic class 3 oversubscription: weight =3D 0 > WRR cost: [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1] > SCHED: Low level config for subport 0: > Token bucket: period =3D 1, credits per period =3D 1, size =3D 10= 0000 > Traffic classes: period =3D 1250000, credits per period =3D [1250= 000, 1250000, > 1250000, 1250000] > Traffic class 3 oversubscription: wm min =3D 0, wm max =3D 0 > EAL: Error - exiting with code: 1 > Cause: Cannot init mbuf pool for socket 3 >=20 > Analysis: >=20 > I have analyzed DPDK source code to find the root cause. I could see that= in > qos_sched, memory allocation while creating each mbug pool > (rte_mempool_create) for corresponding RX port is as below, >=20 >=20 > MBUF_SIZE =3D (1528 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM) >=20 > mp_size =3D 2*1024*1024 >=20 >=20 >=20 > From above I understood that, for each pfc/ rx port around 4G of huge pag= es > are consumed. Whereas ATOM is capable of maximum 7168 huge pages of > 2M which is 14336M in total. So I am not able to configure beyond three p= fc's. > But I would like to measure the performance with 4 port & 6 port scenario > which requires 4-6 pfc's configured. >=20 >=20 >=20 > Is there any other alternative through which I can configure more number = of > pfc's with my system configuration provided above. >=20 >=20 Yes, you are probably running out of memory. QoS hierarchical scheduler can be seen like a big reservoir of packets. Eac= h rte_sched_port object basically has thousands of packet queues internally= (number of queues is configurable). Ideally, to protect against the worst = case scenario, , the number of buffers provisioned per each output port tha= t has the hierarchical scheduler enabled needs to be at least equal to: num= ber of scheduler queues x queue size. For example, for 64K queues (4K pipes= with 16 queues each) of 64 packets each, this means 4M buffers per output = port, which (assuming 2KB buffers in the mempool) leads to 8 GB of memory p= er output port. In our examples/qos_sched sample application we decided to = go mid-way instead of worst case scenario, therefore we use 2M buffers per = output port. So possible workarounds for you to maximize the amount of ports with hierar= chical scheduler support given a fixed amount memory: 1. Lower the amount of buffers you use per output port, e.g. from 2M to 1M = and restest; 2. Use less pipes per output port (configurable), so number of scheduling q= ueues is less, which reduces the pressure on mempool size. Note that you can use distinct mempools per output port (like in examples/q= os_sched application) or a single big global mempool shared by all ports; t= his is transparent for the librte_sched library. Regards, Cristian >=20 > Thanks & Regards, >=20 > Gayathri >=20 > The information contained in this electronic message and any attachments = to > this message are intended for the exclusive use of the addressee(s) and m= ay > contain proprietary, confidential or privileged information. If you are n= ot the > intended recipient, you should not disseminate, distribute or copy this e= - > mail. Please notify the sender immediately and destroy all copies of this > message and any attachments. WARNING: Computer viruses can be > transmitted via email. The recipient should check this email and any > attachments for the presence of viruses. The company accepts no liability= for > any damage caused by any virus transmitted by this email. www.wipro.com