From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 5598D58C4 for ; Mon, 10 Sep 2018 23:46:44 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Sep 2018 14:46:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.53,357,1531810800"; d="scan'208";a="72132066" Received: from irsmsx152.ger.corp.intel.com ([163.33.192.66]) by orsmga008.jf.intel.com with ESMTP; 10 Sep 2018 14:46:41 -0700 Received: from irsmsx101.ger.corp.intel.com ([169.254.1.40]) by IRSMSX152.ger.corp.intel.com ([169.254.6.200]) with mapi id 14.03.0319.002; Mon, 10 Sep 2018 22:46:40 +0100 From: "Trahe, Fiona" To: "Wiles, Keith" , Sofia Baran CC: "users@dpdk.org" , "Trahe, Fiona" Thread-Topic: [dpdk-users] dpdk and bulk data (video/audio) Thread-Index: AQHUSPmHHa3vWdqPH0mf1VbpIcmCkaTpZ14AgACiP3A= Date: Mon, 10 Sep 2018 21:46:39 +0000 Message-ID: <348A99DA5F5B7549AA880327E580B43589618BE2@IRSMSX101.ger.corp.intel.com> References: <73D04E54-F6CC-41C3-A622-744B88592AC5@intel.com> In-Reply-To: <73D04E54-F6CC-41C3-A622-744B88592AC5@intel.com> Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiYTVjYzU4NGYtMmNkOS00OWNkLWIwY2EtNjRhZWM3NGE5M2EyIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoidXhYdlpxQThvY2wrOUQ5a3NoNU40WlluaVBGa0VVUklla244b29zQzdMZ1wvWTVyOGREMGliSFFBbm5PZWhlQ0MifQ== x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-users] dpdk and bulk data (video/audio) X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 10 Sep 2018 21:46:45 -0000 > -----Original Message----- > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Wiles, Keith > Sent: Monday, September 10, 2018 1:52 PM > To: Sofia Baran > Cc: users@dpdk.org > Subject: Re: [dpdk-users] dpdk and bulk data (video/audio) >=20 >=20 >=20 > > On Sep 10, 2018, at 6:28 AM, Sofia Baran wrote: > > > > > > Hi All, > > > > I want/try to us DPDK for transferring larger amount of data, e.g. vide= o frames which usually are > stored within memory buffers with sizes of several MB (remark: by using h= uges pages, these buffers > could be physically contiguous). > > > > When looking at the DPDK documentation, library APIs and examples, I ca= n't find a way/hint how to > transfer larger buffers using DPDK without copying the video buffer fragm= ents to the payload sections > of the mbufs - which results in high CPU loads. > > > > Within the ip_fragmentation example indirect mbufs are used, pointing t= o the payload section of a > direct mbuf (holding the header). But in my understanding the maximum siz= e of a mbuf payload is 65KB > (uint16_t)!? >=20 > It is true that mbufs only hold (64K - 1). The concept of mbufs is normal= ly an ethernet packet and they > are limited to 64K. >=20 > You can create a small mbuf (128 bytes) then set offset/data in the mbuf = to point to the video buffer > only if you can find the physical memory address for the data. The mbuf n= ormally holds the physical > address of the mbuf->data not the attached buffer in this case. This of c= ourse means you have to > manage the mbuf internal structure members yourself and be very careful y= ou do not rearrange the > mbuf members as that can cause a performance problem. >=20 But the 64k-1 limit still applies, unless I'm misunderstanding. A way to get around this is to use chained mbufs. So create lots of small mbufs, each 128bytes, holding no payload, just the = mbuf struct. Chain them together with each buf_iova/buf_addr pointing to the next 64k-1 = segment of the payload. You'll need ~17 mbufs per MB, is this an acceptable overhead? =20 You can also consider using compressdev API to compress the data before tra= nsmitting. However will have the same problem - and can use the same solution - for pa= ssing the data to the PMD to be compressed/decompressed. =20 > > > > I'm pretty new to DPDK so maybe I missed something. I hope that someone= can provide me some hits > how to avoid copying the entire payload. > > > > Thanks > > Sofia Baran > > > > >=20 > Regards, > Keith