From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id EA8AEA0C3F; Mon, 28 Jun 2021 13:14:51 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D081640692; Mon, 28 Jun 2021 13:14:51 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id 281784068A for ; Mon, 28 Jun 2021 13:14:49 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10028"; a="293568325" X-IronPort-AV: E=Sophos;i="5.83,305,1616482800"; d="scan'208";a="293568325" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jun 2021 04:14:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.83,305,1616482800"; d="scan'208";a="407695627" Received: from fmsmsx601.amr.corp.intel.com ([10.18.126.81]) by orsmga006.jf.intel.com with ESMTP; 28 Jun 2021 04:14:35 -0700 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx601.amr.corp.intel.com (10.18.126.81) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4; Mon, 28 Jun 2021 04:14:34 -0700 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4; Mon, 28 Jun 2021 04:14:34 -0700 Received: from FMSEDG603.ED.cps.intel.com (10.1.192.133) by fmsmsx610.amr.corp.intel.com (10.18.126.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2242.4 via Frontend Transport; Mon, 28 Jun 2021 04:14:34 -0700 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (104.47.66.44) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2242.4; Mon, 28 Jun 2021 04:14:34 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nFK3S4I4SC0XYxkNbARbtmeU2RFp/vWLWcM7VuplG4XrmW5X6fwRFHdVmKz84eea/+UsLPHunVh5qDhhxGFV+gOltBY9epKga+0DpOFanH5wOUgY6bluCmeWfXD9NsgdWbi5Ata9KbqL3pJkUvMFJxOQVEr6cK8mwS/DwAnSFpZLax4FaieebY31NARaekNEv/d34YV/Jf23foyf3ydkZTgC2LNqHStML7n3vD6vpfQYvPkQlBido6S8Q2R6lM2sHbhhohoGyYvGsTja7az/BQ1O4WiriTJrKAx+cSWwGyqp+SEnOLfMJ6d++8CvAoCifcNTks2GzyjLjZVFOmbcDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LEaR+cYb8n1dBeM+SNHbLXJGBpjZCAV2uZ8f2praymo=; b=HRdOmOBPKkipPIdeO/Q0lOjdELuMZ7O+x1uC7GbQbfQmt8e2t6Lo8LcIw3FA9asj/ZczgUkPBsTfEjKKvNibJLXEneNzrwxIdEe1ECQnc5t/z3oZhhGpT4GXd4/CR4uj8GerKCO/LJ47HzpFp553MdfP861+PXSk8n/FBzMNVv+0VnH/CH+GsxVJ4QsgerSLMk1ewtA5aX2YjebfWYjysY3qHCr7Lv18QdDTQZ/f4pyLqfJ3RONC+vSzgMBT42TQSd69X0/GhuIiZihQx12MQsfcbYSl/WY5byEBoXEq46BZJzMrTNVuONvCQ/eR6RFr/xE6C8nutLZ3HEjDE41tjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LEaR+cYb8n1dBeM+SNHbLXJGBpjZCAV2uZ8f2praymo=; b=PnwwVztptYt4MaGUNS4Y2Zaad6WjRwRpIOFPsEeh9pScRo5GqKjrM9uw9KBfBtn+Wh5abWzhZdwPBDENBg0WzYshzlmOS7IfhtPCr7Z5jPR6s+60oRlQF+LmdayloNwsm8Vvj7mvgtO/uZTsKH4CcmuY3ZNxuSrpTZfh2RFDl6w= Received: from DM6PR11MB4491.namprd11.prod.outlook.com (2603:10b6:5:204::19) by DM6PR11MB4740.namprd11.prod.outlook.com (2603:10b6:5:2ad::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4264.20; Mon, 28 Jun 2021 11:14:32 +0000 Received: from DM6PR11MB4491.namprd11.prod.outlook.com ([fe80::7dc4:66b0:f76b:6d48]) by DM6PR11MB4491.namprd11.prod.outlook.com ([fe80::7dc4:66b0:f76b:6d48%7]) with mapi id 15.20.4264.026; Mon, 28 Jun 2021 11:14:32 +0000 From: "Ananyev, Konstantin" To: "Richardson, Bruce" , fengchengwen CC: Jerin Jacob , Jerin Jacob , =?iso-8859-1?Q?Morten_Br=F8rup?= , Nipun Gupta , Thomas Monjalon , "Yigit, Ferruh" , dpdk-dev , Hemant Agrawal , Maxime Coquelin , Honnappa Nagarahalli , David Marchand , Satananda Burla , "Prasun Kapoor" Thread-Topic: [dpdk-dev] dmadev discussion summary Thread-Index: AQHXaj/SxK7flUdWVEmaWrHXFDUqfKspNHcAgAAOzkA= Date: Mon, 28 Jun 2021 11:14:31 +0000 Message-ID: References: <25d29598-c26d-8497-2867-9b650c79df49@huawei.com> <3db2eda0-4490-2b8f-c65d-636bcf794494@huawei.com> In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.5.1.3 authentication-results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=intel.com; x-originating-ip: [109.255.184.192] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 296555a8-6b2d-42b8-ccfa-08d93a25e76a x-ms-traffictypediagnostic: DM6PR11MB4740: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5797; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: iBt/PVk5bsjHI4+byLNUyaDZIoYaxUQlT7AxaW1lm3fohp1i7Gpyl8sh6Slx7z55LqUZ/XDGtC5isPfM65GT7jR4UK2bpw/7dK9sF01Be8aQX6dsRal1H6uCxNKXkNCoq438Xel8Tqoz22DFfKbO3uwbyYDpNSJ0UgvKHhENRL4ZbU/hhabKLFoXZvlA8fvjG+T9N8WXcUsiKL/E3UZTn4gtKce7AhMV1ER1l4EwFrUyH9a2rcIKDy9tChN+N9zmakANz4/f3CH1RNS/HT62i/6sr0a3DVXzAt+5lZr+qV+k1fKH36fYxXDsSLdB4N0IdkLWmqCSxEnnoRrCsSTmszDaRU/dMeXW+2bi0PZeaGH63AD0RmdMs4jjdq8Kh3R1/n0p0YnYDmw6PnoIruyZzmLwG4TTTVn1tNhLEGyg0PCqTS2P23Gmw3D4rp65+Unu5xEsxWV7LCrCeRx/fuZ3cgvp2Nrdu7H+J5qW66JeXN4A0ctdaAz3JaTwSednU5M3WJtDmNiuAJncAYsOhFrqu0D9/hTBtrUO1b3E3VHhYCaOCaJzw1yHzAQJe1LHrN3i/tOfd4wc1vLTWVWH0lEAWbqmnuqp/Xz7sTsJPJef67U= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DM6PR11MB4491.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(136003)(396003)(346002)(39860400002)(376002)(83380400001)(8676002)(55016002)(8936002)(9686003)(33656002)(86362001)(66946007)(52536014)(186003)(122000001)(66476007)(71200400001)(54906003)(316002)(26005)(76116006)(7696005)(7416002)(64756008)(55236004)(4326008)(66446008)(6506007)(66556008)(38100700002)(5660300002)(478600001)(110136005)(2906002); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?drXVeQUmzCHDotioKkkBl1o1hLO+erN8erSCAXwoCzFBoyQP+rO2oOReWW?= =?iso-8859-1?Q?51vmSeeKPZ0MDpeKDQcN68/lukH0Lpv2Lo3hNqH+wecIztVnd04wWFQoVR?= =?iso-8859-1?Q?VVFoizWOASi0kwQoi16nz0n4oUInpaFjOpegA9SdglVt0Jogz70Zy8wBKa?= =?iso-8859-1?Q?t9vg+v3fZti/El7eVQjoRQ6JzVQlGZhxvHkyMmIk2sl4jyFrBzqAzCVfiv?= =?iso-8859-1?Q?69nQctU4AGN3DCWfSFah0BvBEmkhfaIC4XT42elxGWUBgaJx8O1R3JLmLm?= =?iso-8859-1?Q?aQI90pth7GI2nQ2OBNa0dkQDhP6k1ewPYteU3yU1EtncCk7/wHQdmKnJ20?= =?iso-8859-1?Q?/NvYCL/FNWB04mJun67q1Ec9b3n+Dqhf5S75nREOvEcC6dRjZD4dLt2s9i?= =?iso-8859-1?Q?xeyLdJsspXGxZ5QzWpiWoVytEFMk8yqnH3k0QwOU0GAsDhsQCh2dRnYGUH?= =?iso-8859-1?Q?A+XwiBA/GbvpxrCncOSEwwvfNcQF7YuTPG52rkBjnhvoL/PrIpM9ycT4Y1?= =?iso-8859-1?Q?vA/fMlgRc6f1f4hbDFyqwerrHHZW5OaF8QSmdxAgzcoLMdzu/ig2H9C0R5?= =?iso-8859-1?Q?Iy9SO+IIXilEJCt9u0c6MsLwoNEXjXyuBdQAh44GnuqLwkYCW9hp5uPCnK?= =?iso-8859-1?Q?UVO2sVHFYcQLNfa9AOpqXJSRJZYFj40a8awlKvSRV5LwsdENbGavhNliEq?= =?iso-8859-1?Q?vDM0oPZbCldvu1nkAE0jGd72pn0J3RQyi6fr5j1nFtbGfER1BMLicb5MVm?= =?iso-8859-1?Q?mKXOhvLG3hUEFfWQf8KwnWoaNEk88Vk5w4IXo2cpD4sNfO5V1yhBikAzu6?= =?iso-8859-1?Q?RP07vDn25x1PvCstFWBb+D+irJ7Y0Q5dJAdBT8WsIALY7L1mFoQk88OqRY?= =?iso-8859-1?Q?43hacUqMv/QLMKSOL3jbf4ECUg8Jw8NReVMUL/swp2qXASIgRDKKNAowIW?= =?iso-8859-1?Q?uS0WcfdNLkYxwSFIDRZP3wLeJtVX3A6JIn9qKubn3F6zvaIEL671LA6cZz?= =?iso-8859-1?Q?JVWa4mzkq7ZLW2ipwB1hG6KOq7v99vobCmEvNrYTqWwt9aXJTWyksAo62P?= =?iso-8859-1?Q?wlpypprwcmwh/sBVshD0FrghIrXwCKhUbaL9bby9S40XmvMIKd+m0YbWJp?= =?iso-8859-1?Q?iDNMtoS7nA2YPFgGPlRk3eWT7WciM0CuLWGA3KR51gG7FLTRUiGHhXic+C?= =?iso-8859-1?Q?b6mUWlHx31mD1jNPXNT7SwqQRP7PghrRyRl+C5PtMXRy/ldbvq49LZcCPK?= =?iso-8859-1?Q?Vw36gf6gJZCIre3ooGD93xrvEejNDkNKGXzZPvZ3FVm8uX1h2rDfMAo6hn?= =?iso-8859-1?Q?NjPDVqOuLMcLsC8EJXUMgbF5fBJyZisbQ6pNeS0i8/RUr54eUiPiY4Vd3S?= =?iso-8859-1?Q?qFyAuDPFlP?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DM6PR11MB4491.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 296555a8-6b2d-42b8-ccfa-08d93a25e76a X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jun 2021 11:14:31.9113 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 9hnkpRhQHvFspDBNF4TxlGSmADKqwEzhg3qr+3WarRB8EXayEyhVXTEdz5JmA9Yx0T5no2R82QthHLRAroIrPRTzSOXISXEc/gMllLM79jY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR11MB4740 X-OriginatorOrg: intel.com Subject: Re: [dpdk-dev] dmadev discussion summary X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi everyone, > On Sat, Jun 26, 2021 at 11:59:49AM +0800, fengchengwen wrote: > > Hi, all > > I analyzed the current DPAM DMA driver and drew this summary in conju= nction > > with the previous discussion, and this will as a basis for the V2 imple= mentation. > > Feedback is welcome, thanks > > > Fantastic review and summary, many thanks for the work. Some comments > inline in API part below, but nothing too major, I hope. >=20 > /Bruce >=20 > > > > > Summary: > > 1) The dpaa2/octeontx2/Kunpeng are all ARM soc, there may acts as end= point of > > x86 host (e.g. smart NIC), multiple memory transfer requirements m= ay exist, > > e.g. local-to-host/local-to-host..., from the point of view of API= design, > > I think we should adopt a similar 'channel' or 'virt-queue' concep= t. > > 2) Whether to create a separate dmadev for each HW-queue? We previous= ly > > discussed this, and due HW-queue could indepent management (like > > Kunpeng_dma and Intel DSA), we prefer create a separate dmadev for= each > > HW-queue before. But I'm not sure if that's the case with dpaa. I = think > > that can be left to the specific driver, no restriction is imposed= on the > > framework API layer. > > 3) I think we could setup following abstraction at dmadev device: > > ------------ ------------ > > |virt-queue| |virt-queue| > > ------------ ------------ > > \ / > > \ / > > \ / > > ------------ ------------ > > | HW-queue | | HW-queue | > > ------------ ------------ > > \ / > > \ / > > \ / > > dmadev > > 4) The driver's ops design (here we only list key points): > > [dev_info_get]: mainly return the number of HW-queues > > [dev_configure]: nothing important > > [queue_setup]: create one virt-queue, has following main parameter= s: > > HW-queue-index: the HW-queue index used > > nb_desc: the number of HW descriptors > > opaque: driver's specific info > > Note1: this API return virt-queue index which will used in lat= er API. > > If user want create multiple virt-queue one the same HW= -queue, > > they could achieved by call queue_setup with the same > > HW-queue-index. > > Note2: I think it's hard to define queue_setup config paramter= , and > > also this is control API, so I think it's OK to use opa= que > > pointer to implement it. > I'm not sure opaque pointer will work in practice, so I think we should t= ry > and standardize the parameters as much as possible. Since it's a control > plane API, using a struct with a superset of parameters may be workable. > Let's start with a minimum set and build up from there. >=20 > > [dma_copy/memset/sg]: all has vq_id input parameter. > > Note: I notice dpaa can't support single and sg in one virt-qu= eue, and > > I think it's maybe software implement policy other than = HW > > restriction because virt-queue could share the same HW-q= ueue. > Presumably for queues which support sq, the single-enqueue APIs can use a > single sg list internally? >=20 > > Here we use vq_id to tackle different scenario, like local-to-loc= al/ > > local-to-host and etc. > > 5) And the dmadev public data-plane API (just prototype): > > dma_cookie_t rte_dmadev_memset(dev, vq_id, pattern, dst, len, flag= s) > > -- flags: used as an extended parameter, it could be uint32_t >=20 > Suggest uint64_t rather than uint32_t to ensure we have expansion room? > Otherwise +1 >=20 > > dma_cookie_t rte_dmadev_memcpy(dev, vq_id, src, dst, len, flags) > +1 >=20 > > dma_cookie_t rte_dmadev_memcpy_sg(dev, vq_id, sg, sg_len, flags) > > -- sg: struct dma_scatterlist array > I don't think our drivers will be directly implementing this API, but so > long as SG support is listed as a capability flag I'm fine with this as a= n > API. [We can't fudge it as a bunch of single copies, because that would > cause us to have multiple cookies rather than one] >=20 > > uint16_t rte_dmadev_completed(dev, vq_id, dma_cookie_t *cookie, > > uint16_t nb_cpls, bool *has_error) > > -- nb_cpls: indicate max process operations number > > -- has_error: indicate if there is an error > > -- return value: the number of successful completed operations. > > -- example: > > 1) If there are already 32 completed ops, and 4th is error, a= nd > > nb_cpls is 32, then the ret will be 3(because 1/2/3th is O= K), and > > has_error will be true. > > 2) If there are already 32 completed ops, and all successful > > completed, then the ret will be min(32, nb_cpls), and has_= error > > will be false. > > 3) If there are already 32 completed ops, and all failed comp= leted, > > then the ret will be 0, and has_error will be true. > +1 for this >=20 > > uint16_t rte_dmadev_completed_status(dev_id, vq_id, dma_cookie_t *= cookie, > > uint16_t nb_status, uint32_t = *status) > > -- return value: the number of failed completed operations. > > And here I agree with Morten: we should design API which adapts to= DPDK > > service scenarios. So we don't support some sound-cards DMA, and 2= D memory > > copy which mainly used in video scenarios. >=20 > Can I suggest a few adjustments here to the semantics of this API. In > future we may have operations which return a status value, e.g. our > hardware can support ops like compare equal/not-equal, which means that > this API would be meaningful even in case of success. Therefore, I sugges= t > that the return value be changed to allow success also to be returned in > the array, and the return value is not the number of failed ops, but the > number of ops for which status is being returned. >=20 > Also for consideration: when trying to implement this in a prototype in o= ur > driver, it would be easier if we relax the restriction on the "completed" > API so that we can flag has_error when an error is detected rather than > guaranteeing to return all elements right up to the error. For example, i= f > we have a burst of packets and one is problematic, it may be easier to fl= ag > the error at the start of the burst and then have a few successful entrie= s > at the start of the completed_status array. [Separate from this] We shoul= d > also have a "has_error" or "more_errors" flag on this API too, to indicat= e > when the user can switch back to using the regular "completed" API. This > means that apps switch from one API to the other when "has_error" is true= , > and only switch back when it becomes false again. >=20 > > 6) The dma_cookie_t is signed int type, when <0 it mean error, it's > > monotonically increasing base on HW-queue (other than virt-queue).= The > > driver needs to make sure this because the damdev framework don't = manage > > the dma_cookie's creation. > +1 to this. > I think we also should specify that the cookie is guaranteed to wrap at a > power of 2 value (UINT16_MAX??). This allows it to be used as an > index into a circular buffer just by masking. >=20 > > 7) Because data-plane APIs are not thread-safe, and user could determ= ine > > virt-queue to HW-queue's map (at the queue-setup stage), so it is = user's > > duty to ensure thread-safe. > > 8) One example: > > vq_id =3D rte_dmadev_queue_setup(dev, config.{HW-queue-index=3Dx, = opaque}); > > if (vq_id < 0) { > > // create virt-queue failed > > return; > > } > > // submit memcpy task > > cookit =3D rte_dmadev_memcpy(dev, vq_id, src, dst, len, flags); > > if (cookie < 0) { > > // submit failed > > return; > > } > > // get complete task > > ret =3D rte_dmadev_completed(dev, vq_id, &cookie, 1, has_error); > > if (!has_error && ret =3D=3D 1) { > > // the memcpy successful complete > > } > +1 I have two questions on the proposed API: 1. Would it make sense to split submission API into two stages: a) reserve and prepare b) actual submit. Similar to what DPDK ioat/idxd PMDs have right now: /* reserve and prepare */ for (i=3D0;i=20 > > 9) As octeontx2_dma support sg-list which has many valid buffers in > > dpi_dma_buf_ptr_s, it could call the rte_dmadev_memcpy_sg API. > > 10) As ioat, it could delcare support one HW-queue at dev_configure s= tage, and > > only support create one virt-queue. > +1 >=20 > > 11) As dpaa2_qdma, I think it could migrate to new framework, but sti= ll wait > > for dpaa2_qdma guys feedback. > > 12) About the prototype src/dst parameters of rte_dmadev_memcpy API, = we have > > two candidates which are iova and void *, how about introduce dma= _addr_t > > type which could be va or iova ? > > >=20 > Many thanks again.