From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 77A0EA04F6; Tue, 7 Jan 2020 08:18:26 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 437441D956; Tue, 7 Jan 2020 08:18:26 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id 40E1D1D94B for ; Tue, 7 Jan 2020 08:18:24 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 0077DBP2012597; Mon, 6 Jan 2020 23:18:19 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=B77Iy1IuA/ery/OuwOYsCZSoFN5dBBb5faQc7i89o/o=; b=JxDJTK0o/r7EzZzIcNel5aRA7AXJLm2QymMAiSAaL94SsEPItpC1wA7V2RhBS2K3KsDb AKCI9FokHLMTPKMP2UnbBqsRbCYXj6uTEFFpUO1VXesHW5Sd04ltBEDVd0l5KLdtjRQX h1H8XEti6YlCnl+wGS2SzuyZsWhEDp3fS63kqmWa2zGizk8jYJNvNnO3kxamXyY+iiv8 B2r1U1i1ZDEFTNGPNqX8PmhHaaNtMBOKENUoPjoa8EZmePGyTBMaUn7o4NY5LFEM1+n9 Ipe9PRQQqg4i5t2+4U89yrgX4Ks1tU7dtMdIdItmnoV3oRKxZ5r7GypL3+Dvwc+XUtSx Jg== Received: from sc-exch03.marvell.com ([199.233.58.183]) by mx0a-0016f401.pphosted.com with ESMTP id 2xarxv9fer-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Mon, 06 Jan 2020 23:18:19 -0800 Received: from SC-EXCH01.marvell.com (10.93.176.81) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 6 Jan 2020 23:18:18 -0800 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (104.47.58.107) by SC-EXCH01.marvell.com (10.93.176.81) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Mon, 6 Jan 2020 23:18:18 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f1xQEVrQUP0DetVg/54QG4sGSUjPa5AlS8v6z9FfPKl9jNh0Ra8iJBba6iBByBdDvnmbto6X1Mtng/Gd1B+NCu1R67S2XhMa4tkvjfG080NCPcCdyqvsr7o7mOlArXWiUAhN/vMiCvAUniyCggmGlPGwM6bGm8D2NjnX97u5sbSjbXZHJVhrfFRFXSiB7M3OKkdsVmrrxqoLcr7XoB1V8Vh/vv3cgGB2aJEnAbFxA6xuSNg0f6lmedeJrKJcVinUQfOhovtaCzvGk7/G8agli9fLFbrnnrMJ0VXX07TEkyxBY0po9bDOG/onzzeySAHdlDYMOzpvljYqMCLvahONKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B77Iy1IuA/ery/OuwOYsCZSoFN5dBBb5faQc7i89o/o=; b=mZ0fBvFgDUwf3f8b6rAe6Ritg1fC8hXED4AMbZ830rmw2pGhChxdD1dnI0S9+WPT8WNqvd9IA8zH43CgR9m/ASiZofLCCC2yATCeXqdWLky7was3V2IBGz2NJfbQXzkXTY4gCAwCQau7AppO2SvT61Gas2hTrmiWUqs9voR6v+yiCt9IrgSuI2wzTwsLI15Im8tdPhxq8t131he5l1nrD0GOPHnC39HB0kc//2wNfkKO4K1GjnXDf4ri5As/eYxjWPRm+vdtW3uftsKBPgJ4brevKQA6BNT9Y4Mhea3gOvDKvK88a0aPBIY+uSaJ6TmFz8iDqqtNsGooZCeMf9VxbQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=B77Iy1IuA/ery/OuwOYsCZSoFN5dBBb5faQc7i89o/o=; b=SnPk7AX+NNTcDNe9KRZ7fnphCYA0/6596ZsrPdW/2X7H3OtRW1K1Mv6dwiQO5WHt5u7aLi/kd4/mu863AULrKhr8I+j3BuSwE8Lq11SI//Chvx1kKd41AE4jFsgnV3AeKOOPZDeGkUbytkgKSHPoSLPsem+HP3KklE1Z+Jze3uk= Received: from MWHPR18MB1376.namprd18.prod.outlook.com (10.173.243.18) by MWHPR18MB1053.namprd18.prod.outlook.com (10.173.125.15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2602.12; Tue, 7 Jan 2020 07:18:16 +0000 Received: from MWHPR18MB1376.namprd18.prod.outlook.com ([fe80::3d27:53fc:7a3a:f73a]) by MWHPR18MB1376.namprd18.prod.outlook.com ([fe80::3d27:53fc:7a3a:f73a%5]) with mapi id 15.20.2602.016; Tue, 7 Jan 2020 07:18:16 +0000 From: Mahipal Challa To: Gavin Hu , "dev@dpdk.org" CC: Jerin Jacob Kollanukkaran , "Narayana Prasad Raju Athreya" , Subrahmanyam Nilla , Venkateshwarlu Nalla , nd Thread-Topic: [dpdk-dev] [PATCH v3 5/6] raw/octeontx2_ep: add dequeue operation Thread-Index: AQHVxIoOSQahCHpD7UCLXfEZFL0ls6fesYqAgAAXc2A= Date: Tue, 7 Jan 2020 07:18:16 +0000 Message-ID: References: <1578312447-17277-1-git-send-email-mchalla@marvell.com> <1578312447-17277-6-git-send-email-mchalla@marvell.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [115.113.156.2] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: fe5198eb-c0d6-45aa-a3d2-08d79341c3e3 x-ms-traffictypediagnostic: MWHPR18MB1053: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6790; x-forefront-prvs: 027578BB13 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(366004)(39860400002)(346002)(396003)(136003)(376002)(189003)(199004)(13464003)(52536014)(5660300002)(478600001)(2906002)(30864003)(4326008)(9686003)(316002)(55016002)(54906003)(26005)(110136005)(186003)(8936002)(81166006)(81156014)(8676002)(33656002)(6506007)(64756008)(55236004)(86362001)(53546011)(66946007)(76116006)(66476007)(66556008)(7696005)(66446008)(71200400001); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR18MB1053; H:MWHPR18MB1376.namprd18.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: SV9LkVriHkgS9A+XGPLePQw8w8AeyfkPjsUVx2/BsDmPRuXkI7WA+DC4S3b1Q0ONZmBrFCDl//C01z3AkKJBQcDt/xnUrQoN+JWvoPE8SiP/ngc8XVPSCL7M27l7uji/7T2hmrnzC2WbUtIE63PZmXAfPUbdt0KDKOCDJXXo/GMBeqFh3K6bkCeAuarK/UnOqTmbkDaqu9MYwPlAcV3MvcvMxd80FQiA/W4g92HO8hOIB2Ye42juk5ZDEKpQIvdnBthIWEA01NBVsBr62ZXvksHAyrdwpmP456xK1lL5U+vUXvbeHShWEFhZ1d/uLahtmzJRGRrzsKSLCvY9RFsTPF13tdI9tLzP1dAcHeXWwkUR/f+jKKvE2JqHPJVGp7J/dp04kw/338/tfHCrUydjxs+N3nv6u01UVX9YVEGfEutoLR3FA8EjMswPEXsbWS4NWWNpLwYqWnV+y/UHvbp7FO1t2WRdLLpTTfxidn8iZNksDGAL5HoE1cvcY4gI3bMQ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: fe5198eb-c0d6-45aa-a3d2-08d79341c3e3 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Jan 2020 07:18:16.2718 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: VsbaNsIWd74Jhdnw+BMOFZRl2GT0Tr3Wk/LWQhBQ1XKAwYwamD3XmM4cvkefQsnVlFvkfucNZUL9Tt6CsvymDg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR18MB1053 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,18.0.572 definitions=2020-01-07_01:2020-01-06,2020-01-07 signatures=0 Subject: Re: [dpdk-dev] [PATCH v3 5/6] raw/octeontx2_ep: add dequeue operation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi Gavin,=20 Please see the response inline. > -----Original Message----- > From: Gavin Hu > Sent: Tuesday, January 7, 2020 11:13 AM > To: Mahipal Challa ; dev@dpdk.org > Cc: Jerin Jacob Kollanukkaran ; Narayana Prasad Raju > Athreya ; Subrahmanyam Nilla > ; Venkateshwarlu Nalla ; nd > > Subject: [EXT] RE: [dpdk-dev] [PATCH v3 5/6] raw/octeontx2_ep: add > dequeue operation >=20 > External Email >=20 > ---------------------------------------------------------------------- > Hi Mahipal, Jerin, >=20 > > -----Original Message----- > > From: Mahipal Challa > > Sent: Monday, January 6, 2020 8:07 PM > > To: dev@dpdk.org > > Cc: jerinj@marvell.com; pathreya@marvell.com; snilla@marvell.com; > > venkatn@marvell.com; Gavin Hu > > Subject: [dpdk-dev] [PATCH v3 5/6] raw/octeontx2_ep: add dequeue > > operation > > > > Add rawdev dequeue operation for SDP VF devices. > > > > Signed-off-by: Mahipal Challa > > --- > > drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c | 197 > > ++++++++++++++++++++++++++++++ > > drivers/raw/octeontx2_ep/otx2_ep_enqdeq.h | 2 + > > drivers/raw/octeontx2_ep/otx2_ep_rawdev.c | 1 + > > drivers/raw/octeontx2_ep/otx2_ep_rawdev.h | 18 ++- > > 4 files changed, 217 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c > > b/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c > > index 6910f08..dd270b5 100644 > > --- a/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c > > +++ b/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.c > > @@ -260,6 +260,7 @@ > > rte_mempool_get(sdpvf->enqdeq_mpool, &buf); > > if (buf =3D=3D NULL) { > > otx2_err("OQ buffer alloc failed"); > > + droq->stats.rx_alloc_failure++; > > /* sdp_droq_destroy_ring_buffers(droq);*/ > > return -ENOMEM; > > } > > @@ -645,3 +646,199 @@ > > return SDP_IQ_SEND_FAILED; > > } > > > > +static uint32_t > > +sdp_droq_refill(struct sdp_device *sdpvf, struct sdp_droq *droq) { > > + struct sdp_droq_desc *desc_ring; > > + uint32_t desc_refilled =3D 0; > > + void *buf =3D NULL; > > + > > + desc_ring =3D droq->desc_ring; > > + > > + while (droq->refill_count && (desc_refilled < droq->nb_desc)) { > > + /* If a valid buffer exists (happens if there is no dispatch), > > + * reuse the buffer, else allocate. > > + */ > > + if (droq->recv_buf_list[droq->refill_idx].buffer !=3D NULL) > > + break; > > + > > + rte_mempool_get(sdpvf->enqdeq_mpool, &buf); > > + /* If a buffer could not be allocated, no point in > > + * continuing > > + */ > > + if (buf =3D=3D NULL) { > > + droq->stats.rx_alloc_failure++; > > + break; > > + } > > + > > + droq->recv_buf_list[droq->refill_idx].buffer =3D buf; > > + desc_ring[droq->refill_idx].buffer_ptr =3D > > rte_mem_virt2iova(buf); > > + > > + /* Reset any previous values in the length field. */ > > + droq->info_list[droq->refill_idx].length =3D 0; > > + > > + droq->refill_idx =3D sdp_incr_index(droq->refill_idx, 1, > > + droq->nb_desc); > > + > > + desc_refilled++; > > + droq->refill_count--; > > + > > + } > > + > > + return desc_refilled; > > +} > > + > > +static int > > +sdp_droq_read_packet(struct sdp_device *sdpvf __rte_unused, > > + struct sdp_droq *droq, > > + struct sdp_droq_pkt *droq_pkt) { > > + struct sdp_droq_info *info; > > + uint32_t total_len =3D 0; > > + uint32_t pkt_len =3D 0; > > + > > + info =3D &droq->info_list[droq->read_idx]; > > + sdp_swap_8B_data((uint64_t *)&info->length, 1); > > + if (!info->length) { > > + otx2_err("OQ info_list->length[%ld]", (long)info->length); > > + goto oq_read_fail; > > + } > > + > > + /* Deduce the actual data size */ > > + info->length -=3D SDP_RH_SIZE; > > + total_len +=3D (uint32_t)info->length; > > + > > + otx2_sdp_dbg("OQ: pkt_len[%ld], buffer_size %d", > > + (long)info->length, droq->buffer_size); > > + if (info->length > droq->buffer_size) { > > + otx2_err("This mode is not supported: pkt_len > > > buffer_size"); > > + goto oq_read_fail; > > + } > > + > > + if (info->length <=3D droq->buffer_size) { > > + pkt_len =3D (uint32_t)info->length; > > + droq_pkt->data =3D droq->recv_buf_list[droq- > > >read_idx].buffer; > > + droq_pkt->len =3D pkt_len; > > + > > + droq->recv_buf_list[droq->read_idx].buffer =3D NULL; > > + droq->read_idx =3D sdp_incr_index(droq->read_idx, 1,/* > > count */ > > + droq->nb_desc /* max rd > > idx */); > > + droq->refill_count++; > > + > > + } > > + > > + info->length =3D 0; > > + > > + return SDP_OQ_RECV_SUCCESS; > > + > > +oq_read_fail: > > + return SDP_OQ_RECV_FAILED; > > +} > > + > > +static inline uint32_t > > +sdp_check_droq_pkts(struct sdp_droq *droq, uint32_t burst_size) { > > + uint32_t min_pkts =3D 0; > > + uint32_t new_pkts; > > + uint32_t pkt_count; > > + > > + /* Latest available OQ packets */ > > + pkt_count =3D rte_read32(droq->pkts_sent_reg); > > + > > + /* Newly arrived packets */ > > + new_pkts =3D pkt_count - droq->last_pkt_count; > > + otx2_sdp_dbg("Recvd [%d] new OQ pkts", new_pkts); > > + > > + min_pkts =3D (new_pkts > burst_size) ? burst_size : new_pkts; > > + if (min_pkts) { > > + rte_atomic64_add(&droq->pkts_pending, min_pkts); > > + /* Back up the aggregated packet count so far */ > > + droq->last_pkt_count +=3D min_pkts; > > + } > > + > > + return min_pkts; > > +} > > + > > +/* Check for response arrival from OCTEON TX2 > > + * returns number of requests completed */ int > > +sdp_rawdev_dequeue(struct rte_rawdev *rawdev, > > + struct rte_rawdev_buf **buffers, unsigned int count, > > + rte_rawdev_obj_t context __rte_unused) { > > + struct sdp_droq_pkt *oq_pkt; > > + struct sdp_device *sdpvf; > > + struct sdp_droq *droq; > > + > > + uint32_t q_no =3D 0, pkts; > > + uint32_t new_pkts; > > + uint32_t ret; > > + > > + sdpvf =3D (struct sdp_device *)rawdev->dev_private; > > + > > + droq =3D sdpvf->droq[q_no]; > > + if (!droq) { > > + otx2_err("Invalid droq[%d]", q_no); > > + goto deq_fail; > > + } > > + > > + /* Grab the lock */ > > + rte_spinlock_lock(&droq->lock); > > + > > + new_pkts =3D sdp_check_droq_pkts(droq, count); > > + if (!new_pkts) { > > + otx2_sdp_dbg("Zero new_pkts:%d", new_pkts); > > + goto deq_fail; /* No pkts at this moment */ > > + } > > + > > + otx2_sdp_dbg("Received new_pkts =3D %d", new_pkts); > > + > > + for (pkts =3D 0; pkts < new_pkts; pkts++) { > > + > > + /* Push the received pkt to application */ > > + oq_pkt =3D (struct sdp_droq_pkt *)buffers[pkts]; > > + > > + ret =3D sdp_droq_read_packet(sdpvf, droq, oq_pkt); > > + if (ret) { > > + otx2_err("DROQ read pakt failed."); > > + goto deq_fail; > > + } > > + > > + /* Stats */ > > + droq->stats.pkts_received++; > > + droq->stats.bytes_received +=3D oq_pkt->len; > > + } > > + > > + /* Ack the h/w with no# of pkts read by Host */ > > + rte_write32(pkts, droq->pkts_sent_reg); > Rte_write32 has a built-in rte_io_wmb, aka "dsb" for aarch64, inside, thi= s > hurts performance, this is on the fast data path? > > + rte_cio_wmb(); > This barrier should be hoisted up to between the dequeue operation and > register write to keep the ordering of two. > I am surprised why rte_cio_mb is not defined within DPDK, it is required = here > to keep the read/write ordering, neither rmb nor wmb can fulfill the orde= ring > required. [Mahipal]: We would like to go with this now, and separately handle perform= ance improvement optimizations in the future. > > + > > + droq->last_pkt_count -=3D pkts; > > + > > + otx2_sdp_dbg("DROQ pkts[%d] pushed to application", pkts); > > + > > + /* Refill DROQ buffers */ > > + if (droq->refill_count >=3D 2 /* droq->refill_threshold */) { > > + int desc_refilled =3D sdp_droq_refill(sdpvf, droq); > > + > > + /* Flush the droq descriptor data to memory to be sure > > + * that when we update the credits the data in memory is > > + * accurate. > > + */ > > + rte_write32(desc_refilled, droq->pkts_credit_reg); > Rte_cio_wmb should be used instead of the built-in rte_io_wmb to improve > the performance. > But if performance is not a concern here, rte_io_wmb rescued the situatio= n. [Mahipal]: As mentioned above we would like to go with this now, and separa= tely handle performance improvement optimizations in the future. > > + > > + /* Ensure mmio write completes */ > > + rte_wmb(); > > + otx2_sdp_dbg("Refilled count =3D %d", desc_refilled); > > + } > > + > > + /* Release the spin lock */ > > + rte_spinlock_unlock(&droq->lock); > > + > > + return pkts; > > + > > +deq_fail: > > + rte_spinlock_unlock(&droq->lock); > > + return SDP_OQ_RECV_FAILED; > > +} > > diff --git a/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.h > > b/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.h > > index b9b7c0b..172fdc5 100644 > > --- a/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.h > > +++ b/drivers/raw/octeontx2_ep/otx2_ep_enqdeq.h > > @@ -11,6 +11,8 @@ > > #define SDP_IQ_SEND_FAILED (-1) > > #define SDP_IQ_SEND_SUCCESS (0) > > > > +#define SDP_OQ_RECV_FAILED (-1) > > +#define SDP_OQ_RECV_SUCCESS (0) > > > > static inline uint64_t > > sdp_endian_swap_8B(uint64_t _d) > > diff --git a/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c > > b/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c > > index 22a6beb..7158b97 100644 > > --- a/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c > > +++ b/drivers/raw/octeontx2_ep/otx2_ep_rawdev.c > > @@ -252,6 +252,7 @@ > > .dev_stop =3D sdp_rawdev_stop, > > .dev_close =3D sdp_rawdev_close, > > .enqueue_bufs =3D sdp_rawdev_enqueue, > > + .dequeue_bufs =3D sdp_rawdev_dequeue, > > }; > > > > static int > > diff --git a/drivers/raw/octeontx2_ep/otx2_ep_rawdev.h > > b/drivers/raw/octeontx2_ep/otx2_ep_rawdev.h > > index 8fd06fb..a77cbab 100644 > > --- a/drivers/raw/octeontx2_ep/otx2_ep_rawdev.h > > +++ b/drivers/raw/octeontx2_ep/otx2_ep_rawdev.h > > @@ -279,6 +279,18 @@ struct sdp_recv_buffer { }; > > #define SDP_DROQ_RECVBUF_SIZE (sizeof(struct sdp_recv_buffer)) > > > > +/* DROQ statistics. Each output queue has four stats fields. */ > > +struct sdp_droq_stats { > > + /* Number of packets received in this queue. */ > > + uint64_t pkts_received; > > + > > + /* Bytes received by this queue. */ > > + uint64_t bytes_received; > > + > > + /* Num of failures of rte_pktmbuf_alloc() */ > > + uint64_t rx_alloc_failure; > > +}; > > + > > /* Structure to define the configuration attributes for each Output qu= eue. > > */ > > struct sdp_oq_config { > > /* Max number of OQs available */ > > @@ -345,6 +357,9 @@ struct sdp_droq { > > */ > > void *pkts_sent_reg; > > > > + /* Statistics for this DROQ. */ > > + struct sdp_droq_stats stats; > > + > > /* DMA mapped address of the DROQ descriptor ring. */ > > size_t desc_ring_dma; > > > > @@ -476,6 +491,7 @@ struct sdp_device { > > > > int sdp_rawdev_enqueue(struct rte_rawdev *dev, struct rte_rawdev_buf > > **buffers, > > unsigned int count, rte_rawdev_obj_t context); > > - > > +int sdp_rawdev_dequeue(struct rte_rawdev *dev, struct rte_rawdev_buf > > **buffers, > > + unsigned int count, rte_rawdev_obj_t context); > > > > #endif /* _OTX2_EP_RAWDEV_H_ */ > > -- > > 1.8.3.1 Thanks, Mahipal