From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0077.outbound.protection.outlook.com [104.47.40.77]) by dpdk.org (Postfix) with ESMTP id E28731B1D5 for ; Wed, 10 Jan 2018 15:52:56 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=p/JSdgvtkCguop3XLfq38MiTcy6KyuXizN0JgKB3bfk=; b=S6KwQZ76ACdZ5RFUvIPXsztLlSFfYaTEEOHCJnpwXmUENg9NwIMi1Jd2N03Du530YjYR4c5mREhq8+/nJNeVGMI7JQzJPSVOtdYTHzrVDBp/T3zMISHj8jkd9omrQfxmrM7PKK6tAA6LCo5LblYgGfNiowb0PwFxq6JKVuRSOJ4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from Pavan-LT.caveonetworks.com (111.93.218.67) by DM5PR07MB3466.namprd07.prod.outlook.com (10.164.153.21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.386.5; Wed, 10 Jan 2018 14:52:51 +0000 From: Pavan Nikhilesh To: jerin.jacob@caviumnetworks.com, santosh.shukla@caviumnetworks.com, harry.van.haaren@intel.com, gage.eads@intel.com, hemant.agrawal@nxp.com, nipun.gupta@nxp.com, liang.j.ma@intel.com Cc: dev@dpdk.org, Pavan Nikhilesh Date: Wed, 10 Jan 2018 20:21:43 +0530 Message-Id: <20180110145144.28403-11-pbhagavatula@caviumnetworks.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180110145144.28403-1-pbhagavatula@caviumnetworks.com> References: <20171130072406.15605-1-pbhagavatula@caviumnetworks.com> <20180110145144.28403-1-pbhagavatula@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: BN6PR16CA0009.namprd16.prod.outlook.com (10.172.212.147) To DM5PR07MB3466.namprd07.prod.outlook.com (10.164.153.21) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 30371ae4-b95a-4b5f-0e59-08d55839d4b0 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:DM5PR07MB3466; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3466; 3:vWNn2FxXbZ3gqs4SYHgx4yXh5XPv6GOJfN7+lkj8t/pLV97X3T2HNr7pHhDAb8vEU2B3RTLN1KRdSPx2qQDoU5Pi1W1O2swVFeiALxPGMVgrMFrXYZjVLV+taZgo51UZIShTTqcXDDLRcu3RpK2bB3oUKVc0ZIEbcUG7EALXF1JzJJTJIJQnG6EBIgH8U+bgwyz2hEv/RYBYEbWHVwCcSNL+VlPSpQHmvwo0Pu1kMx9z88ItHpzm0Ccs7q5hjnYZ; 25:jgyekvTkyTNod8HaJ8KTi0p5/2J6PVx0vEVE41ZhSASvV9u+SvTcklM9T6HCb+8hDZgGAx+C/Om1gXBE7ono4qnYbRorSgWnoar8gQclknRrqoxvFCAT+aTJMJkuL/K6R9TmJD4YEE4C2VMq1zhAtBqbOHJFPWEKrv1IUyTLINCqTMfZ2yZ0ykB1rfYrFEpZVpYQgNygkkfXq7xA6YVDctJrMMpTkvA1Yfx5Dlndfv0eRFRtJdIolDy+O1eCdjUsEULH90S3AJ1hMXySxdZSTHXZldeUCq/MScvXvUYihSQQQUO4GXKPRRBJroUDS6R9Pzd5sXnXwnm9RXzlQv8vkA==; 31:zbHOseY/5/ilpXyYw8mryC9JYoQHmaUf3KvD9wiF6vAc+Sqtysf4ZWl9WgmOaD5Vjf/iR1n8aL8d8bhmrhfQhC0h3JQbcHtw9OQtdPvpHlGffcglLtrZLCvESX0ILsAe3UF2b3kvyi+FQTkiFbk8tOQWN0zgi9a7b2TkCZP0cgcVEm2Ttox4D8o2D/jG+JDP3U6t5u1fOLaYw6trr4uMjgTWROs/FZZ89E32eJilKAA= X-MS-TrafficTypeDiagnostic: DM5PR07MB3466: X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3466; 20:GvPbk9dYjbfkPiYzpcK/HzBqFkMwY0htJp+hDAfQjPGYwX51yn8x1Eg7c/ar8B2OSFX2gx8xSS4pfRNs0xjkpcT29djS3LK3pRF3VtMDGQxSfGCyEa2MKe7QuXUeeCFRBEJ7a3YKfUZXuIpZ5VaGTrcOzIPrPoPl06cJJ3EkGSMNw6FLOLKRSaPbsZ9UC7sNmV4huf2PGiUEPVXMHNhaX2nTfl6sYtZk/GZNkMEGkMK5FBKXJFi87PjjJmxOMdhkO8gCJ7Hu+todiW0Ylkm8tY1apc4zzlqfThubnyrADjpaC2lvVnHsZswWSl/S5DJDdhqYCaLmE8X1dlm4Hr8jxqS8qFJw4l/epfx0j6wZyc9t35h1oDfk61MtdLN+gvPZi5AzB1hQlOrRgURsleG9xebWxOGMMasjsM5CkVmYSVyhKRhPANsqX/MW3GJpKImupy6iTp0TTlRE2Fb+hnvZPQW/PhlFkushmPcSC9ItM1ySP183k/XvOXFX1ZYIxmKXNdTolW5pPy8mvtGcjaBUKaeos+T945Mc0un5Hrs+D86HBkEeq6FVx3VV9H3lKzCAehJMX6+epNtaFrkZSU/uy9YBsEb+o6No2nN1ntwm+xA=; 4:IyZ/2snNewu3XmLwlmvongYKo13TgIOYHGQXjNpc0ARZNdnfkIUwW+hV8sxolR9ma56LY5DNhhqZTsvmDXsJNLMzxC/7sh+Hv0QBMjGU/FWqFOUZzpLjEhUJP1+uBojM5WKbdoGxYH3GwRg95a5Ydw+WAIFXRJu02slUd7das9jDBk++i1aTaj+E62ythWDOL/cUidB7QSSUd+t4rVSZAYS7xjNdAJQ7e57ieBZmiuPpRcLlqPJPGNshUiF05vnbNSdq6WZi4OSJzbekt+3tuw== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(5005006)(8121501046)(10201501046)(3002001)(93006095)(3231023)(944501075)(6041268)(20161123560045)(20161123564045)(20161123558120)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123562045)(6072148)(201708071742011); SRVR:DM5PR07MB3466; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:DM5PR07MB3466; X-Forefront-PRVS: 0548586081 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(39860400002)(346002)(366004)(376002)(396003)(39380400002)(199004)(189003)(76176011)(25786009)(107886003)(4326008)(6666003)(2950100002)(106356001)(42882006)(8656006)(8676002)(59450400001)(105586002)(53936002)(6486002)(52116002)(53416004)(6512007)(50466002)(47776003)(5009440100003)(305945005)(68736007)(51416003)(386003)(6506007)(69596002)(36756003)(72206003)(2906002)(16586007)(50226002)(316002)(478600001)(7736002)(48376002)(3846002)(1076002)(6116002)(5660300001)(66066001)(16526018)(81166006)(97736004)(81156014)(8936002)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3466; H:Pavan-LT.caveonetworks.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3466; 23:OrKJP5cuVVoGB4JKtxt97XQkvtkiScTpcyboB8XSD?= =?us-ascii?Q?wMvLJUn0vEVZZ7a7xEYd3j13HjyatM8R9kkwuHqM1IGt4HdSj6zNIXi0Unaa?= =?us-ascii?Q?QM9OGhrlED5qPngpYLa7joNwu5hT9aRT0WLZGo7dYxbc+c5na8gi213NG0AP?= =?us-ascii?Q?jSi0ns3wpj4e4FAx5V5yqXwfQ0e0vDa2HqQ0JH1/hUnZkyAqo4mWxynmqkpG?= =?us-ascii?Q?CFmT+KqTm4P2XQu8x+EPkO8Ft3TXXQLv5ELFhcEnFVM4flA/WV6EdVg8/9NY?= =?us-ascii?Q?Mv0Mh6l+FEc7BIEg07sguyb8nVqtbWazDReexjzgSbfqfxO+Gn4KtgN2H1KY?= =?us-ascii?Q?3bycQPhgzTFdDF6hCVFyje086Dvr/wylgaurKVqH5ojTm9FE3A9VXAsu2kR2?= =?us-ascii?Q?NOvn6hmZRW0yyeD92EzvdoI8UTAtkBnSQSQ61b8O9Cv5wXQGJdMLigHFeV9H?= =?us-ascii?Q?ZX6kXSmIJY4WBtpkekuU8DUkvQqgvJ4uaqbCZfRY/7TpJ0hPE56o4gq/5OIr?= =?us-ascii?Q?YFiDCDVzLjNIJxVWOcDDK7IaIU1ond5aiAGdBU/wkKE+q1gkS/9KRjq3ZrJP?= =?us-ascii?Q?W6WvsEK3BdF/OTdifOjKAaarOEpMXjnXiZEQh7zP9r2w/4Fbu0s8lgvontKl?= =?us-ascii?Q?8P3CmJMrRSJmpKTV9vGmFOTzliuyuDB0hY2Ole5fSBeRsFHNTDhILdgOWeJL?= =?us-ascii?Q?X+fdbiXkLMM1S8NirheIBrylbi2ke2hHaFVLJcNO8THxfLdSiwEvdtpKPnbl?= =?us-ascii?Q?TFRz6U1RqjZ4VHRIqgDZ7575vv1swg3fgIR5ksWGtUIvXIuofmI4vhrMArKE?= =?us-ascii?Q?xqRp4BcU4ESAImMDp/sUklc0Qm8BUFcbebmH2emeBjUIGvi/NBV/QhhKKt32?= =?us-ascii?Q?QHLBx+Rc0ClurGkJEnrIX4EBOIRw6dWvjbeWTSohsYTmgYyl88laGAEW0ivu?= =?us-ascii?Q?JfbKQyN0AsIIGMPEOCxKfDwwEtFExGTUyiQmYX8CjhNFyR1cK30smTGj71UN?= =?us-ascii?Q?2qo3XK7s7Ze93/Mu+6H2yLpWrrvigSrg871UTiAY6pcXRygt/Ec2z+jFaQkU?= =?us-ascii?Q?eVBgPHGkW/hGNujo6AlluE35Ec4IM3IOX5BTBi52+oDKWSeaYlmrF5KVykTl?= =?us-ascii?Q?Y365J88h8czsXcb9pAo6hn4/4iOvf8PbaijhPuN4CzqPA5dKsFnYQ8VmkxO+?= =?us-ascii?Q?mPnWgVbAukQxyn8ZchZoZ8kuUJX7vQXy1PZFL5Mq+Aalas8W1dOxwSurQh8/?= =?us-ascii?Q?9Q2cFClN3gwsE4XrzQ=3D?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3466; 6:lCC9Ge2WD7qcdY9IWVcqCdnpOxKGZ2OXftP1kCtAySJDjlYLegzrrv53eRC9moZQ5ZG6NtDDYYLJBP3nJVxoOhHKKGpOwNtIMx6k46V41TCxUZcZcukiWsUBFsktjjevnu0xK+pBq7pZdkJZECY/K/g1r8AIyDQGaTa+kNI/P8cDk9xkGkU8vk+QJcRK323f0aFnOzI7gtAeia9XsXnGoEghPQ/hmyTqla3H2v4y4eMgiMHMmsdOwv0MNzVXGt0E20a2iUKLQ83rX3rIQXPkzICzRqbEZvjGgZyJVyyuwdr+Ag9IZZTMWd5fPNInDt2cHAD/ugLDYRroNzodSzKyIBSK2ZKfZhLmVPSWzVoK8mo=; 5:xtd2I4b0FK/W/NYbAjyLKA/OEhQw7H5gYNx3N699Bi8li4YdeHxSUVT3EmDuqPINQWI7GTVLcQemJil3kJM92ZCtoVLBOMSfp1G/er1SAPPO+TXX+Pu54vLtnGPUvhp5jgRR0HfzbfDmZJUtdGTbWO0PdMfj+gPnnzQ842ReL7Q=; 24:0tx92b7b51hiD/Hvr3RvubEyJxyj/Zft48C+M4bcJTMTW+AlANk6BDPwlKvIktCaPJ2QUb6FNnLosHV8nCI0WQpfwPD159QzCyD5V/eMqis=; 7:goR+yLoFCYUHDagH62jZJGqiQzncX8k1JmApw5UEt7Jc1SJXDhtYnnYuJ1v0520wbqPLMENXTLsEvxn+egnDeWFaDQQDXMtmO0prbZV77PAaw8EQ7F36zXwXYpExvmp3yOOcru3SsrLYqE3RO76DY9OaO1RWPOtp3/yzBmowh3z2PCjYRsxAednkHfWC4R7aH2Zx+8NdfsjazFFT/nzKbn6+Ebyo1mq7y1WUZdq2C2rzLpUJxl4spb3pLRQ0W0TC SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Jan 2018 14:52:51.2011 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 30371ae4-b95a-4b5f-0e59-08d55839d4b0 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3466 Subject: [dpdk-dev] [PATCH v3 11/12] app/eventdev: add pipeline atq worker functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Jan 2018 14:52:57 -0000 Signed-off-by: Pavan Nikhilesh --- app/test-eventdev/test_pipeline_atq.c | 354 +++++++++++++++++++++++++++++++++- 1 file changed, 353 insertions(+), 1 deletion(-) diff --git a/app/test-eventdev/test_pipeline_atq.c b/app/test-eventdev/test_pipeline_atq.c index 642ef0375..720cb96b5 100644 --- a/app/test-eventdev/test_pipeline_atq.c +++ b/app/test-eventdev/test_pipeline_atq.c @@ -15,10 +15,362 @@ pipeline_atq_nb_event_queues(struct evt_options *opt) return rte_eth_dev_count(); } +static int +pipeline_atq_worker_single_stage_safe(void *arg) +{ + struct worker_data *w = arg; + struct test_pipeline *t = w->t; + const uint8_t dev = w->dev_id; + const uint8_t port = w->port_id; + struct rte_event ev; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + if (ev.sched_type == RTE_SCHED_TYPE_ATOMIC) { + pipeline_tx_pkt_safe(ev.mbuf); + w->processed_pkts++; + continue; + } + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + pipeline_event_enqueue(dev, port, &ev); + } + + return 0; +} + +static int +pipeline_atq_worker_single_stage_unsafe(void *arg) +{ + struct worker_data *w = arg; + struct test_pipeline *t = w->t; + const uint8_t dev = w->dev_id; + const uint8_t port = w->port_id; + struct rte_event ev; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + if (ev.sched_type == RTE_SCHED_TYPE_ATOMIC) { + pipeline_tx_pkt_unsafe(ev.mbuf, t); + w->processed_pkts++; + } else { + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + pipeline_event_enqueue(dev, port, &ev); + } + } + + return 0; +} + +static int +pipeline_atq_worker_single_stage_burst_safe(void *arg) +{ + int i; + struct worker_data *w = arg; + struct test_pipeline *t = w->t; + const uint8_t dev = w->dev_id; + const uint8_t port = w->port_id; + struct rte_event ev[BURST_SIZE + 1]; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + if (ev[i].sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt_safe(ev[i].mbuf); + ev[i].op = RTE_EVENT_OP_RELEASE; + w->processed_pkts++; + } else + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + + return 0; +} + +static int +pipeline_atq_worker_single_stage_burst_unsafe(void *arg) +{ + int i; + struct worker_data *w = arg; + struct test_pipeline *t = w->t; + const uint8_t dev = w->dev_id; + const uint8_t port = w->port_id; + struct rte_event ev[BURST_SIZE + 1]; + const uint16_t nb_ports = rte_eth_dev_count(); + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + pipeline_tx_flush(t, nb_ports); + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + if (ev[i].sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_unsafe_burst(ev[i].mbuf, t); + ev[i].op = RTE_EVENT_OP_RELEASE; + w->processed_pkts++; + } else + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + + return 0; +} + +static int +pipeline_atq_worker_multi_stage_safe(void *arg) +{ + struct worker_data *w = arg; + struct test_pipeline *t = w->t; + const uint8_t dev = w->dev_id; + const uint8_t port = w->port_id; + const uint8_t last_queue = t->opt->nb_stages - 1; + const uint8_t nb_stages = t->opt->nb_stages; + uint8_t *const sched_type_list = &t->sched_type_list[0]; + uint8_t cq_id; + struct rte_event ev; + + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + cq_id = ev.sub_event_type % nb_stages; + + if (cq_id == last_queue) { + if (ev.sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt_safe(ev.mbuf); + w->processed_pkts++; + continue; + } + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + } else { + ev.sub_event_type++; + pipeline_fwd_event(&ev, sched_type_list[cq_id]); + } + + pipeline_event_enqueue(dev, port, &ev); + } + return 0; +} + +static int +pipeline_atq_worker_multi_stage_unsafe(void *arg) +{ + struct worker_data *w = arg; + struct test_pipeline *t = w->t; + const uint8_t dev = w->dev_id; + const uint8_t port = w->port_id; + const uint8_t last_queue = t->opt->nb_stages - 1; + const uint8_t nb_stages = t->opt->nb_stages; + uint8_t *const sched_type_list = &t->sched_type_list[0]; + uint8_t cq_id; + struct rte_event ev; + + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + cq_id = ev.sub_event_type % nb_stages; + + if (cq_id == last_queue) { + if (ev.sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt_unsafe(ev.mbuf, t); + w->processed_pkts++; + continue; + } + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + } else { + ev.sub_event_type++; + pipeline_fwd_event(&ev, sched_type_list[cq_id]); + } + + pipeline_event_enqueue(dev, port, &ev); + } + return 0; +} + +static int +pipeline_atq_worker_multi_stage_burst_safe(void *arg) +{ + int i; + struct worker_data *w = arg; + struct test_pipeline *t = w->t; + const uint8_t dev = w->dev_id; + const uint8_t port = w->port_id; + uint8_t *const sched_type_list = &t->sched_type_list[0]; + const uint8_t last_queue = t->opt->nb_stages - 1; + const uint8_t nb_stages = t->opt->nb_stages; + uint8_t cq_id; + struct rte_event ev[BURST_SIZE + 1]; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + cq_id = ev[i].sub_event_type % nb_stages; + + if (cq_id == last_queue) { + if (ev[i].sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_unsafe_burst(ev[i].mbuf, t); + ev[i].op = RTE_EVENT_OP_RELEASE; + w->processed_pkts++; + continue; + } + + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + } else { + ev[i].sub_event_type++; + pipeline_fwd_event(&ev[i], + sched_type_list[cq_id]); + } + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + return 0; +} + +static int +pipeline_atq_worker_multi_stage_burst_unsafe(void *arg) +{ + int i; + struct worker_data *w = arg; + struct test_pipeline *t = w->t; + const uint8_t dev = w->dev_id; + const uint8_t port = w->port_id; + uint8_t *const sched_type_list = &t->sched_type_list[0]; + const uint8_t last_queue = t->opt->nb_stages - 1; + const uint8_t nb_stages = t->opt->nb_stages; + uint8_t cq_id; + struct rte_event ev[BURST_SIZE + 1]; + const uint16_t nb_ports = rte_eth_dev_count(); + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + pipeline_tx_flush(t, nb_ports); + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + cq_id = ev[i].sub_event_type % nb_stages; + + if (cq_id == last_queue) { + if (ev[i].sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_unsafe_burst(ev[i].mbuf, t); + ev[i].op = RTE_EVENT_OP_RELEASE; + w->processed_pkts++; + continue; + } + + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + } else { + ev[i].sub_event_type++; + pipeline_fwd_event(&ev[i], + sched_type_list[cq_id]); + } + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + return 0; +} + static int worker_wrapper(void *arg) { - RTE_SET_USED(arg); + struct worker_data *w = arg; + struct evt_options *opt = w->t->opt; + const bool burst = evt_has_burst_mode(w->dev_id); + const bool mt_safe = !w->t->mt_unsafe; + const uint8_t nb_stages = opt->nb_stages; + RTE_SET_USED(opt); + + /* allow compiler to optimize */ + if (nb_stages == 1) { + if (!burst && mt_safe) + return pipeline_atq_worker_single_stage_safe(arg); + else if (!burst && !mt_safe) + return pipeline_atq_worker_single_stage_unsafe( + arg); + else if (burst && mt_safe) + return pipeline_atq_worker_single_stage_burst_safe( + arg); + else if (burst && !mt_safe) + return pipeline_atq_worker_single_stage_burst_unsafe( + arg); + } else { + if (!burst && mt_safe) + return pipeline_atq_worker_multi_stage_safe(arg); + else if (!burst && !mt_safe) + return pipeline_atq_worker_multi_stage_unsafe(arg); + if (burst && mt_safe) + return pipeline_atq_worker_multi_stage_burst_safe( + arg); + else if (burst && !mt_safe) + return pipeline_atq_worker_multi_stage_burst_unsafe( + arg); + + } rte_panic("invalid worker\n"); } -- 2.15.1