From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0056.outbound.protection.outlook.com [104.47.33.56]) by dpdk.org (Postfix) with ESMTP id D226312009 for ; Fri, 12 Jan 2018 17:45:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=vxdDmbAIKPXaivWjn4KZnFUaRwD+af2qYCLNflvkEEY=; b=LfxfgWKDrMCQwJAyctpXxHh6PZd9kkIdsHsOQcXPNLZWHnn/O8EJh8uLx1bqW4OPvsEpyaEqkGaRIDp/DMJyrGxv+uQKt8ADAA0hvIUfWj9tJDmeNx4H6VyYZbNU0bon8adiCeRuGVM7+1n1atR9zKQ8USJnGPsp6xTIwDwC0lQ= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from Pavan-LT.caveonetworks.com (119.82.125.132) by DM5PR07MB3465.namprd07.prod.outlook.com (10.164.153.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.386.5; Fri, 12 Jan 2018 16:45:28 +0000 From: Pavan Nikhilesh To: jerin.jacob@caviumnetworks.com, santosh.shukla@caviumnetworks.com, harry.van.haaren@intel.com, gage.eads@intel.com, hemant.agrawal@nxp.com, nipun.gupta@nxp.com, liang.j.ma@intel.com Cc: dev@dpdk.org, Pavan Nikhilesh Date: Fri, 12 Jan 2018 22:14:13 +0530 Message-Id: <20180112164416.21374-10-pbhagavatula@caviumnetworks.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180112164416.21374-1-pbhagavatula@caviumnetworks.com> References: <20171130072406.15605-1-pbhagavatula@caviumnetworks.com> <20180112164416.21374-1-pbhagavatula@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [119.82.125.132] X-ClientProxiedBy: YQXPR0101CA0023.CANPRD01.PROD.OUTLOOK.COM (52.132.74.164) To DM5PR07MB3465.namprd07.prod.outlook.com (10.164.153.20) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6d5bbdb2-c037-42ea-e59e-08d559dbe582 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020084)(4652020)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:DM5PR07MB3465; X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3465; 3:6FFaw3fk8AMFcdahEjpRhih7V0kzdw62V+vm/Lfhd4ZM5bDQxpV90pexHkH9v7IcW8owlEXP2VNlQomnDKrX4awNY/MguBKLUNzmOKHXCHSXy4d5hSXiv4Q7ZZKCkWEEbzvHr0d3XXAZeXLTyPRJzrrfWDSovbiT9+OtgbsO7DhEMpDZFqsA4P53vN3CmRNf14GZBqF1EScJ0Xi+Gi4+M6AoC8WC4BbWx8oNIQyZokJHnv/20TA7XykijmaYap38; 25:ynvRDmPpGuiYqC1fa0WSnL8pUpgBLFPanLUFluPx7sLAO00O44zClO/z1BQRBLYbrNpIOVYACTvAntoGxQDbIB+g7QRm6VaWrIKBhqrHH6zcAyoPiMjrdaniWV8+ULL/djcvyc3Kc/8KE36ea9NsUwpSsb0g3LzyeqWAzhmFj/TegfqGpjNlHfJAVI48fRgKNyY8lXlqww8+g0IBWr8sO/0cIhDGIzxZ8kvhlcyPu/PHrOTGuTHIesIjvsrsUSTYdiO/I/dCPrpCbCCNt72GAboJlw5OYN800sClgZ4tDWPNpybdyAxIEFi9qFOX8POnbmvGpAAdhCCptj+cfTKBZw==; 31:IMaIKRYzRPWvh7re7KYkhHuUaB1JZniYMkOtFgOqmrMfYJfEIaLOO6LaEq2+GWy5YpUbyekrU/0tU9ZdpCh+Tg+OA+hsivLlopJVfZXxBaxXcUKWnhMi329hwbAFI9PhW1HYfsrPdZ2DFQAOlGgQ4ne+o8ZfbxAFZhcDZzbaFw5zYOlM6s5g50XV73s34QazDN6rdb+PM8pccmMwjE9yh+rgM0J9v20LxzHiYxRYMOo= X-MS-TrafficTypeDiagnostic: DM5PR07MB3465: X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3465; 20:+qcO5kV15y5oHPPjhVaLgvydihzPLFJKHwx3XcK4t2v9sEIpVjrrTreS/NgI9cZhc5ki/OQwnEpnBfCDB+jTK3x7DH9eWHvvhoyqSpfbU6j7WAiaqemiO6xJy88GMAlHzq0enBfU7F2DaYeSpsYgiFMM5DSHXaDMQGEZPyfxDFjr2nR117QzfBSr07I+V7uOx5DOQ74BVUUTTF1p9SlzIFGccguje3Bk59CrEtzP6QSw1XefwRRN/rRPvIqHcaWdDSXUz9E72IeV3M0bR0R4si3mJzqqsrNfYbltnc5fcVUPbkM151nf9WCmly7n+4+Ev0GbngQBbdEMogp9B6gBAbMUNMAx4dq4h9atjhxTtyNw1Ja3lsaBw9XswUOXdcDrWKagVwEpPb3oUJamGpSeqhg6aR/6PhTT/UP3XvRumzQMOPIZ7DYR3mHIpAK3MLiHrqorHkZUahw2nAJwQpzH+zvlTnN5xJn+AZmn1WO+kylUVDuEZY6D4eUbkzW2HD7yIT451i4Za/lOK2H75fm7VbAY49OJpT/aFe5DjTXsQz2eT8r/GUwQ5jdiZFXcfACEIDINFU5yVtIAWVc+uwgbbSumlpH+siNVtaZpO3h7OGI=; 4:5nXmwMihWmjsf/V2mKVaRGeU+y7FYXTLPXy5ARfJKfYoYsQg8xeoviiJBayD5zpE1yW2Sl3E4HvZgKG3eC1JlItlk3cOhkbVLkGaJ6oLLo69+EQNXXyaxAkVhjeolOArw+zr9ZbqhrgaBX9iliuAbUyW7Q9Clw8l1Zm6NsNeuH6qw62NNxc3kkwNa8ZX5vknXa6Nr0FMMjTwyIXwmrGhNAzreYjkzL6GogDye0o4PNuyk1pDSq4lSxR5lwu07UoHHdp/0eRm8SwmFMg9Lz47uA== X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(8121501046)(5005006)(3002001)(93006095)(10201501046)(3231023)(944501145)(6041268)(20161123560045)(20161123564045)(20161123558120)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:DM5PR07MB3465; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:DM5PR07MB3465; X-Forefront-PRVS: 0550778858 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(39860400002)(346002)(39380400002)(376002)(396003)(366004)(189003)(199004)(50226002)(76176011)(8676002)(6666003)(316002)(48376002)(53416004)(81156014)(53936002)(69596002)(16586007)(81166006)(59450400001)(386003)(16526018)(52116002)(6506007)(51416003)(1076002)(6486002)(2950100002)(105586002)(66066001)(6512007)(50466002)(106356001)(47776003)(4326008)(5660300001)(3846002)(42882006)(6116002)(7736002)(68736007)(97736004)(8656006)(478600001)(107886003)(8936002)(72206003)(36756003)(2906002)(305945005)(25786009)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:DM5PR07MB3465; H:Pavan-LT.caveonetworks.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; DM5PR07MB3465; 23:gn4+L6V0AYEAXUfxnbFLkJAGpnZxW0N4I/M+QQ4rh?= =?us-ascii?Q?lKM4azcn96FPb3qphgGwOJUIw+4EG9OJvXPlEDWu45GKnZ/qspwiZ80Zk2ob?= =?us-ascii?Q?nsPFe7XG1JlJNrWEGSojyMMzvciNmULaq1LzWLTStd3LTQ39t+kEVsND+LTm?= =?us-ascii?Q?kMUYioZYKBfq+kuFRyjQ5H8WpeMxFXMEs3c9E1SXPJZEFuOe6/YdtHKMI9Af?= =?us-ascii?Q?qkdhdlW2Vkj8DCpDNLxpLdlNxmNG0OQGG0rRzOc+fbUTnPq6j/RHrkPzQyCv?= =?us-ascii?Q?qpp9RjwJeExFkcUt/q8lg11Qb3/R2lAtjA0d7IMCnrlPFQpAPuG/aEWwK73R?= =?us-ascii?Q?vjHKvRDtkGMy08mAzuY+QjRCI+ETOm35Ut4yKL3s+dMkb8T1Z3LqTDU0p31p?= =?us-ascii?Q?8iIca4PNYwBfgszNizqSVTGvkGCpb/EMySHu5+mGZ9132U40iV/cLUUsjAwE?= =?us-ascii?Q?t4eldI3Noel2AjFTyco6D/VnS8n82KiL3EXNGpsARlplzlVzzmB1flrzAI1T?= =?us-ascii?Q?EaFxPVdha4kSLmQnDB4MhGBPzrsjtiugoKGuYRQZZT5jsouF/QmIy0A5awdh?= =?us-ascii?Q?d2t2493D+1ML/P3WUapOPq4VM2+wDsBCvFlbP1eqCGXh0g25OaCjMOTNcDmZ?= =?us-ascii?Q?3X1Zb7tOee8THqOBRuw0AUklRAuicBFz+Y9MFQH1MI5OflPzjWeCRH3KoAa7?= =?us-ascii?Q?OKPBa3p7R8VLs8oL/eKouTW+07Mmd4bXuFSspODRu4s6gz6E3+Nnv55/JMCr?= =?us-ascii?Q?lc5b9Wgsvg06qgU5D75biH4m27yoeihkSfCM9hxO1DNZWKg05Ly24LifEGYg?= =?us-ascii?Q?4MZYkJRmX+JsJLQWwaCVRb0qhyXU4rwfko5goP5sLECRoo2x4KW19jww79WG?= =?us-ascii?Q?/kOOgc0OSEfZ1FPuH41j0Qa2zOtrBkjb9qTGa4fDvSUELlzGHYzePiyW9JUT?= =?us-ascii?Q?5hWMLwQe1n1l8DY4lPvcM37/+Ftd5XPP6uvw29Wis8WXVbDjICxSZBQR1Prk?= =?us-ascii?Q?Sr0QD5GnD6Unn2LCtvS0NyEHSTmR5qALzls6e/FmlptOG1mVm7iK+E0DJaGe?= =?us-ascii?Q?S3J5XidR6Elp1N9bpbICoyHGHFngMRPFmt2PjzV6E16FCotToQ3KYsD/zYt1?= =?us-ascii?Q?E3PIPmIusXGQ7EregceaVBGbE56aPV+igePPByutZAjGzbFXalqRI//JKNwH?= =?us-ascii?Q?0/UEc7WQRsxZsZ0zYtXBZRTPOY8u4TiMFP2TBmhN9rtBKeZRM+FDrzWYg=3D?= =?us-ascii?Q?=3D?= X-Microsoft-Exchange-Diagnostics: 1; DM5PR07MB3465; 6:nNOsEUnpYNh3kWQj8GVMLFMaXraOCeurp3C8HzcaQwZtQ8IapMxEmAbd5bT9ELqEZ15rZsH28rM5SxruGb0I99zc2cwQ19Ksy0lRF+OP6Qk2matXVd5N9JYRfaSovfIqmHOLnTvk5gVYZmlRS33b9IRXhjZO9lkoP7ABVbCHz/yPjerzL3t3cxS+pfCCOSWwd+pJRAPXCTAWGFG0ONlPyIj/FtOV7urwfL6PzLYvPeazH/+ENX8a5wxTOEb+XEgxztXg5bnBVzVyUtVYENf4MhNzSD0GwWf7UvnN+0QeniF68AnyxkGoe1EIEAcz6IMifOwAk2CQ0hdE2ixX8G5iKGcauc7+fb2o/uKXqngmWE8=; 5:e5J7CJHY65pyL9s5ZSronB1B/5R2nFDXYrrbByHIM4/SONYbATYdkzHkfhIL2fyIrf5m+i1SwTt5ItX5DJ0lZF3qUiI4zMWY5bm5OLHJ3UVriyoTtURlYQwGKSkrKZpOuXVrv92NlPjtx6i/w3kgiUKHeQz3sRvmGAccCPAEqp4=; 24:UKeIAlQN8mWv1N08T+yUNklgRTrEe2MkgCGPSP6QLQi+nq2/rONW2OUKCR2JSqU5PMid5NJE7YH0o9/strLn43GzNCKOk6M4l56+xhylhVM=; 7:LHT+Go3bkvRSm0D6KHWmqNEr/JHV3S4aSSLXQJD/4Wffg/qr+jnn3CiXFyZJcOHrCu+pbcZDX2n/25+wG2UYcsgwGzPa253GjIbFv0gdxwdwbChpJ3M++vZY/aRPMXGDsY7MkMGL8Z9C3fKWYqp8dGXQoB84bYR4aeLwKHwVHsXONkeUrRPvmM5iZWqwy4J+kbBP0ojpzRmOy98yYNZbnp93ACd4+aO6/NI7/s91ZOWzuSnY8UrqvW6SsRbF6WzK SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jan 2018 16:45:28.4032 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 6d5bbdb2-c037-42ea-e59e-08d559dbe582 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR07MB3465 Subject: [dpdk-dev] [PATCH v4 10/13] app/eventdev: add pipeline queue worker functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jan 2018 16:45:38 -0000 Signed-off-by: Pavan Nikhilesh --- app/test-eventdev/test_pipeline_common.h | 80 +++++++++ app/test-eventdev/test_pipeline_queue.c | 289 ++++++++++++++++++++++++++++++- 2 files changed, 368 insertions(+), 1 deletion(-) diff --git a/app/test-eventdev/test_pipeline_common.h b/app/test-eventdev/test_pipeline_common.h index db2517baf..5fb91607d 100644 --- a/app/test-eventdev/test_pipeline_common.h +++ b/app/test-eventdev/test_pipeline_common.h @@ -63,6 +63,86 @@ struct test_pipeline { uint8_t sched_type_list[EVT_MAX_STAGES] __rte_cache_aligned; } __rte_cache_aligned; +#define BURST_SIZE 16 + +#define PIPELINE_WROKER_SINGLE_STAGE_INIT \ + struct worker_data *w = arg; \ + struct test_pipeline *t = w->t; \ + const uint8_t dev = w->dev_id; \ + const uint8_t port = w->port_id; \ + struct rte_event ev + +#define PIPELINE_WROKER_SINGLE_STAGE_BURST_INIT \ + int i; \ + struct worker_data *w = arg; \ + struct test_pipeline *t = w->t; \ + const uint8_t dev = w->dev_id; \ + const uint8_t port = w->port_id; \ + struct rte_event ev[BURST_SIZE + 1] + +#define PIPELINE_WROKER_MULTI_STAGE_INIT \ + struct worker_data *w = arg; \ + struct test_pipeline *t = w->t; \ + uint8_t cq_id; \ + const uint8_t dev = w->dev_id; \ + const uint8_t port = w->port_id; \ + const uint8_t last_queue = t->opt->nb_stages - 1; \ + uint8_t *const sched_type_list = &t->sched_type_list[0]; \ + struct rte_event ev + +#define PIPELINE_WROKER_MULTI_STAGE_BURST_INIT \ + int i; \ + struct worker_data *w = arg; \ + struct test_pipeline *t = w->t; \ + uint8_t cq_id; \ + const uint8_t dev = w->dev_id; \ + const uint8_t port = w->port_id; \ + const uint8_t last_queue = t->opt->nb_stages - 1; \ + uint8_t *const sched_type_list = &t->sched_type_list[0]; \ + struct rte_event ev[BURST_SIZE + 1] + +static __rte_always_inline void +pipeline_fwd_event(struct rte_event *ev, uint8_t sched) +{ + ev->event_type = RTE_EVENT_TYPE_CPU; + ev->op = RTE_EVENT_OP_FORWARD; + ev->sched_type = sched; +} + +static __rte_always_inline void +pipeline_event_enqueue(const uint8_t dev, const uint8_t port, + struct rte_event *ev) +{ + while (rte_event_enqueue_burst(dev, port, ev, 1) != 1) + rte_pause(); +} + +static __rte_always_inline void +pipeline_event_enqueue_burst(const uint8_t dev, const uint8_t port, + struct rte_event *ev, const uint16_t nb_rx) +{ + uint16_t enq; + + enq = rte_event_enqueue_burst(dev, port, ev, nb_rx); + while (enq < nb_rx) { + enq += rte_event_enqueue_burst(dev, port, + ev + enq, nb_rx - enq); + } +} + +static __rte_always_inline void +pipeline_tx_pkt(struct rte_mbuf *mbuf) +{ + while (rte_eth_tx_burst(mbuf->port, 0, &mbuf, 1) != 1) + rte_pause(); +} + +static inline int +pipeline_nb_event_ports(struct evt_options *opt) +{ + return evt_nr_active_lcores(opt->wlcores); +} + int pipeline_test_result(struct evt_test *test, struct evt_options *opt); int pipeline_opt_check(struct evt_options *opt, uint64_t nb_queues); int pipeline_test_setup(struct evt_test *test, struct evt_options *opt); diff --git a/app/test-eventdev/test_pipeline_queue.c b/app/test-eventdev/test_pipeline_queue.c index 773c3ecaa..ea784d2c6 100644 --- a/app/test-eventdev/test_pipeline_queue.c +++ b/app/test-eventdev/test_pipeline_queue.c @@ -15,10 +15,297 @@ pipeline_queue_nb_event_queues(struct evt_options *opt) return (eth_count * opt->nb_stages) + eth_count; } +static int +pipeline_queue_worker_single_stage_tx(void *arg) +{ + PIPELINE_WROKER_SINGLE_STAGE_INIT; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + if (ev.sched_type == RTE_SCHED_TYPE_ATOMIC) { + pipeline_tx_pkt(ev.mbuf); + w->processed_pkts++; + } else { + ev.queue_id++; + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + pipeline_event_enqueue(dev, port, &ev); + } + } + + return 0; +} + +static int +pipeline_queue_worker_single_stage_fwd(void *arg) +{ + PIPELINE_WROKER_SINGLE_STAGE_INIT; + const uint8_t tx_queue = t->tx_service.queue_id; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + ev.queue_id = tx_queue; + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + pipeline_event_enqueue(dev, port, &ev); + w->processed_pkts++; + } + + return 0; +} + +static int +pipeline_queue_worker_single_stage_burst_tx(void *arg) +{ + PIPELINE_WROKER_SINGLE_STAGE_BURST_INIT; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + if (ev[i].sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt(ev[i].mbuf); + ev[i].op = RTE_EVENT_OP_RELEASE; + w->processed_pkts++; + } else { + ev[i].queue_id++; + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + } + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + + return 0; +} + +static int +pipeline_queue_worker_single_stage_burst_fwd(void *arg) +{ + PIPELINE_WROKER_SINGLE_STAGE_BURST_INIT; + const uint8_t tx_queue = t->tx_service.queue_id; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + ev[i].queue_id = tx_queue; + pipeline_fwd_event(&ev[i], RTE_SCHED_TYPE_ATOMIC); + w->processed_pkts++; + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + + return 0; +} + + +static int +pipeline_queue_worker_multi_stage_tx(void *arg) +{ + PIPELINE_WROKER_MULTI_STAGE_INIT; + const uint8_t nb_stages = t->opt->nb_stages + 1; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + cq_id = ev.queue_id % nb_stages; + + if (cq_id >= last_queue) { + if (ev.sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt(ev.mbuf); + w->processed_pkts++; + continue; + } + ev.queue_id += (cq_id == last_queue) ? 1 : 0; + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + } else { + ev.queue_id++; + pipeline_fwd_event(&ev, sched_type_list[cq_id]); + } + + pipeline_event_enqueue(dev, port, &ev); + } + return 0; +} + +static int +pipeline_queue_worker_multi_stage_fwd(void *arg) +{ + PIPELINE_WROKER_MULTI_STAGE_INIT; + const uint8_t nb_stages = t->opt->nb_stages + 1; + const uint8_t tx_queue = t->tx_service.queue_id; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + cq_id = ev.queue_id % nb_stages; + + if (cq_id == last_queue) { + ev.queue_id = tx_queue; + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + w->processed_pkts++; + } else { + ev.queue_id++; + pipeline_fwd_event(&ev, sched_type_list[cq_id]); + } + + pipeline_event_enqueue(dev, port, &ev); + } + return 0; +} + +static int +pipeline_queue_worker_multi_stage_burst_tx(void *arg) +{ + PIPELINE_WROKER_MULTI_STAGE_BURST_INIT; + const uint8_t nb_stages = t->opt->nb_stages + 1; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + cq_id = ev[i].queue_id % nb_stages; + + if (cq_id >= last_queue) { + if (ev[i].sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt(ev[i].mbuf); + ev[i].op = RTE_EVENT_OP_RELEASE; + w->processed_pkts++; + continue; + } + + ev[i].queue_id += (cq_id == last_queue) ? 1 : 0; + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + } else { + ev[i].queue_id++; + pipeline_fwd_event(&ev[i], + sched_type_list[cq_id]); + } + + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + return 0; +} + +static int +pipeline_queue_worker_multi_stage_burst_fwd(void *arg) +{ + PIPELINE_WROKER_MULTI_STAGE_BURST_INIT; + const uint8_t nb_stages = t->opt->nb_stages + 1; + const uint8_t tx_queue = t->tx_service.queue_id; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + cq_id = ev[i].queue_id % nb_stages; + + if (cq_id == last_queue) { + ev[i].queue_id = tx_queue; + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + w->processed_pkts++; + } else { + ev[i].queue_id++; + pipeline_fwd_event(&ev[i], + sched_type_list[cq_id]); + } + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + return 0; +} + static int worker_wrapper(void *arg) { - RTE_SET_USED(arg); + struct worker_data *w = arg; + struct evt_options *opt = w->t->opt; + const bool burst = evt_has_burst_mode(w->dev_id); + const bool mt_safe = !w->t->mt_unsafe; + const uint8_t nb_stages = opt->nb_stages; + RTE_SET_USED(opt); + + /* allow compiler to optimize */ + if (nb_stages == 1) { + if (!burst && mt_safe) + return pipeline_queue_worker_single_stage_tx(arg); + else if (!burst && !mt_safe) + return pipeline_queue_worker_single_stage_fwd(arg); + else if (burst && mt_safe) + return pipeline_queue_worker_single_stage_burst_tx(arg); + else if (burst && !mt_safe) + return pipeline_queue_worker_single_stage_burst_fwd( + arg); + } else { + if (!burst && mt_safe) + return pipeline_queue_worker_multi_stage_tx(arg); + else if (!burst && !mt_safe) + return pipeline_queue_worker_multi_stage_fwd(arg); + else if (burst && mt_safe) + return pipeline_queue_worker_multi_stage_burst_tx(arg); + else if (burst && !mt_safe) + return pipeline_queue_worker_multi_stage_burst_fwd(arg); + + } rte_panic("invalid worker\n"); } -- 2.14.1