From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-cys01nam02on0043.outbound.protection.outlook.com [104.47.37.43]) by dpdk.org (Postfix) with ESMTP id 8E18B1B2F3 for ; Tue, 16 Jan 2018 16:41:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=W/yitMKubwKqBg9SGCnWTP8g9BRzL8KIMI9sL22ri10=; b=Ymx9YfJgVHwLhZJ3ZU8h2naM8Wxw1zWkrOsXroB/RgRk9jzsm6JSlB2T7P1ZdOmvHnEFJc7DjeV2EAvjuaVv+BIfmQNUMMGfL4sNgIAmDXFKU6Y0vbQ1m7J27opU3Btmin+x4IgMiU/kV+sBMxCL0TNDFi+XBl+r6yD7lIt54jA= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Pavan.Bhagavatula@cavium.com; Received: from Pavan-LT.caveonetworks.com (111.93.218.67) by CY4PR07MB3464.namprd07.prod.outlook.com (10.171.252.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.407.7; Tue, 16 Jan 2018 15:41:01 +0000 From: Pavan Nikhilesh To: jerin.jacob@caviumnetworks.com, santosh.shukla@caviumnetworks.com, harry.van.haaren@intel.com, gage.eads@intel.com, hemant.agrawal@nxp.com, nipun.gupta@nxp.com, liang.j.ma@intel.com Cc: dev@dpdk.org, Pavan Nikhilesh Date: Tue, 16 Jan 2018 21:09:59 +0530 Message-Id: <20180116154003.3762-10-pbhagavatula@caviumnetworks.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180116154003.3762-1-pbhagavatula@caviumnetworks.com> References: <20171130072406.15605-1-pbhagavatula@caviumnetworks.com> <20180116154003.3762-1-pbhagavatula@caviumnetworks.com> MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [111.93.218.67] X-ClientProxiedBy: MWHPR02CA0016.namprd02.prod.outlook.com (10.168.209.154) To CY4PR07MB3464.namprd07.prod.outlook.com (10.171.252.145) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 56a66556-fb71-4775-b510-08d55cf78d17 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4604075)(2017052603307)(7153060)(7193020); SRVR:CY4PR07MB3464; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3464; 3:6pvanYNkXJjx4Qkk39yymt5s1UOHQ1/LWRzOICgd4FQDbnnPN6oXVs1y80BWc70UhQ4bKOekOOo5nsBcKtphYA2U3Kcn3pUVIWu9VCYe15Ph+TkTb4q9/oqW8ZH22cfTfxpWz2EgkAzcOZftJ0F33dleqitYWsHzo1KhCGjpU53fLr7C8uqPH0ioBenL1yB3uMZcxeNFDwA1P+oBU5Bio1o+C0ap8mMhewoSBlq62Wm4LifbbYqx0lFrt2U8y1Rt; 25:aR39oH2rC8Zy1DEw4gNGq2Keo7iwRpB6TW9A6oATqsoZC39KdCEitV/S708lD28c0tvt81QnS5DIM1SMJn4IrjKgs72CU80bZrKtDtcDX3DvhCADFpGB/mTYG7vzlUqpSQ/5ZcPXOSotBCbTLJQOeJmR7wh3DSK+2VrLOp+Nd87LS6q4FFw+xIY4O2lBGK7Bp+lGHfS1HWXHoaMp1UG32v8roMeSoalFZOTvRWlLgMfseXk51MyhB7c3iGQlXSibpybsIGdMjQkuH94RvUNSC8DrgfcaQk2VDs1Wv1KZOotXZLQL2oIzC5tuJYNIWlqcHADqVVMqD6psDPazsZZCXA==; 31:S0kiLjqRZk5h6pcMf76MfDBSCiLcNa+x2myvulE5wxyfkQkgdhhKhP8sdhVE5a8bYqxlPiMEiIOMmVhj2tYzGVBKm19QB/dxO3K+eT4fP0r06MxoKgi7Fq6lS3hlRmPvx49+89xZe6v40BmcRU0/c9Lo+hX7XFRHA+mjiiu77Hn3SgmgxNnQpTwc4DP0UFX/Xyc1BonEEdG/m4LyQGP6ye7UeagFNPggG46DDYnZBxU= X-MS-TrafficTypeDiagnostic: CY4PR07MB3464: X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3464; 20:g+XfnH/xvwLy49tLKX+GvdrGl0OdWyI1JK2nRlbf/xCmtF2bejeE4XedTNnVp+hJew6EqnKtYRv76Vj0x5wdZy63I78rWME+2XdmBwtTmkj7Z1VPk/mVP9Z00ZG6PQ9s2R3U5M+28GpkieusWE4yTaVdddvKETScF1N4Nc0n/dhQ7ErvaQJNUtXzi3Alo7WwIApA43ewycidPWjawQLE/GmtnbKP4GVUEoBl2o4TZd3kE9Ucsqdp42HlDkRA88ZP3Bx7AZCg/Xz0glCh0+Uk8QTUCpULpAJg9oMLKGyDFdZmunvRv8CGVPgjcBUNTLXd6IfcVvvsS1n6HvGUkk695yiUAsSOU/A3p3Eckx1TcdwxAO7h2Oul5oYFBazewgQzJYgi4w2THJTt9xn0lAWQiiIrfj50f81LaYtkDgTAxaqhaopVF6qU0bLAKzOZDQt3eFDsKwaz11N/Hsdcc7rhqYxRiGVb51xDeSk9tGlttbVWOvoW0o892N+ZQstXSopBH4dOwRRyrGXUeAcl6jVOceWx0LW2ZXS4z0CzheeONAhmKzjT1va4dfb1nnYGVXpUoIeBKYp4Bp4gJ4VB91VebAjxx81YBPGKobrbBATrIKE=; 4:+FQTlMz9TffqNtIwkq696WoLzoa8KTUVNH2RwSVXlw1r8YnFTLtEAU3ABYwynJFKTIq3hqx6gUGFCJhg0ECFuAUi53+HV/03Tv9O7oMFG9dD6jdvF+OdBsGIHyLYff+X7cRYL/NHyGppYMbRmP429dQHIy55z6GOZkv+XVZPNH/WrQqCKSey1KNkozWWmOdxyuUxmag5JFtXOKaBdiPio9EVwMoHkgh6WTAnd/ukMisPEG7Se7tl1ioZgZxZHUbv6omHwUOWlkJtt84i/5oqpOBdhvDWz7v3Q++KGD/I6WxMr4f82uYm4w0Y+z8ySDl6 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040470)(2401047)(5005006)(8121501046)(93006095)(3231023)(944501161)(10201501046)(3002001)(6041268)(20161123564045)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123558120)(20161123562045)(6072148)(201708071742011); SRVR:CY4PR07MB3464; BCL:0; PCL:0; RULEID:(100000803101)(100110400095); SRVR:CY4PR07MB3464; X-Forefront-PRVS: 0554B1F54F X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(39860400002)(366004)(346002)(39380400002)(396003)(376002)(189003)(199004)(81166006)(97736004)(8676002)(6116002)(5009440100003)(52116002)(305945005)(8936002)(107886003)(1076002)(51416003)(81156014)(25786009)(8656006)(4326008)(50466002)(50226002)(6486002)(5660300001)(3846002)(7736002)(16526018)(68736007)(69596002)(42882006)(106356001)(26005)(76176011)(53936002)(6512007)(59450400001)(47776003)(48376002)(16586007)(386003)(6506007)(66066001)(72206003)(2950100002)(36756003)(2906002)(478600001)(105586002)(316002)(53416004)(42262002); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3464; H:Pavan-LT.caveonetworks.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR07MB3464; 23:vxPiQitBSXhY5Lkv8ZWXv44DG0yAa1R0QsfIqgUUc?= =?us-ascii?Q?4U4oUhV5hVvJlWDFglTcE7xDtzgq453LHaJ8SutpIs1UnilyM09GRwQ7sjG9?= =?us-ascii?Q?+lpcx6hmPy8on55scRpzsQ4AT2aJ3jUY+sgODZoef5VhReAsfNMX18UQlIqA?= =?us-ascii?Q?I7fgzYeNQjp8lT8oPCwe50VQvr4A/jGvZSTLzvS+pDTtgDhH512IKYttmcAe?= =?us-ascii?Q?zIN4wNi/5bglPnyk69fBGOnEa2TplqcmfRVY+v4KLPgPDYjCljYwbli680Wt?= =?us-ascii?Q?S4QwdxHpjGVDUFoLZwOey9HmR9P7KRgj7oZtx/BnRLwDprVjasA2Zwg22Re5?= =?us-ascii?Q?id4RW+xToUWC4f/87Lpta+iGQP2RnzHV0qbdAynIeY7Ilj3PPkZLpqEp23ul?= =?us-ascii?Q?UMIeybbHz0DQh2eoQi/NzpE3h6Z4Ny/Di5xr0puJLchx7KelfFGpl0NKAcI1?= =?us-ascii?Q?YxYEjuAMlgDF3HIzhDERt7T9t1jrextMg2UJlkc1Rkx3rmwdbvxG0JMPpU6e?= =?us-ascii?Q?6wSVrlNgpGdnyX0qLv2gGcpq2El8wSdeBfI6suzswB5yDMWQKxu2PK27j6wD?= =?us-ascii?Q?jCk6gYMRxttBIbOHkANcaCkE7i65gJaUClKoP15x6qUT7vWZITskupqU7jAk?= =?us-ascii?Q?5k1Z9KCHQQZ11PbZQkNH+VLRCKwngbNbFauEDHlGbX7qYnVzjMDeqXKK3PjA?= =?us-ascii?Q?V/i4x2wbxTHl/iaAf95ZfuVH0GZKHYH2TqXS0RFZVrkICMGlPF2YYckKYUDg?= =?us-ascii?Q?1+zk9ZffaOgVRf7eBehkj5DMpSvB8rgc7uAFyAdCdST/Oli8Dm2JcvePIbJN?= =?us-ascii?Q?NJf++2C9Ky8v6s2Ii5aHZF6hUc0uCiVktTOT4Wb/gfG9+389kBp8YMtYBIVR?= =?us-ascii?Q?S4qxtz0dh98kDhfK3igI9o449+z2dJcPxVx+7WWQydEBLIItk9uEb7JZ9rIu?= =?us-ascii?Q?myOtfzwS6YTDdZGrU2s5xEOnCZ/h3tuHnYl8hJ+oXs8khD2l1fyoKe5p9jz+?= =?us-ascii?Q?fdULTzkuE+oQdUDePVe7daf9N9iUBb+iYJlBOpcjnd0GnlMupRxcLKM01uYg?= =?us-ascii?Q?HBFjhBTev7Hpsh+b3z4dRh/OtpNkOg2iKMmGlNK3OjPxJr/3ZDQmJ+YHMPYd?= =?us-ascii?Q?TlgiA0ZjZiGzZ1z7J1jpXV05zA0eSmPVs2BzsiR8HsyP3+1RmFnjXUJ+AGID?= =?us-ascii?Q?hRFmCdT9X+XHTqbgLS4HVoc9z+Qp/sOA4anD5xdSp8xeWCWphJZGwEhx37Vv?= =?us-ascii?Q?VmurgwnnA4RNUSjDGM=3D?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3464; 6:TvruFaOYzP2SdvURMHo3McmdGFzuVkwSIR9OlAdMIqdoAT7gt6sKW5VZTveDgk1TZl8mng5/D/S32CqhEcx787BeUpvhoMgkvGEOYWrZ78a2KVjkeVnaMd7Jl0wOjZpaNnvIlPHxhBmxXkFEueowarOyc03reG/VjpTbSY7zvm0kdymzb7azmwGFiJv6WsaE86hSjxpn6TiDxtQMjQwuPAXYkxJ2MGGTtPNsCq9XsRgpDKiYcMLKKa7BhWhBFbkOOpuzFg3h9l4vKTtYOLQ2HOd62W+78ZLq7lcDBXd/00O14ePdeiPiWAu4uyuQaaRHG0xC/2mDsu15jsJ+AdtIjvMwLQstYh3cQ2P7OAUGa7g=; 5:5gNiAnN4kt4GLaOIvBt1gyKkx5r23PYFHdU2uZxYcnRctvoKDOgYcTb+OQS8Pwxcve6j2tlSm3H6AMOwO0Y5rWrpONXyzVX0cJ385uBWxz91CmVrXNnTMOxwZ7cNf+RJiFhbdV2c54js1c1yL8MVMH/fxSoudTFwqpJCW9EGLDo=; 24:Bvgwy2c7vQF/VYBqLVFXuEgzbh7HtezBCGWADN9bApO7P1Z4jVKvFFJHnM+4JA4tXlCrIWlIRiUfxVw6zXV4QZIbEgyQmzhHKAebPZ+hyl8=; 7:8CY36wXWDKxM6DCqbEVN9AuYUaqrVfNOb6+mJt+XnIimANbzL+qluIZVhQumG2GKTiWwpPFuUK3GJatDZAXikSDRZrsYTphjoB06TDF6+6mcICOU5tnf5kyn2Mu5PEGRqPlgwtEKuLlxNq494FVgP0mpoKlz0ddgbvJDMxpTGiv7sMpxdDPITBtoEhsbycZANYdHxSycmJ/IVX8FHku72WBzT5HNSUhseMi6AdWeguUCXIM9BYqpeewEOWdHXM7/ SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Jan 2018 15:41:01.0830 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 56a66556-fb71-4775-b510-08d55cf78d17 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3464 Subject: [dpdk-dev] [PATCH v6 10/14] app/eventdev: add pipeline queue worker functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jan 2018 15:41:10 -0000 Signed-off-by: Pavan Nikhilesh Acked-by: Harry van Haaren --- app/test-eventdev/test_pipeline_common.h | 80 +++++++++ app/test-eventdev/test_pipeline_queue.c | 288 ++++++++++++++++++++++++++++++- 2 files changed, 367 insertions(+), 1 deletion(-) diff --git a/app/test-eventdev/test_pipeline_common.h b/app/test-eventdev/test_pipeline_common.h index db2517baf..5fb91607d 100644 --- a/app/test-eventdev/test_pipeline_common.h +++ b/app/test-eventdev/test_pipeline_common.h @@ -63,6 +63,86 @@ struct test_pipeline { uint8_t sched_type_list[EVT_MAX_STAGES] __rte_cache_aligned; } __rte_cache_aligned; +#define BURST_SIZE 16 + +#define PIPELINE_WROKER_SINGLE_STAGE_INIT \ + struct worker_data *w = arg; \ + struct test_pipeline *t = w->t; \ + const uint8_t dev = w->dev_id; \ + const uint8_t port = w->port_id; \ + struct rte_event ev + +#define PIPELINE_WROKER_SINGLE_STAGE_BURST_INIT \ + int i; \ + struct worker_data *w = arg; \ + struct test_pipeline *t = w->t; \ + const uint8_t dev = w->dev_id; \ + const uint8_t port = w->port_id; \ + struct rte_event ev[BURST_SIZE + 1] + +#define PIPELINE_WROKER_MULTI_STAGE_INIT \ + struct worker_data *w = arg; \ + struct test_pipeline *t = w->t; \ + uint8_t cq_id; \ + const uint8_t dev = w->dev_id; \ + const uint8_t port = w->port_id; \ + const uint8_t last_queue = t->opt->nb_stages - 1; \ + uint8_t *const sched_type_list = &t->sched_type_list[0]; \ + struct rte_event ev + +#define PIPELINE_WROKER_MULTI_STAGE_BURST_INIT \ + int i; \ + struct worker_data *w = arg; \ + struct test_pipeline *t = w->t; \ + uint8_t cq_id; \ + const uint8_t dev = w->dev_id; \ + const uint8_t port = w->port_id; \ + const uint8_t last_queue = t->opt->nb_stages - 1; \ + uint8_t *const sched_type_list = &t->sched_type_list[0]; \ + struct rte_event ev[BURST_SIZE + 1] + +static __rte_always_inline void +pipeline_fwd_event(struct rte_event *ev, uint8_t sched) +{ + ev->event_type = RTE_EVENT_TYPE_CPU; + ev->op = RTE_EVENT_OP_FORWARD; + ev->sched_type = sched; +} + +static __rte_always_inline void +pipeline_event_enqueue(const uint8_t dev, const uint8_t port, + struct rte_event *ev) +{ + while (rte_event_enqueue_burst(dev, port, ev, 1) != 1) + rte_pause(); +} + +static __rte_always_inline void +pipeline_event_enqueue_burst(const uint8_t dev, const uint8_t port, + struct rte_event *ev, const uint16_t nb_rx) +{ + uint16_t enq; + + enq = rte_event_enqueue_burst(dev, port, ev, nb_rx); + while (enq < nb_rx) { + enq += rte_event_enqueue_burst(dev, port, + ev + enq, nb_rx - enq); + } +} + +static __rte_always_inline void +pipeline_tx_pkt(struct rte_mbuf *mbuf) +{ + while (rte_eth_tx_burst(mbuf->port, 0, &mbuf, 1) != 1) + rte_pause(); +} + +static inline int +pipeline_nb_event_ports(struct evt_options *opt) +{ + return evt_nr_active_lcores(opt->wlcores); +} + int pipeline_test_result(struct evt_test *test, struct evt_options *opt); int pipeline_opt_check(struct evt_options *opt, uint64_t nb_queues); int pipeline_test_setup(struct evt_test *test, struct evt_options *opt); diff --git a/app/test-eventdev/test_pipeline_queue.c b/app/test-eventdev/test_pipeline_queue.c index 773c3ecaa..835fe0782 100644 --- a/app/test-eventdev/test_pipeline_queue.c +++ b/app/test-eventdev/test_pipeline_queue.c @@ -15,10 +15,296 @@ pipeline_queue_nb_event_queues(struct evt_options *opt) return (eth_count * opt->nb_stages) + eth_count; } +static int +pipeline_queue_worker_single_stage_tx(void *arg) +{ + PIPELINE_WROKER_SINGLE_STAGE_INIT; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + if (ev.sched_type == RTE_SCHED_TYPE_ATOMIC) { + pipeline_tx_pkt(ev.mbuf); + w->processed_pkts++; + } else { + ev.queue_id++; + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + pipeline_event_enqueue(dev, port, &ev); + } + } + + return 0; +} + +static int +pipeline_queue_worker_single_stage_fwd(void *arg) +{ + PIPELINE_WROKER_SINGLE_STAGE_INIT; + const uint8_t tx_queue = t->tx_service.queue_id; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + ev.queue_id = tx_queue; + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + pipeline_event_enqueue(dev, port, &ev); + w->processed_pkts++; + } + + return 0; +} + +static int +pipeline_queue_worker_single_stage_burst_tx(void *arg) +{ + PIPELINE_WROKER_SINGLE_STAGE_BURST_INIT; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + if (ev[i].sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt(ev[i].mbuf); + ev[i].op = RTE_EVENT_OP_RELEASE; + w->processed_pkts++; + } else { + ev[i].queue_id++; + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + } + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + + return 0; +} + +static int +pipeline_queue_worker_single_stage_burst_fwd(void *arg) +{ + PIPELINE_WROKER_SINGLE_STAGE_BURST_INIT; + const uint8_t tx_queue = t->tx_service.queue_id; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + ev[i].queue_id = tx_queue; + pipeline_fwd_event(&ev[i], RTE_SCHED_TYPE_ATOMIC); + w->processed_pkts++; + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + + return 0; +} + + +static int +pipeline_queue_worker_multi_stage_tx(void *arg) +{ + PIPELINE_WROKER_MULTI_STAGE_INIT; + const uint8_t nb_stages = t->opt->nb_stages + 1; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + cq_id = ev.queue_id % nb_stages; + + if (cq_id >= last_queue) { + if (ev.sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt(ev.mbuf); + w->processed_pkts++; + continue; + } + ev.queue_id += (cq_id == last_queue) ? 1 : 0; + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + } else { + ev.queue_id++; + pipeline_fwd_event(&ev, sched_type_list[cq_id]); + } + + pipeline_event_enqueue(dev, port, &ev); + } + return 0; +} + +static int +pipeline_queue_worker_multi_stage_fwd(void *arg) +{ + PIPELINE_WROKER_MULTI_STAGE_INIT; + const uint8_t nb_stages = t->opt->nb_stages + 1; + const uint8_t tx_queue = t->tx_service.queue_id; + + while (t->done == false) { + uint16_t event = rte_event_dequeue_burst(dev, port, &ev, 1, 0); + + if (!event) { + rte_pause(); + continue; + } + + cq_id = ev.queue_id % nb_stages; + + if (cq_id == last_queue) { + ev.queue_id = tx_queue; + pipeline_fwd_event(&ev, RTE_SCHED_TYPE_ATOMIC); + w->processed_pkts++; + } else { + ev.queue_id++; + pipeline_fwd_event(&ev, sched_type_list[cq_id]); + } + + pipeline_event_enqueue(dev, port, &ev); + } + return 0; +} + +static int +pipeline_queue_worker_multi_stage_burst_tx(void *arg) +{ + PIPELINE_WROKER_MULTI_STAGE_BURST_INIT; + const uint8_t nb_stages = t->opt->nb_stages + 1; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + cq_id = ev[i].queue_id % nb_stages; + + if (cq_id >= last_queue) { + if (ev[i].sched_type == RTE_SCHED_TYPE_ATOMIC) { + + pipeline_tx_pkt(ev[i].mbuf); + ev[i].op = RTE_EVENT_OP_RELEASE; + w->processed_pkts++; + continue; + } + + ev[i].queue_id += (cq_id == last_queue) ? 1 : 0; + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + } else { + ev[i].queue_id++; + pipeline_fwd_event(&ev[i], + sched_type_list[cq_id]); + } + + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + return 0; +} + +static int +pipeline_queue_worker_multi_stage_burst_fwd(void *arg) +{ + PIPELINE_WROKER_MULTI_STAGE_BURST_INIT; + const uint8_t nb_stages = t->opt->nb_stages + 1; + const uint8_t tx_queue = t->tx_service.queue_id; + + while (t->done == false) { + uint16_t nb_rx = rte_event_dequeue_burst(dev, port, ev, + BURST_SIZE, 0); + + if (!nb_rx) { + rte_pause(); + continue; + } + + for (i = 0; i < nb_rx; i++) { + rte_prefetch0(ev[i + 1].mbuf); + cq_id = ev[i].queue_id % nb_stages; + + if (cq_id == last_queue) { + ev[i].queue_id = tx_queue; + pipeline_fwd_event(&ev[i], + RTE_SCHED_TYPE_ATOMIC); + w->processed_pkts++; + } else { + ev[i].queue_id++; + pipeline_fwd_event(&ev[i], + sched_type_list[cq_id]); + } + } + + pipeline_event_enqueue_burst(dev, port, ev, nb_rx); + } + return 0; +} + static int worker_wrapper(void *arg) { - RTE_SET_USED(arg); + struct worker_data *w = arg; + struct evt_options *opt = w->t->opt; + const bool burst = evt_has_burst_mode(w->dev_id); + const bool mt_safe = !w->t->mt_unsafe; + const uint8_t nb_stages = opt->nb_stages; + RTE_SET_USED(opt); + + if (nb_stages == 1) { + if (!burst && mt_safe) + return pipeline_queue_worker_single_stage_tx(arg); + else if (!burst && !mt_safe) + return pipeline_queue_worker_single_stage_fwd(arg); + else if (burst && mt_safe) + return pipeline_queue_worker_single_stage_burst_tx(arg); + else if (burst && !mt_safe) + return pipeline_queue_worker_single_stage_burst_fwd( + arg); + } else { + if (!burst && mt_safe) + return pipeline_queue_worker_multi_stage_tx(arg); + else if (!burst && !mt_safe) + return pipeline_queue_worker_multi_stage_fwd(arg); + else if (burst && mt_safe) + return pipeline_queue_worker_multi_stage_burst_tx(arg); + else if (burst && !mt_safe) + return pipeline_queue_worker_multi_stage_burst_fwd(arg); + + } rte_panic("invalid worker\n"); } -- 2.14.1