From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM02-BL2-obe.outbound.protection.outlook.com (mail-bl2nam02on0063.outbound.protection.outlook.com [104.47.38.63]) by dpdk.org (Postfix) with ESMTP id C85B0DE5 for ; Wed, 8 Feb 2017 11:23:26 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=FwTH2i5UNT1FXjHO/ZJ9Qf6LEESHwBVfn7rrfB6P6mk=; b=bbM/HaKeyUt/IGvB5Xkhdvv6oaMAZt6w1yWOPceaSKMLMgmDZqEKWVRKw9Uaeqx+ZO/Tp+KU+O/w5LQqudim5tDIE0NOOPJViwOR6ehR5WRA+ZFBPtp9ufczEUhslJ/+hO02NCCRTbKqLLDcaBPTywQku3ZXLFKVQant2SIB1fw= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Jerin.Jacob@cavium.com; Received: from localhost.localdomain (122.167.145.188) by BLUPR0701MB1714.namprd07.prod.outlook.com (10.163.85.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.888.16; Wed, 8 Feb 2017 10:23:22 +0000 Date: Wed, 8 Feb 2017 15:53:08 +0530 From: Jerin Jacob To: Harry van Haaren CC: , Bruce Richardson , David Hunt , , , Message-ID: <20170208102306.GA19597@localhost.localdomain> References: <1484580885-148524-1-git-send-email-harry.van.haaren@intel.com> <1485879273-86228-1-git-send-email-harry.van.haaren@intel.com> <1485879273-86228-16-git-send-email-harry.van.haaren@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <1485879273-86228-16-git-send-email-harry.van.haaren@intel.com> User-Agent: Mutt/1.7.1 (2016-10-04) X-Originating-IP: [122.167.145.188] X-ClientProxiedBy: PN1PR01CA0107.INDPRD01.PROD.OUTLOOK.COM (10.174.144.23) To BLUPR0701MB1714.namprd07.prod.outlook.com (10.163.85.140) X-MS-Office365-Filtering-Correlation-Id: d169cc39-778b-414d-e7ae-08d4500c8411 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001); SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 3:o7sEoUEo/ydvPPQR3+dS+OXQIOpCYk6QMZtGtOV/USlUhbVsrnFZwpmmCDLPst72QI8/lOx3ReOhDFc+QQeUSqVXPCxkQpErpRz6sNrnHKc3M0hHU950zXOY3gcLxf3g4N6iwiUoXFM2W7ru3R+ZytDE9TQvkcr6R4t+IyN7TZkm2+16kRAkTZHHLcrHejO2iMjNmCjKgwE6UjnHLF+eIdqEqbqTlIRqq36M5dhMEQEoQ3GiGa0euVsPEmgIsimFqWGxxHu3GipaU73ncSkTBw==; 25:v6VGFqwTaHL5R6tRf2CWb1qbU2rjauGVpCq6o84DPMhXZLoBurQvNYW2U+Fv9edtTQ0ZV48D/TvV7JW2FcM/0voxYfah4zdYeE09k8yQF9jTQn7JDpDSebdix9M4um7VAhcDmkuHUSp25VpU60lIY+q/zLs5D7059nahuFXN5dtDSqxtcafvp2VvOzYliykfZESBfFmGY0a1LBYt/aJxNznOiQWmPX/YrvVejO+/0tehQwbRpaY9BxmdnCs8LxtYtrjQ6u7lPl6i+yNpqhKETECl2eFsPm7kAHHGP4BhrO7fvDuzmYMy7dD7VD2wg7An8oI/GdmXuU7iRWEfNSFvLwBlEx3GI86xPNHMKhJ+COkBVS9TW8IYDzXS5Pr/zZq4vFoN1JTCXA2iJeauymG8jsjiE4NYFoENpyFUDnZ+Jp3nFcDIriTf/Dgo3+NxZDqmz1Pbnlfv2uAgEQvHHomNeg== X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 31:tSodsIyqgBxG1pSb+lxNXAzPE0cfF1GSTi5zZRd6G3q1vUFnjhtQeZ3sel85t6ImgOhuB9f+/Np0vocuUIDUzvZn/PdwPn9RwDLRyZUWOySqLK6PJkLqtEuWRb55gndKf2ocfYv/t7IEAx5SGLwIA35+YUL99J2JMUiF6e5JE5ICgRtmm9ks8y7DW0J6mNkc1H00jL4rP40dh1PVc4KMoLXQgM38SGiskV/lMhrq4o5qzi6mSUUSHLZ+MBHzm3hgvMh0AN4Ez73yUhly3V6PBw==; 20:foof4STw1dQIuJ1uHFNrEKCc04OCuaSlG64oUHqTTcMVJqqhZ5MeTG/PAEsL9Li7+vWel9rQMC+70hXm51eK5f4CRDB8qLo2ItzDFb5mZA703mzA4ZL/dAPz62NPCb/o1Wa8kmj5Is390XWkSNdTIB3fSeNHT+DZHCX9ti9o31dVmyfNVKh7pnWXP+8k2fNFlRpc0X5euanKryFkiElvbkIU9/9DsR6ugycxT0h+A2y/A0WbJ338uKKNQdDLEKYozy7gMKABR7a+EPFjGnaP0Qq5xVf7VJNPTo5UMiUAaKA3OI7OoFgYUVttRdMfgn0FX/ZRUPhmD1Rpp8i1xXMY/RSEYLF4aQZXFxtv0Dkx8xiUbAl1t5pGCI4dlj/GxCyZhiX5pmjrhyN7YRhu3DIIcxp7wY6i+eKDVupMzu80elEy9tF1/brYnviL1ISc+Cbj25ZNXOakckUn5qyKUGs7/Ybdk5XAOwuWxeiPtkKzUROja8LCN7s9wZRTmynvg4BIZ+r42JDQfAtrmAA0Wy6ixkUkaM/JBIpeFkHTJ2FDqcKXh6GijSbGBTqeVjVe1bZvYnRg5WKsNFK4Tm0F0+B7Af2nyG0PzJL5PjchCS7GDLI= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(100405760836317)(155532106045638)(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040375)(601004)(2401047)(5005006)(20170203043)(2017020702029)(8121501046)(3002001)(10201501046)(6041248)(20161123560025)(20161123564025)(20161123562025)(20161123555025)(20161123558025)(6072148); SRVR:BLUPR0701MB1714; BCL:0; PCL:0; RULEID:; SRVR:BLUPR0701MB1714; X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 4:XgKt8HWq04X27+qf+NcrmWEjyxATFJA23ckSvIFyMdhSJ/ZSxMyBbcKiCCrAb4CEo8XYtzIX1shvHAZibQm/3oP6xnKSV2avhwFJHP7Ckjrt+e7kre9n7pG138aBPWHAwtXl1O1su8Vv0Mpya5vdcoAG3M64900RjAipA/r6SNODppZyaNaaJGEZd7Ucy/pzjfyqXCOdcXF3r10Q5jAIX1mxkg515Cx5xz0wqG1dychM2sUs5naIW7Mu3bXg5Oh2xg7mUFX6EUBisGCxNEUUZGdpIlXmWJjdc7KLhWXG2UFE7/H2IQEMHVPBNSNNMPjOWai8iETRVcVzfPDrYm6I3Z/DPon50aqB/fzVMPItePuOYfWSFr8wEKadJkhkdkH2AEu2rbQqeNyvYdwsHFccnYmtHTOpWmEdiBeMI+95PVvbVvIpX+2Pujj5XCtBnUIiWP7AtCdOc7QEtZ2NopOKXgxyHxKsqWXlEcz6Jo588PoeJ/6t0YJi7lF4/vlyP+9rGLdJJ2K1/omFU4QEeB/LVnIDDSKFl50PJv8d1oh3n/i4O6ArlYMbFN4khLcOOWuMGVEJ9+NHAtxShoesURGehMsgbc2HU5UBSONk0GzQKWghgimNLv3ATt6ftvio0CKeiN2Enb3xeMz/KxSL6Fr7DaxnuXtaBI5QNW4FSbOT5ipHQHK/2mfl+kK9ObQY7uXEEC5AWoJ/RbDSvSf+PbNjAVNQ3iIi+C2keUMdprDtkHkV8WUoYCjrcyW9QcF2mik1O/rjQC/E/E7gfT3Ds29sGw== X-Forefront-PRVS: 0212BDE3BE X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6069001)(6009001)(7916002)(39450400003)(199003)(189002)(24454002)(105586002)(97756001)(6506006)(561944003)(229853002)(47776003)(189998001)(66066001)(92566002)(4001350100001)(106356001)(68736007)(2906002)(23726003)(61506002)(6116002)(3846002)(6666003)(305945005)(42882006)(2950100002)(6916009)(76176999)(50466002)(54356999)(53936002)(101416001)(46406003)(81166006)(54906002)(50986999)(8676002)(1076002)(81156014)(6306002)(83506001)(4326007)(33656002)(5660300001)(9686003)(7736002)(8656002)(55016002)(966004)(42186005)(53376002)(6246003)(110136004)(38730400002)(25786008)(97736004)(7099028)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:BLUPR0701MB1714; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BLUPR0701MB1714; 23:Alh352g+Dd42YoMrk0//5bOHMrvo8zC4O6qSnUL?= =?us-ascii?Q?A2oOymbcxR2Ls/qRbmc2kzUeHh6qyWrbuIOcng3PHuaYkvBn+7ZnK0KxsTiW?= =?us-ascii?Q?G8f7lerTICBmDfGN/lYyxjCpB5bvhL7CQHO3t+vbVILkPyVqPBVeKFbonmGn?= =?us-ascii?Q?RM7hdSAx4vtIgIEjYJq4MQRQWvFf6dARh7vqLP3dJtJto570BqXEPeh2yOm3?= =?us-ascii?Q?pzBTGHOrXzQcLw90o1XpO9qcoRg1JQhHh8t0prN8YWlUsLh/4f7TFbHounQS?= =?us-ascii?Q?Xhl63ltNa5HaWJXHm3iZmzUtiSVaNfgA2GrNzVZh/Aqgm5oHo25o1CrWYQLh?= =?us-ascii?Q?9WinpUHC32jncdjzU5yktwqmMfAjXCXAYd+qCQimnIH0riEsXB0zKo0GLSWA?= =?us-ascii?Q?5IXfXKCDY1odZWd5CfOqpMjRGz6NtS9b5kTydZfSwFI1+Yy2BZ18NwhMVWCX?= =?us-ascii?Q?FLfzJKfWYfClAXC+ZZ7jX3QBcbTc54+VYD0OuGy2p20qafKcxw3El5Jj75MD?= =?us-ascii?Q?kYY7ovgBUyVkZ2yfT+xus6DfXW03vi/hNuPHAT8cQzCGtmFGe2CZzLAcdwqM?= =?us-ascii?Q?177OT4uPbFFerGO4bVsF7HnoUZvtTFBHKq59h7cOzANN3yKL3HHJ/eDqGBI8?= =?us-ascii?Q?GtCjnNN5MBea4vo766GvgcSmmbMPHLMKYpiDOPbht2VPhyV1i2mU8XbysJOv?= =?us-ascii?Q?A5iR7j27qrzlx5JhCNlmguE9K2E3qbJEBYwXrftRO+Mw0gHIxEDSU2zPTvlV?= =?us-ascii?Q?IPyISm988aZ78LPjFSctcJzZgJAqWsAQiNm5J5fe0MBhboseFaMvjnGG5484?= =?us-ascii?Q?8cGiqG66Q8T8p4w5A9fy5FCmDP4rzY2OqWuQF1eJ9fSebF8x45JQbKenBX8j?= =?us-ascii?Q?JLiqqY1O9Ggf0mNGrsjTIKUy4TarnWe7Gcf0Ylwjvb2id6w+lOxcmLZCl/q+?= =?us-ascii?Q?/4v7HQXVbzFW1bFcGKu5L6NweED7skIylIr4huzJb4kT2qWddcEylzMxWvwi?= =?us-ascii?Q?iERe/V0PGnlpkS0EuzNXGviVoumlMqjHeBKQgtiBnypMeGlDqo2jAqeq09On?= =?us-ascii?Q?vKEA8RqFzINOmCcsms+uBI1tYScim6mJ1EcHP8X+c4mSNqwESU+UvcMyY2r5?= =?us-ascii?Q?sTerEdKsXSGd8GHyCHzt4l2twqd15S13MnQuFTh936BB+fsaOVpKv14sf07r?= =?us-ascii?Q?hcagKpeT/BasZ4lzFrwS6HVjGIcM+DlOpNBMnfbhWrvNO1SBDg0FYmuQvg5h?= =?us-ascii?Q?LpkyJCyKSAuevUppR59ruqeS6FwOHUYdSGj5s4uqSc79zRdAgtrZt/9o4FDG?= =?us-ascii?Q?fDct5PfCLnFcEZ2+CwLGAX0ZijMoMRIfZflIdVLeO10b25ePeFuINMPbIrld?= =?us-ascii?Q?oG/Z0uCDUGMxR6qUhDSiJpjD5ONDF/TNgWj/++wwUcLl6TsYOPWtqp6qzDfA?= =?us-ascii?Q?Dqa1bI1vnnw=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 6:WWF7R0dIl+ueoWIg+4WBaZEkH6iRYkmKqSDbQH0/gBdY6Qm7I90wBC33ywXw5Ame321TDIXRrvUvHNRkRcMH7LeGFFI6FZFxgHmRI+FsVHtoYIvVtRgtfd47kiKsLoBgc3lt28tpP6xpWNh2qXDPZ7fDQRySOkpAUPEPXV/s6ei0EMWUnvKdY1uj9Frc2cThYE/aY3bnamqNkIdUMo4CKtuognrwMBvnKrZRFKf7L1FuttM+KjSuB/pzYWIqH+G5KveP7x3NyVJIkV2GWZtriqwRkzCZP9nL6TwUF0uUz2hoRJjnoR/f+zU2PjpRvaAosq4HkVT+u9+KoIUNZr9MAXIcUtKxiOkmmsMvxcZqzR4y2lrSb5wA5rCsA2HlmMGgqGjMo+MAEVzm/+Txbyt8mg==; 5:TCjvQjd8NVKNGUd30S0cWQAJq75Lv0JI0L+hueZkIE86EsTz/lfgCxerfyeYQL7cinqIbdJvjmgdrAfGPgI3dJYf/vhUi1eIjGWOYOtIbNuq0F/pVDMGhO/lpJ9j6/LH4DEw2imNOh0K5H7GalAnFw==; 24:reC00yrmMfCejAcDuWdP2Ud0bOsnFsWmtaUOuzQhPPW7ydY2ajqO9PfGXZ8Rl62jJBpGXbo4pj6hW+n6vfqNcURjcZSYD6KKCIIrEjpSEOo= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BLUPR0701MB1714; 7:73R/wjobUn2wECfAzRjT+gfsGaeg084xSkW2xvzGtUGi/tSyM4i3RffML7s+t3OBDdBBO4RUihHDJ6AQDEFt0nTv0ydKWTC+V9FocXaBPwcy/jhCX53U0tVNXkpN5FqddheRklJv0aSQyGCml8vxeo1G3ZbjxTKPSwRjEWQmSz4Syhhr0Kcv1J8s7ayr7Mo0uZPaCbltSJqEnsgSFFIh/tw2QhALth2YIoURYN5YCKgDoimT1Kjmw+fOm7WqkHgHZJ6dOGDbln0wRk+tdcWwtQbU9lbffK2H535sMb+/CoRnbCa1zfUz6hlko6EvTLzolxs4/2xnLjiXgTneUaZ1kWY6mF5x3rS24gUlKXtoyXi9LcFdP2GwkXtiiG7LBOPIndgdEo0HruGg6Ghdx9ifbmaa+ARgsZoFSO9hznPjT3vKZVqUevDbyoALtYBA1SYs4+Mf0SrBOA8a7x5/OwWt04BJO1tFYycDpjmpoC0A7TqBsFbOzoxx0Sf5NvvtU7oQj7fMqIajM7XQFHkkInxt2g== X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Feb 2017 10:23:22.6955 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR0701MB1714 Subject: Re: [dpdk-dev] [PATCH v2 15/15] app/test: add unit tests for SW eventdev driver X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Feb 2017 10:23:27 -0000 On Tue, Jan 31, 2017 at 04:14:33PM +0000, Harry van Haaren wrote: > From: Bruce Richardson > > Since the sw driver is a standalone lookaside device that has no HW > requirements, we can provide a set of unit tests that test its > functionality across the different queue types and with different input > scenarios. > Thanks for SW driver specific test cases. It provided me a good insight of expected application behavior from SW driver perspective and in turn it created some challenge in portable applications. I would like highlight a main difference between the implementation and get a consensus on how to abstract it? Based on existing header file, We can do event pipelining in two different ways a) Flow-based event pipelining b) queue_id based event pipelining I will provide an example to showcase application flow in both modes. Based on my understanding from SW driver source code, it supports only queue_id based event pipelining. I guess, Flow based event pipelining will work semantically with SW driver but it will be very slow. I think, the reason for the difference is the capability of the context definition. SW model the context is - queue_id Cavium HW model the context is queue_id + flow_id + sub_event_type + event_type AFAIK, queue_id based event pipelining will work with NXP HW but I am not sure about flow based event pipelining model with NXP HW. Appreciate any input this? In Cavium HW, We support both modes. As an open question, Should we add a capability flag to advertise the supported models and let application choose the model based on implementation capability. The downside is, a small portion of stage advance code will be different but we can reuse the STAGE specific application code(I think it a fair trade off) Bruce, Harry, Gage, Hemant, Nipun Thoughts? Or any other proposal? I will take an non trivial realworld NW use case show the difference. A standard IPSec outbound processing will have minimum 4 to 5 stages stage_0: -------- a) Takes the pkts from ethdev and push to eventdev as RTE_EVENT_OP_NEW b) Some HW implementation, This will be done by HW. In SW implementation it done by service cores stage_1:(ORDERED) ------------------ a) Receive pkts from stage_0 in ORDERED flow and it process in parallel on N of cores b) Find a SA belongs that packet move to next stage for SA specific outbound operations.Outbound processing starts with updating the sequence number in the critical section and followed by packet encryption in parallel. stage_2(ATOMIC) based on SA ---------------------------- a) Update the sequence number and move to ORDERED sched_type for packet encryption in parallel stage_3(ORDERED) based on SA ---------------------------- a) Encrypt the packets in parallel b) Do output route look-up and figure out tx port and queue to transmit the packet c) Move to ATOMIC stage based on tx port and tx queue_id to transmit the packet _without_ losing the ingress ordering stage_4(ATOMIC) based on tx port/tx queue ----------------------------------------- a) enqueue the encrypted packet to ethdev tx port/tx_queue 1) queue_id based event pipelining ================================= stage_1_work(assigned to event queue 1)# N ports/N cores establish link to queue 1 through rte_event_port_link() on_each_cores_linked_to_queue1(stage1) while(1) { /* STAGE 1 processing */ nr_events = rte_event_dequeue_burst(ev,..); if (!nr_events); continue; sa = find_sa_from_packet(ev.mbuf); /* move to next stage(ATOMIC) */ ev.event_type = RTE_EVENT_TYPE_CPU; ev.sub_event_type = 2; ev.sched_type = RTE_SCHED_TYPE_ATOMIC; ev.flow_id = sa; ev.op = RTE_EVENT_OP_FORWARD; ev.queue_id = 2; /* move to stage 2(event queue 2) */ rte_event_enqueue_burst(ev,..); } on_each_cores_linked_to_queue2(stage2) while(1) { /* STAGE 2 processing */ nr_events = rte_event_dequeue_burst(ev,..); if (!nr_events); continue; sa_specific_atomic_processing(sa /* ev.flow_id */);/* seq number update in critical section */ /* move to next stage(ORDERED) */ ev.event_type = RTE_EVENT_TYPE_CPU; ev.sub_event_type = 3; ev.sched_type = RTE_SCHED_TYPE_ORDERED; ev.flow_id = sa; ev.op = RTE_EVENT_OP_FORWARD; ev.queue_id = 3; /* move to stage 3(event queue 3) */ rte_event_enqueue_burst(ev,..); } on_each_cores_linked_to_queue3(stage3) while(1) { /* STAGE 3 processing */ nr_events = rte_event_dequeue_burst(ev,..); if (!nr_events); continue; sa_specific_ordered_processing(sa /*ev.flow_id */);/* packets encryption in parallel */ /* move to next stage(ATOMIC) */ ev.event_type = RTE_EVENT_TYPE_CPU; ev.sub_event_type = 4; ev.sched_type = RTE_SCHED_TYPE_ATOMIC; output_tx_port_queue = find_output_tx_queue_and_tx_port(ev.mbuff); ev.flow_id = output_tx_port_queue; ev.op = RTE_EVENT_OP_FORWARD; ev.queue_id = 4; /* move to stage 4(event queue 4) */ rte_event_enqueue_burst(ev,...); } on_each_cores_linked_to_queue4(stage4) while(1) { /* STAGE 4 processing */ nr_events = rte_event_dequeue_burst(ev,..); if (!nr_events); continue; rte_eth_tx_buffer(); } 2) flow-based event pipelining ============================= - No need to partition queues for different stages - All the cores can operate on all the stages, Thus enables automatic multicore scaling, true dynamic load balancing, - Fairly large number of SA(kind of 2^16 to 2^20) can be processed in parallel Something existing IPSec application has constraints on http://dpdk.org/doc/guides-16.04/sample_app_ug/ipsec_secgw.html on_each_worker_cores() while(1) { rte_event_dequeue_burst(ev,..) if (!nr_events); continue; /* STAGE 1 processing */ if(ev.event_type == RTE_EVENT_TYPE_ETHDEV) { sa = find_it_from_packet(ev.mbuf); /* move to next stage2(ATOMIC) */ ev.event_type = RTE_EVENT_TYPE_CPU; ev.sub_event_type = 2; ev.sched_type = RTE_SCHED_TYPE_ATOMIC; ev.flow_id = sa; ev.op = RTE_EVENT_OP_FORWARD; rte_event_enqueue_burst(ev..); } else if(ev.event_type == RTE_EVENT_TYPE_CPU && ev.sub_event_type == 2) { /* stage 2 */ sa_specific_atomic_processing(sa /* ev.flow_id */);/* seq number update in critical section */ /* move to next stage(ORDERED) */ ev.event_type = RTE_EVENT_TYPE_CPU; ev.sub_event_type = 3; ev.sched_type = RTE_SCHED_TYPE_ORDERED; ev.flow_id = sa; ev.op = RTE_EVENT_OP_FORWARD; rte_event_enqueue_burst(ev,..); } else if(ev.event_type == RTE_EVENT_TYPE_CPU && ev.sub_event_type == 3) { /* stage 3 */ sa_specific_ordered_processing(sa /* ev.flow_id */);/* like encrypting packets in parallel */ /* move to next stage(ATOMIC) */ ev.event_type = RTE_EVENT_TYPE_CPU; ev.sub_event_type = 4; ev.sched_type = RTE_SCHED_TYPE_ATOMIC; output_tx_port_queue = find_output_tx_queue_and_tx_port(ev.mbuff); ev.flow_id = output_tx_port_queue; ev.op = RTE_EVENT_OP_FORWARD; rte_event_enqueue_burst(ev,..); } else if(ev.event_type == RTE_EVENT_TYPE_CPU && ev.sub_event_type == 4) { /* stage 4 */ rte_eth_tx_buffer(); } } /Jerin Cavium