From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 85C2843A18; Wed, 31 Jan 2024 08:43:09 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 66E9B402D7; Wed, 31 Jan 2024 08:43:09 +0100 (CET) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2079.outbound.protection.outlook.com [40.107.94.79]) by mails.dpdk.org (Postfix) with ESMTP id 9646D402A1; Wed, 31 Jan 2024 08:43:07 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GFijWpyOwJypkDh4nTZfplOp8tGyjk7jJnxpV47dUryolCnf3wWMXM+2aqVrHvzmTHyn1pZ0Ur0Sp87gxpAUieG7LE3i3S2nSR0Y474Ia4TWGOdvN72kG4ZIw2JqNoZ9lHYLxvJIVXji7jRmIfXI3dZgsoHjwgcgD/nmEp5x73c40bAvdmuMGqDsRsJ2v20oqunRIrNxfZ4oWM3f4KTQs5BnziCGSsDabEnY0oMJG29GZNA1HorXLuRNWyz3LVnxU8C/kZ/Zh2+y1LScAj3D58SjivdgcHZf99U0LW+Gukklkk8qha/2JceaKUXOW7E1N9pR2ZHzCl9Kl+2idIWO1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vvT9UaNpFvBy1IG60ubWpgHv+K9Vg+DstxVP5pZeQwc=; b=PxCQprcevcU0ctNOZ2BZwRqXqw8UG9oQQ+h1JS4tcHECvDIiBZwSFSJXgnFIrkwkZ4UOw98MJbavv+4a+dp1N8vc+ZvgM4D+fTTaJ3V0M+Xl9LrjXQeUR3eROwYtA6rptIUkhOygPkc66PF8BNdrMcIKG4ZPzcPkF0UCCmbmI7FH+EJfiemSF5XdolqgTOwwu4yRLqhpTTkRi0J4sKjOuA7cEkMpTz9JYFHAJrz/9YweMNVJczKPc7ba1zJaBeLeQeSR2Vpq0o/VI7fv6QVYVXzWAEvVvMwWkUIRR1udo1OtBYG+kFYZV/5bYVbc2p3aw6BcZux7sbJYRFhjXMdMfA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vvT9UaNpFvBy1IG60ubWpgHv+K9Vg+DstxVP5pZeQwc=; b=WJJMY2cZ3GGvRuGYYB+id1vO5Pzz8fjl8PLq6GIyOD2J7+1yTqmu7KnbJXmZt/iZBiH2KofMFKoVlELtcP97aKBivAtOjuypuzHC1m3P3V6beMyJGpL1cIIY2L+9D9tmcSm1CadW73oQGUL4B2XTwaR106yznrphZyOAtWqXvhoSehyJRL+2T5U7fdBJOJxUlTNSkfsXzFp4dCPGlpEUYEe4eYYd0A7B2UXI05TDUc82++9pHq8DP0k4EtiUz297rKGkk17BUcJTS0w+b4NGxgryBxO4uafQXeGW4wlNLVeli8PEeohZwitkD33/o7Kwzb7DJSxIMJZuh/e+u07acg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from IA1PR12MB6332.namprd12.prod.outlook.com (2603:10b6:208:3e2::13) by BN9PR12MB5227.namprd12.prod.outlook.com (2603:10b6:408:100::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.32; Wed, 31 Jan 2024 07:43:03 +0000 Received: from IA1PR12MB6332.namprd12.prod.outlook.com ([fe80::5610:835d:9e24:6cfd]) by IA1PR12MB6332.namprd12.prod.outlook.com ([fe80::5610:835d:9e24:6cfd%4]) with mapi id 15.20.7249.017; Wed, 31 Jan 2024 07:43:03 +0000 Date: Wed, 31 Jan 2024 09:42:58 +0200 (IST) From: "Etelson, Gregory" To: Patrick Robb cc: Gregory Etelson , Jeremy Spewock , "NBU-Contact-Thomas Monjalon (EXTERNAL)" , Honnappa Nagarahalli , =?ISO-8859-15?Q?Juraj_Linke=A8?= , Paul Szczepanek , Yoan Picchi , Luca Vizzarro , "ci@dpdk.org" , "dev@dpdk.org" , nd , Maayan Kashani , Asaf Penso Subject: Re: DTS testpmd and SCAPY integration In-Reply-To: Message-ID: References: <2a287ee7-cda4-f2ab-a4e6-a47021f8573f@nvidia.com> <2127794.bB369e8A3T@thomas> <6608561.G0QQBjFxQf@thomas> Content-Type: multipart/mixed; boundary="8323329-690611550-1706686983=:95184" X-ClientProxiedBy: LO4P123CA0310.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:197::9) To IA1PR12MB6332.namprd12.prod.outlook.com (2603:10b6:208:3e2::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: IA1PR12MB6332:EE_|BN9PR12MB5227:EE_ X-MS-Office365-Filtering-Correlation-Id: b698b618-9df6-4786-b573-08dc223041a0 X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JLnjLfBq1LWOh8n1k+jTAlnzJFKs5UZTuPoTWgyW20lFhoIjGAbz4UF7BuWcxoUTaPYl7fTYgMHZS0URiNHmkuDrAMuXbBnhk2UC6/NSxsDzDHgaMYHuKcxAt/RJu6dJuX3sngw1nmjE/B5NJXSKeblKrdJ0kpd57oMbdbUO/cnMkDLaV6kCAeaemwduMJgEYDDQMyoCkP+NIJQoqtrYoIxK9TNFV8AFJVDE7yX2XZP6zyhasfrROvTjrjXa9p3vk78ic91H5xy1hbKgeQsXW9wH4QXGYVoYkgu0oWI4eh0LM8dlvr4PK4kQoe1oq32YbezxDa9sam9nfd8FcU1OJb6OVRLp5tQpexN8w/imP4OqeA4fqiZv9d1LElYGYp/8tHiJyQf47jZ165sWnUWOxXd/3AamqpqsoJuwVr3aJAnpxN/yscNBp1/HELKiOpTJrFjBRCi23p816jST+w/45Q8kbxgbw2FVwt2ylDRUEC49+GvCJkQhKZfx+N9xgI8TdLshpOgihjtX82k8UfVsvU3McCtNQCeFXXtNtcbLl/Nz1Hr3K0IaFlVLJcK0PPL5elj+3ZoSGyCauLgjPwOLLVhGHOziGN//DRdq9qf13uYrw3kQHvTT4sWb6A0YeqDPdgyouTp44PEOqW5gdNSwt0GupM3xXw1opRJFp99O00W4mnxTCzG6QvJ1nQSGF7TsrUl9r/4OSFTaiY5pPPhxiw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:IA1PR12MB6332.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(39860400002)(376002)(396003)(346002)(366004)(136003)(230173577357003)(230273577357003)(230922051799003)(1800799012)(186009)(64100799003)(451199024)(54906003)(8936002)(8676002)(4326008)(2906002)(5660300002)(7416002)(31696002)(86362001)(66946007)(66556008)(66476007)(6916009)(316002)(36756003)(6486002)(38100700002)(6506007)(33964004)(6512007)(478600001)(6666004)(83380400001)(107886003)(2616005)(41300700001)(31686004)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?SmFUZVlsL2dtQ2VXMEJaSnNUbUZSdEZic3NXaVRXSFQrSlI5K2tXamVwZDV1?= =?utf-8?B?Mk93WnhOWm1FTUhUd2FTSEVNZWx0TmRicVJKTEwzVXFjUTdUZTNKaGh1R1Bo?= =?utf-8?B?Vy9YaEY3SDVRWGd3U3FMNmE3TGl6bVVkNGlrYkxEekRSd2hzK0d2TzBudjdX?= =?utf-8?B?S2tOZDdJWjN6MDRMTkRMQ3lEbGxNQ3dodkQwQTVBZkxGakZOVjgyQ1lhby9v?= =?utf-8?B?Ky9LTDJrSFRsT0ZEMGNTY1VvcEkzZitQWE1sZExjdmJyTURiUTRFK0dvSThq?= =?utf-8?B?OERVSGtrRkhiYzRIUUhJYURlQlpNbUJZOXVwcGk2STZmOEl5cDZrNDFWQ2Mr?= =?utf-8?B?VGo3UUR2ME9INVY1QmZQeHIvQXZKYVFhL2RxK3RmTE0xd1MrUEFXSHR4ZVIx?= =?utf-8?B?SnptQTJlcTdqWkVndzluWS9DVFNyUG90UDFsT2h4bEE0dURGNzdLMzZ5TTh2?= =?utf-8?B?VytHYUd1M2ZtRXBxZ3RtY2pTdys2VktNZ2E3S2xVdzdXKzB6dnlid0JXSWVm?= =?utf-8?B?Z0FDbG4vblJHYWtZbmpBUnhxTWJoZ1Zzd0V1eHJod0ZlKzNXZ09QWXRVRGM2?= =?utf-8?B?MENBcWo1M2cxQ3hTZFpwSjRpNHgrSHNxWXhKWDN5U2M3clF4amVnNnpFbTE0?= =?utf-8?B?WThDS2pUNUFzcjRweG16YVJZZ2h4SnQ4Y2hUZVJRYVA5T3JUcFFYUGZJbGlS?= =?utf-8?B?QWE4RlRBaitkZjZRQXp2cStmVlJlbVdJMFQrMndVWm9PMk5WS0psNXVRdWh0?= =?utf-8?B?NTFxQzZ6N0pMS3Y1K0UzVTRhTnI1YndaZ1lDcURnc0kvY21GNHFJdnlLYmdh?= =?utf-8?B?RVBRSjlFM2JhN21UUXcwcUJoY1dqZTA0dnAySmI0aTVPczBveUlrblNDNXlM?= =?utf-8?B?RDJiSU9oOHp2eTRCekhybUozK3FuRS84UlZxVHhXUklxb29KUTc0bGMyK3Rh?= =?utf-8?B?a0FNUU95RlJ4em02d1c4YVRVank4ait0Vys3TGIzMmFGYWt1bEFmenNxWjlt?= =?utf-8?B?c2JoZG9wVEh0R2I2Qmc2Y1Z5S2hQem41dmVSSnBMaldyRmY5VmQwaE1QTDVw?= =?utf-8?B?SjhuRUpKdkNqVHZtOFJtUkE0ZVNtYWJDME5KZ1dSQlRuckViWFNBVVY3WnFY?= =?utf-8?B?VWVOVXIxMFBJcXppUkk3TVRNWEhqN2t3MUp0Y1VpTEpkd01MZ2l5WlVzK0xI?= =?utf-8?B?Mm1pUUlsZzBWeGd3Y2NzclRmTW5WSDJaWi85dndwZ2xtTTFHT0xGcEI2R2Ex?= =?utf-8?B?Z1NoQjVEclVkRDZQaC9IcGo5eDUzMXNVZ1I1a2ZNSnY2MXhPTHMzZDA1SVd3?= =?utf-8?B?cVBnQUhEVWUwSjdqQXJId2FiSUlaaDN3dFFkYVo4QUdzY0FCcXZ4aDEvbjB4?= =?utf-8?B?MzByNDdVVEwzZ1RkQnZjSlBKbHJ2eG9xU1VBYmpMcTNvUmE2bW1pcmU3TXlE?= =?utf-8?B?dDhiOXpxRW9vOTNKeFJVZU1LaTU3Q0h5ZmNUaXNnMmczdjZ0cjFtRlkvVUdq?= =?utf-8?B?Vm4yUGpQR3hQYklmM1B5NGc3TTdRVVFiN29JQlY1TzFjWWdhdGNLcm9MZHZq?= =?utf-8?B?Tkx3eXZwa2kyT3JYaW1vZCtTbnozUmI5czJFTGQ0dUduSCtFTEFwZmF3alV6?= =?utf-8?B?ZDhhMHo5SmJYY1N2RWZ2QVI4SExGY3pkUXZaaUM1ZEw3SDR5NHlacHFkUHBU?= =?utf-8?B?UUVPY3I4RXkzdkU0bWpLYzYwZEFOQnZ6VXo5aFVxVEZyb3RqcHZYNDBvcFFI?= =?utf-8?B?bVhXRTV6R0Rrb3N5S3BkL3dRZkR1OXpsa0JUU3ZydGRZRkRmWE1xQ3IrZXRx?= =?utf-8?B?blhRaE9Ic3owMnZUeW1ZU0R4NTd2RDhvZ1BkVXYrQ2xtV1NMUjExTW94Q3J4?= =?utf-8?B?bE5yMTNTTjhhVi95Zko4VlJhT09XQmg1K2ZBVG9ZSnhrNzN6QWhFTnJSeTJH?= =?utf-8?B?WWNBWFcwS2llN1VVUS9VQU1xcGFodUp6eldSQnQ3bFZzN3Y0WnFlK0dtREw0?= =?utf-8?B?U1YvRFFXVEVsb2NScElYSjIwV2tZOE5TWlc0UFEzYzJxdGkzaWFIaGtVZ0hV?= =?utf-8?B?V0I0a2xlcXh4YUtGUm5nY1VXQmF5WC9TZWpRU2ZudCt1RVZhOEtmRlBPTWVt?= =?utf-8?B?MlBWM3hvZkVUU3NKR1JDdXJFN0VCTkEvZWRsak5VNGtIdFhHdTIrd2VJTWh0?= =?utf-8?Q?UhdQuDZncn44udAwECHHyfVL+MNjT6wC0TQk9hyMybBB?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: b698b618-9df6-4786-b573-08dc223041a0 X-MS-Exchange-CrossTenant-AuthSource: IA1PR12MB6332.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jan 2024 07:43:03.5761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xFQwcghN3JNvAhTKWbsRW2vb3AzyieoLLZZKK3eYmIgqLQv+B1Qq5vUHaQV7GYKAbGsHO6AkxTwBE0aFcM+29w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN9PR12MB5227 X-BeenThere: ci@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK CI discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ci-bounces@dpdk.org --8323329-690611550-1706686983=:95184 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT Hello Patrick, > External email: Use caution opening links or attachments > Thank you for sharing Gregory. I did not get an opportunity to look through the code today, but I did run > through the presentation. A few points I noted: > 1. The presentation shows an example testpmd testcase for creating a flow rule, and then shows a > validation step in which standard out is compared against the expected string ("flow rule x created") and > we can conclude whether we are able to create flow rules. Are you also sending packets according to the > flow rules and validating that what is sent/received corresponds to the expected behavior of the flow > rules? When I look at the old DTS framework, and an example flow rules testsuite > (https://doc.dpdk.org/dts/test_plans/rte_flow_test_plan.html) which we want feature parity with, I think > that validation for this testing framework needs to primarily rely on comparing packets sent and packets > received. The unit test infrastructure validates flow rule creation and a result produced by that flow. Flow result is triggered by a packet. However, flow result validation does not always can be done by testing a packet. Unit test implements 2 flow validation methods. The first validation method tests testpmd output triggered by a test packet. Example: use the MODIFY_FIELD action to copy packet VLAN ID to flow TAG item. Flow tag is internal flow resource. It must be validated in DPDK application. Test creates 2 flow rules: Rule 1: use MODIFY_FILED to copy packet VLAN ID to flow TAG item pattern eth / vlan / end \ actions modify_field op set dst_type tag ... src_type vlan_id ... / end Rule 2: validate the TAG item: pattern tag data is 0x31 ... / end actions mark id 0xaaa / rss / end The test sends a packet with VLAN ID 0x31: / Dot1Q(vlan=0x31) / The test matches tespmd output triggered by the packet for `FDIR matched ID=0xaaa`. The second validation method tests a packet after it was processed by a flow. Unit test operates in a static environment. It does not compare source and target packets. The test "knows" valid target packet configuration. Example: push VLAN header into a packet. There is a single flow rule in that example: pattern eth / end \ actions of_push_vlan ethertype 0x8100 / \ of_set_vlan_vid vlan_vid 3103 .../ port_id id 1 / end There are 2 SCAPY processes in that test: `tg` runs on peer host and sends a source packet. `vm` runs on the same host as testpmd. It validates incoming packet. Phase 0 prepares test packet on the `tg` and starts AsyncSniffer on the `vm`. Phase 1 sends the packet. Phase 2 validates the packet. The test can repeat phases 1 and 2. phase0: vm: | sniff = AsyncSniffer(iface=pf1vf0, filter='udp and src port 1234') tg: | udp_packet = Ether(src='11:22:33:44:55:66', dst='aa:bb:cc:dd:ee:aa')/ IP(src='1.1.1.1', dst='2.2.2.2')/ UDP(sport=1234, dport=5678)/Raw('== TEST ==') phase1: &phase1 vm: sniff.start() tg: sendp(udp_packet, iface=pf1) phase2: &phase2 vm: | cap = sniff.stop() if len(cap[UDP]) > 0: cap[UDP][0][Ether].command() result: vm: vlan=3103 >In any case, there may be some testsuites which can be written which are small >enough in scope > that validating by standard out in this way may be appropriate. I'm not sure but we should keep our > options open.  > > 2. If the implementation overhead is not too significant for the configuration step in the DTS execution a > "--fast" option like you use may be a good improvement for the framework. In your mind, is the main > benefit A. reduced execution time, B. reduced user setup time (don't have to write full config file) or C. > Something else? A user must always provide test configuration. However a host can already have prepared setup before the test execution. In that case a user can skip host setup phase and reduce execution time.   > > Thanks for making this available to use so we can use it as a reference in making DTS better. :)  > > --8323329-690611550-1706686983=:95184--