From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 68A5BA051F; Wed, 10 Jun 2020 17:16:15 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 097721BEDC; Wed, 10 Jun 2020 17:16:15 +0200 (CEST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140049.outbound.protection.outlook.com [40.107.14.49]) by dpdk.org (Postfix) with ESMTP id 4AF641BEB4 for ; Wed, 10 Jun 2020 17:16:14 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FaMAzwW2JPOAN1EblyRWaNA3XV/CQus6CfeWbGJxUEapUj2sbAupxvR7vLp2YNV1u2uj+FVcqeKLLugO1iHE2ieRo4HmMbo6rY0e7mOL2+83zcVYEAMM0U4Xyi+8Sl11bdVii5UM1xxWkZdSxwB3gmWvO45fkA6ZHFG7QyLypDGZA01yqEKic1vGSa7JmlifYXErfbskru1amAyfc1IN8k7bnKQDDgw2AaAevIX8aRszdTOvtR+sdfYdSA2aN5AKCdisJnDO+FiEdz3jbzi6sBxhoKqEOw5asYLE6hEQcefIZPVAebNcarCIaohL3nF8j3iDz03OfAbqsvDZtgzxlQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XgzANuntY6U54RzGwM5BCuo9ju9ocSvWFlcfIRUCJPM=; b=N/dqsV2aNyxzqo5ckBYcLTTyP7cgQCHcHdmTwRkGmYP7ANH2SMHev2NUryI+CRfHsYP7AuhsxadoJTMfzNADMFgbUMGmNsDcZESgyZcxAlaFwyXDx9MSfFcX4xirxeti5WP3FkobMujXROFTlEdcbRP5KGlX7m87CMe92ddu0OubjwCuQeWqVyk9QhWOvMcZ9oBpn5XpFu2TwVHsWyuHeszb0Nkc0lxUDftIRidSgBlay7lNH/iHUQXjWpbaDUoJzN4ULMW8yNZAtz/gx6MhjBFF3TsRutlw6cR8SklmO1MBtch8EORg/+Jn5OH+c3sIRXPyxYnoonEhVq7EL3a/fA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XgzANuntY6U54RzGwM5BCuo9ju9ocSvWFlcfIRUCJPM=; b=iAIBKS7A9vDECKVsD1yWocYWj84d4wotUUXOOcbvSILFWtfkBzPUG/OUDYe+5/YPKglKChHigsNaUTqolPSwW+p7CAR+phLfUiVQhe4o5lidZVsAyL42ZENUCgop7IATxOnQTOf6T3kc5bSTUUVyjD4tSoocresbRg1J2PQWK1c= Received: from AM4PR05MB3265.eurprd05.prod.outlook.com (2603:10a6:205:8::26) by AM4PR05MB3266.eurprd05.prod.outlook.com (2603:10a6:205:c::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3066.23; Wed, 10 Jun 2020 15:16:12 +0000 Received: from AM4PR05MB3265.eurprd05.prod.outlook.com ([fe80::bdf0:88a3:3a39:4be4]) by AM4PR05MB3265.eurprd05.prod.outlook.com ([fe80::bdf0:88a3:3a39:4be4%7]) with mapi id 15.20.3088.018; Wed, 10 Jun 2020 15:16:12 +0000 From: Slava Ovsiienko To: Harman Kalra CC: "dev@dpdk.org" , Thomas Monjalon , Matan Azrad , Raslan Darawsheh , Ori Kam , "olivier.matz@6wind.com" , Shahaf Shuler Thread-Topic: [dpdk-dev] [RFC] mbuf: accurate packet Tx scheduling Thread-Index: AQHWPvG+DxdRrWYbQUGL7krwhFv6yqjR2ZYAgAAWxMA= Date: Wed, 10 Jun 2020 15:16:12 +0000 Message-ID: References: <1591771085-24959-1-git-send-email-viacheslavo@mellanox.com> <20200610133332.GA54613@outlook.office365.com> In-Reply-To: <20200610133332.GA54613@outlook.office365.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: marvell.com; dkim=none (message not signed) header.d=none;marvell.com; dmarc=none action=none header.from=mellanox.com; x-originating-ip: [95.164.10.10] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: ad05cf9b-ccd1-4b2e-535d-08d80d513639 x-ms-traffictypediagnostic: AM4PR05MB3266: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtFwd,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 0430FA5CB7 x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 6VjIwi4eFz7ezV/0DdChyOpbNqVx4djwkmhvGtwo258Hk+toQvBz2uTbESVK9fv++deHO7seK+J9in9bcsmhFjhitnvk4LuLkXOfjeFNXaA9ESiOvrIH+x8xVG7GHyNSMA8QJer614XVLCvcPj5vakeMRh58BtBUHeq/B/zPTyKUzsNWDTg8GTECthkksNv8t07wTQQ2PCMqfq7yx6m/3T9oxbWAMHcUrm3P4UPYODE6HeB9dDw5azGK0XU0+YSNrr7QVhp5QmqCUkGVE1BnvEF9b5eJ7PQTsxca6Du2gUUjC7+PPyQ3y2NKi2ARLoZjPA2Af/T9NtIJ+vYWpYxVa7eRCb+xzykzLt7l2wn7QYd0vxeevZfkLvs37tdf4cE7 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:AM4PR05MB3265.eurprd05.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(4636009)(39860400002)(376002)(346002)(366004)(136003)(396003)(8936002)(66446008)(66946007)(71200400001)(5660300002)(26005)(186003)(2906002)(76116006)(64756008)(54906003)(8676002)(66556008)(66476007)(316002)(107886003)(7696005)(478600001)(9686003)(6916009)(53546011)(86362001)(4326008)(33656002)(6506007)(55016002)(83380400001)(52536014)(60764002); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: REBed3k79qq8N2HxpozlMGUeJNkTXM+IKtespkfai8NhLrMFkV8cai/eWWWx4SO2lhqVtsxQnNjvOXAfkBUC55W3oP8drhFIZE6m3okr6LWV696VxtB6oJ/hXfzlz96HRVR5WkaQ2XPumcWyPA/mRSP2RwlOxjn2wLMEUW8z3KFjNg/vHissuGiT0sk2QRXkTumOW2IgzxpCqQQeREcS1kUY+4Pd0arU+sgp0VRC35DhL2nQPvako/jNltNv5cPobJqHG81VEwY26Kz4Vaq53arvC5Id5qwXJHdwNkvGHx8HXVAc7Fw+COVhHoCE640Cm2HKAiftJFvEoveGahDnHqcZR1o10s12eI+X3j+qoUuBgql++bO8fmd9pzUvsuLZ4XLo3cmfAH14kf42vypgdS65HCBTiSjvJ1Hx2Nc8rJ39ITKIhcPX0IGbZclTgmMgx5TTBsnuxRZv2sGW8lp0RQfB1IXGWbDG1J9vJvBObHg= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: ad05cf9b-ccd1-4b2e-535d-08d80d513639 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Jun 2020 15:16:12.5324 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: NxY2ViUsbiNVvfSWsIKaaCn0Q/9k0h2GS3a/lUXTHwUe5RBn9isB+3VSHthlOc5Jl0P0YDRb1AyT7vo5auEPWcDPSqrWWa7WPb5MdWEwCEU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR05MB3266 Subject: Re: [dpdk-dev] [RFC] mbuf: accurate packet Tx scheduling X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, Harman > -----Original Message----- > From: Harman Kalra > Sent: Wednesday, June 10, 2020 16:34 > To: Slava Ovsiienko > Cc: dev@dpdk.org; Thomas Monjalon ; Matan > Azrad ; Raslan Darawsheh > ; Ori Kam ; > olivier.matz@6wind.com; Shahaf Shuler > Subject: Re: [dpdk-dev] [RFC] mbuf: accurate packet Tx scheduling >=20 > On Wed, Jun 10, 2020 at 06:38:05AM +0000, Viacheslav Ovsiienko wrote: >=20 > Hi Viacheslav, >=20 > I have some queries below: >=20 > > There is the requirement on some networks for precise traffic timing > > management. The ability to send (and, generally speaking, receive) the > > packets at the very precisely specified moment of time provides the > > opportunity to support the connections with Time Division Multiplexing > > using the contemporary general purpose NIC without involving an > > auxiliary hardware. For example, the supporting of O-RAN Fronthaul > > interface is one of the promising features for potentially usage of > > the precise time management for the egress packets. > > > > The main objective of this RFC is to specify the way how applications > > can provide the moment of time at what the packet transmission must be > > started and to describe in preliminary the supporting this feature > > from > > mlx5 PMD side. > > > > The new dynamic timestamp field is proposed, it provides some timing > > information, the units and time references (initial phase) are not > > explicitly defined but are maintained always the same for a given port. > > Some devices allow to query rte_eth_read_clock() that will return the > > current device timestamp. The dynamic timestamp flag tells whether the > > field contains actual timestamp value. For the packets being sent this > > value can be used by PMD to schedule packet sending. > > > > After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation and > > obsoleting, these dynamic flag and field will be used to manage the > > timestamps on receiving datapath as well. > > > > When PMD sees the "rte_dynfield_timestamp" set on the packet being > > sent it tries to synchronize the time of packet appearing on the wire > > with the specified packet timestamp. It the specified one is in the > > past it should be ignored, if one is in the distant future it should > > be capped with some reasonable value (in range of seconds). These > > specific cases ("too late" and "distant future") can be optionally > > reported via device xstats to assist applications to detect the > > time-related problems. > > > > There is no any packet reordering according timestamps is supposed, > > neither within packet burst, nor between packets, it is an entirely > > application responsibility to generate packets and its timestamps in > > desired order. The timestamps can be put only in the first packet in > > the burst providing the entire burst scheduling. >=20 > Since its applicaiton responsibility to care of packet reordering and man= y > other parameters, so why cant application itself take the responsibility = of > packet scheduling, i.e. applicaton can hold for the required time before > calling tx-burst? Why are we even offloading this job to PMD? >=20 - The scheduling is required to be very precise. Within handred(s) of nanos= econds. - It saves CPU cycles. Application just should prepare the packets, put the= desired timestamps and call tx_burst(). "Shut-n-forget" approach.=20 SW approach is potentially possible, application can hold the time and sche= dule packets itself. But... Can we guarantee the stable delay between tx_burst call and data on = the wire? Should we waste CPU cycles to wait the desired moment of time? Can we guara= ntee stable interrupt latency if we choose to schedule on interrupts approach? This RFC splits the responsibility - application should prepare the data an= d specify when it desires to send, the rest is on PMD. =20 > > > > PMD reports the ability to synchronize packet sending on timestamp > > with new offload flag: > > > > This is palliative and is going to be replaced with new eth_dev API > > about reporting/managing the supported dynamic flags and its related > > features. This API would break ABI compatibility and can't be > > introduced at the moment, so is postponed to 20.11. > > > > For testing purposes it is proposed to update testpmd "txonly" > > forwarding mode routine. With this update testpmd application > > generates the packets and sets the dynamic timestamps according to > > specified time pattern if it sees the "rte_dynfield_timestamp" is regis= tered. >=20 > So what I am understanding here is "rte_dynfield_timestamp" will provide > information about three parameters: > - timestamp at which TX should start > - intra packet gap > - intra burst gap. >=20 > If its about "intra packet gap" then PMD can take care, but if it is abou= t intra > burst gap, application can take care of it. Not sure - the intra-burst gap might be pretty small. It is supposed to handle intra-burst in the same way - by specifying the timestamps. Waiting is supposed to be implemented on tx_burst() retry. Prepare the packets with timestamps, tx_burst - if not all packets are sent= - it means queue is waiting for the schedult, retry with the remaining packet= s. As option - we can implement intra-burst wait basing rte_eth_read_clock(). > > The new testpmd command is proposed to configure sending pattern: > > > > set tx_times , > > > > - the delay between the packets within the burst > > specified in the device clock units. The number > > of packets in the burst is defined by txburst parameter > > > > - the delay between the bursts in the device clock units > > > > As the result the bursts of packet will be transmitted with specific > > delays between the packets within the burst and specific delay between > > the bursts. The rte_eth_get_clock is supposed to be engaged to get the >=20 > I think here you mean "rte_eth_read_clock". Yes, exactly. Thank you for the correction. With best regards, Slava >=20 >=20 > Thanks > Harman >=20 > > current device clock value and provide the reference for the timestamps= . > > > > Signed-off-by: Viacheslav Ovsiienko > > --- > > lib/librte_ethdev/rte_ethdev.h | 4 ++++ > > lib/librte_mbuf/rte_mbuf_dyn.h | 16 ++++++++++++++++ > > 2 files changed, 20 insertions(+) > > > > diff --git a/lib/librte_ethdev/rte_ethdev.h > > b/lib/librte_ethdev/rte_ethdev.h index a49242b..6f6454c 100644 > > --- a/lib/librte_ethdev/rte_ethdev.h > > +++ b/lib/librte_ethdev/rte_ethdev.h > > @@ -1178,6 +1178,10 @@ struct rte_eth_conf { > > /** Device supports outer UDP checksum */ #define > > DEV_TX_OFFLOAD_OUTER_UDP_CKSUM 0x00100000 > > > > +/** Device supports send on timestamp */ #define > > +DEV_TX_OFFLOAD_SEND_ON_TIMESTAMP 0x00200000 > > + > > + > > #define RTE_ETH_DEV_CAPA_RUNTIME_RX_QUEUE_SETUP 0x00000001 > /**< > > Device supports Rx queue setup after device started*/ #define > > RTE_ETH_DEV_CAPA_RUNTIME_TX_QUEUE_SETUP 0x00000002 diff --git > > a/lib/librte_mbuf/rte_mbuf_dyn.h b/lib/librte_mbuf/rte_mbuf_dyn.h > > index 96c3631..fb5477c 100644 > > --- a/lib/librte_mbuf/rte_mbuf_dyn.h > > +++ b/lib/librte_mbuf/rte_mbuf_dyn.h > > @@ -250,4 +250,20 @@ int rte_mbuf_dynflag_lookup(const char *name, > > #define RTE_MBUF_DYNFIELD_METADATA_NAME > "rte_flow_dynfield_metadata" > > #define RTE_MBUF_DYNFLAG_METADATA_NAME > "rte_flow_dynflag_metadata" > > > > +/* > > + * The timestamp dynamic field provides some timing information, the > > + * units and time references (initial phase) are not explicitly > > +defined > > + * but are maintained always the same for a given port. Some devices > > +allow > > + * to query rte_eth_read_clock() that will return the current device > > + * timestamp. The dynamic timestamp flag tells whether the field > > +contains > > + * actual timestamp value. For the packets being sent this value can > > +be > > + * used by PMD to schedule packet sending. > > + * > > + * After PKT_RX_TIMESTAMP flag and fixed timestamp field deprecation > > + * and obsoleting, these dynamic flag and field will be used to > > +manage > > + * the timestamps on receiving datapath as well. > > + */ > > +#define RTE_MBUF_DYNFIELD_TIMESTAMP_NAME > "rte_dynfield_timestamp" > > +#define RTE_MBUF_DYNFLAG_TIMESTAMP_NAME > "rte_dynflag_timestamp" > > + > > #endif > > -- > > 1.8.3.1 > >