From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 52576A04B3; Mon, 16 Dec 2019 09:39:20 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 6964E1C195; Mon, 16 Dec 2019 09:39:19 +0100 (CET) Received: from dispatch1-us1.ppe-hosted.com (dispatch1-us1.ppe-hosted.com [67.231.154.164]) by dpdk.org (Postfix) with ESMTP id 425851C191 for ; Mon, 16 Dec 2019 09:39:18 +0100 (CET) X-Virus-Scanned: Proofpoint Essentials engine Received: from webmail.solarflare.com (uk.solarflare.com [193.34.186.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mx1-us1.ppe-hosted.com (PPE Hosted ESMTP Server) with ESMTPS id C4B7EB00056; Mon, 16 Dec 2019 08:39:16 +0000 (UTC) Received: from [192.168.38.17] (10.17.10.39) by ukex01.SolarFlarecom.com (10.17.10.4) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Mon, 16 Dec 2019 08:39:09 +0000 To: Ferruh Yigit , Shahaf Shuler , Olivier Matz , "dev@dpdk.org" , "Ananyev, Konstantin" , "Thomas Monjalon" , Bruce Richardson CC: Matan Azrad , Jerin Jacob Kollanukkaran References: <20180123135308.tr7nmuqsdeogm7bl@glumotte.dev.6wind.com> <65f5f247-15e7-ac0a-183e-8a66193f426f@intel.com> From: Andrew Rybchenko Autocrypt: addr=arybchenko@solarflare.com; keydata= mQINBF2681gBEACbdTxu8eLL3UX2oAelsnK9GkeaJeUYSOHPJQpV7RL/iaIskqTwBRnhjXt7 j9UEwGA+omnOmqQMpeQTb/F9Ma2dYE+Hw4/t/1KVjxr3ehFaASvwR4fWJfO4e2l/Rk4rG6Yi 5r6CWU2y8su2654Fr8KFc+cMGOAgKoZTZHZsRy5lHpMlemeF+VZkv8L5sYJWPnsypgqlCG3h v6lbtfZs+QqYbFH6bqoZwBAl5irmxywGR7ZJr1GLUZZ1lfdazSY8r6Vz0/Ip/KVxGu2uxo81 QCsAj0ZsQtwji9Sds/prTiPrIjx8Fc/tfbnAuVuPcnPbczwCJACzQr4q26XATL3kVuZhSBWh 4XfO/EAUuEq5AemUG5DDTM87g7Lp4eT9gMZB6P+rJwWPNWTiV3L7Cn+fO+l9mTPnOqdzBgDe OaulKiNSft1o0DY4bGzOmM2ad2cZt0jfnbMPMTE9zsr6+RFa+M8Ct20o6U1MUE4vP6veErMK of4kZ8PdoMM+Sq1hxMPNtlcVBSP9xMmdSZPlfDYI5VWosOceEcz7XZdjBJKdwKuz70V7eac4 ITSxgNFCTbeJ03zL2MR5s0IvD9ghISAwZ6ieCjU5UATn5+63qpD0nVNLsAdb/UpfvQcKAmvj 0fKlxu/PMVkjBa7/4cfNogYOhWDKUO+1pMaFwvb6/XTo6uMpfQARAQABtCxBbmRyZXcgUnli Y2hlbmtvIDxhcnliY2hlbmtvQHNvbGFyZmxhcmUuY29tPokCVAQTAQoAPhYhBP6NPgcKRj/Y X0yXQahue0sAy4m+BQJduvNYAhsDBQkB4TOABQsJCAcDBRUKCQgLBRYCAwEAAh4BAheAAAoJ EKhue0sAy4m+t3gP/j1MNc63CEozZo1IZ2UpVPAVWTYbLdPjIRdFqhlwvZYIgGIgIBk3ezKL K0/oc4ZeIwL6wQ5+V24ahuXvvcxLlKxfbJ6lo2iQGC7GLGhsDG9Y2k6sW13/sTJB/XuR2yov k5FtIgJ+aHa1PDZnepnGGOt9ka9n/Jzrc9WKYapOIIyLRe9U26ikoVgyqsD37PVeq5tLWHHA NGTUKupe9G6DFWidxx0KzyMoWDTbW2AWYcEmV2eQsgRT094AZwLFN5ErfefYzsGdO8TAUU9X YTiQN2MvP1pBxY/r0/5UfwV4UKBcR0S3ZvzyvrPoYER2Kxdf/qurx0Mn7StiCQ/JlNZb/GWQ TQ7huduuZHNQKWm7ufbqvKSfbPYvfl3akj7Wl8/zXhYdLqb5mmK45HXrgYGEqPN53OnK2Ngx IgYKEWr05KNv09097jLT5ONgYvszflqlLIzC4dV245g7ucuf9fYmsvmM1p/gFnOJBJL18YE5 P1fuGYNfLP+qp4WMiDqXlzaJfB4JcinyU49BXUj3Utd6f6sNBsO8YWcLbKBV9WmA324S3+wj f4NPRp3A5E+6OmTVMLWire2ZvnYp3YvifUj1r8lhoZ2B2vKuWwiTlHOKYBEjnOQJQnqYZEF0 JQQ1xzVDBQKE01BPlA3vy6BGWe6I4psBVqMOB9lAev/H+xa4u6Z3uQINBF269JsBEAC2KB3W 8JES/fh74avN7LOSdK4QA7gFIUQ4egVL81KnxquLzzilABuOhmZf3Rq6rMHSM8xmUAWa7Dkt YtzXStjEBI/uF0mAR3mMz1RcL2Wp+WD/15HjVpA7hPjXSEsWY0K2ymPerK4yrLcfFTHdMonY JfuACCC9NtOZxrWHOJoUS+RT7AWk80q/6D2iwQ47/2dBTznVG+gSeHSes9l91TB09w6f9JX/ sT+Ud0NQfm7HJ7t2pmGI9O6Po/NLZsDogmnIpJp/WwYOZN9JK7u2FyX2UyRzR8jK42aJkRsh DXs16Cc2/eYGakjrdO3x9a+RoxN7EuFtYhGR1PzMXdUiB5i+FyddYXkYUyO43QE/3VPA5l1v TUOagzZq6aONsdNonGJkV3TIG3JmUNtM+D/+r6QKzmgoJ8w576JxEZI09I/ZFN+g7BnUmlMx 6Z3IUOXVX/SWfGFga0YajwajHz03IBhChEbYbbqndVhmshu2GFURxrfUPYWdDXEqkh+08a5U Didia9jm2Opv4oE1e1TXAePyYJl/Zyps4Cv00GObAxibvMBQCUZQ+IBnNldRBOwXXRQV2xpx P+9iO1VYA/QXn0KqRK+SH1JGRXbJYi42YFaW1gE0EU0fiR2Wb9pK+doNEjjOhlzUGuvOEAUS +4m0m3dlfEvpCV9GMr7ERRpZzh9QkQARAQABiQI8BBgBCgAmFiEE/o0+BwpGP9hfTJdBqG57 SwDLib4FAl269JsCGwwFCQlmAYAACgkQqG57SwDLib7x6g//e+eCtNnJz7qFGbjWRJYNLCe5 gQwkhdyEGk4omr3VmjGj3z9kNFy/muh4pmHUngSAnnpwZggx14N4hhKf9y8G4Dwvsqa6b1zB Jq/c4t/SBDtGW4M/E331N04PaQZpcrbTfp1KqHNknk2N7yOk4CcoLVuIZmA5tPguASV8aAfz ZwhWAwn6vUEw9552eXEAnGFGDTCbyryNwzB5jtVQOEEDjTxcCkpcXMB45Tb1QUslRTu/sBAe HhPCQSUcJHR+KOq+P6yKICGAr291PZd6Qc7C3UyE+A3pY/UfdEVWj0STBWx1qvYLaHLrI4O9 KXDgh7luLjZZafcueCaPYmNo4V2lmNb3+7S4TvqhoZS+wN+9ldRQ4gH3wmRZybN6Y/ZCqxol RaZpE3AqdWsGvIgAkD0FpmtZNii9s2pnrhw0K6S4t4tYgXGTossxNSJUltfFQZdXM1xkZhtv dBZuUEectbZWuviGvQXahOMuH2pM64mx2hpdZzPcI2beeJNHkAsGT2KcaMETgvtHUBFRlLVB YxsUYz3UZmi2JSua4tbcGd6iWVN90eb8CxszYtivfpz6o2nPSjNwg0NaVGSHXjAK0tdByZ9t SkwjC3tEPljVycRSDpbauogOiAkvjENfaPd/H26V5hY822kaclaKDAW6ZG9UKiMijcAgb9u5 CJoOyqE8aGS5Ag0EXbr1RwEQAMXZHbafqmZiu6Kudp+Filgdkj2/XJva5Elv3fLfpXvhVt0Y if5Rzds3RpffoLQZk9nPwK8TbZFqNXPu7HSgg9AY7UdCM94WRFTkUCGKzbgiqGdXZ7Vyc8cy teGW+BcdfQycDvjfy50T3fO4kJNVp2LDNdknPaZVe8HJ80Od63+9ksB6Ni+EijMkh6Uk3ulB CSLnT4iFV57KgU2IsxOQVLnm+0bcsWMcCnGfphkY0yKP+aJ6MfmZkEeaDa7kf24N14ktg50m vOGDitcxA/+XXQXOsOIDJx1VeidxYsQ2FfsKu1G8+G6ejuaLf4rV5MI/+B/tfLbbOdikM5PF pxZVgTir9q13qHumMxdme7w5c7hybW412yWAe9TsrlXktFmFjRSFzAAxQhQSQxArS6db4oBk yeYJ59mW52i4occkimPWSm/raSgdSM+0P6zdWUlxxj+r1qiLgCYvruzLNtp5Nts5tR/HRQjE /ohQYaWDSVJEsc/4eGmgwzHzmvHtXeKkasn01381A1Lv3xwtpnfwERMAhxBZ8EGKEkc5gNdk vIPhknnGgPXqKmE1aWu8LcHiY+RHAF8gYPCDMuwyzBYnbiosKcicuIUp0Fj8XIaPao6F+WTi In4UOrqrYhsaCUvhVjsTBbNphGih9xbFJ8E+lkTLL8P3umtTcMPnpsB4xqcDABEBAAGJBHIE GAEKACYWIQT+jT4HCkY/2F9Ml0GobntLAMuJvgUCXbr1RwIbAgUJCWYBgAJACRCobntLAMuJ vsF0IAQZAQoAHRYhBNTYjdjWgdaEN5MrAN+9UR5r/4d3BQJduvVHAAoJEN+9UR5r/4d3EiQP /3lyby6v49HTU94Q2Fn2Xat6uifR7kWE5SO/1pUwYzx6v+z5K2jqPgqUYmuNoejcGl0CTNhg LbsxzUmAuf1OTAdE+ZYvOAjjKQhY4haxHc4enby/ltnHfWJYWJZ9UN5SsIQLvITvYu6rqthO CYjpXJhwkj3ODmC9H1TrvjrBGc6i7CTnR8RCjMEwCs2LI2frHa4R6imViEr9ScMfUnzdABMQ B0T5MOg8NX92/FRjTldU2KovG0ML9mSveSvVHAoEBLy4UIs5nEDdNiO1opJgKb5CXvWQugub 7AR52phNdKVdEB0S4tigJT4NalyTaPiUhFEm+CzZpMQDJ5E+/OowaPRfN4HeJX+c8sB+vUAZ mkAaG75N+IEk5JKFK9Z+bBYgPgaBDFZYdWDB/TMH0ANt+KI5uYg0i12TB4M8pwKG1DEPUmWc F2YpvB3jnbwzsOpSFiJOOlSs6nOB0Sb5GRtPOO3h6XGj+6mzQd6tcL63c9TrrUkjq7LDkxCz SJ2hTYRC8WNX8Uw9skWo5728JNrXdazEYCenUWmYiKLNKLslXCFodUCRDh/sUiyqRwS7PHEA LYC/UIWLMomI0Yvju3KA5v3RQVXhL+Gx2CzSj3GDz9xxGhJB2LfRfjzPbTR/Z27UpjCkd8z0 Ro3Ypmi1FLQwnRgoOKDbetTAIhugEShaLTITzJAP/iRDJCQsrZah5tE8oIl81qKEmBJEGcdt HYikbpQe7ydcXhqTj7+IECa3O7azI5OhCxUH2jNyonJ/phUslHH2G1TTBZK8y4Hrx5RpuRNS esn3P9uKu9DHqBAL7DMsCPwb2p1VNnapD72DBmRhzS/e6zS2R4+r9yNv03Hv7VCxKkmtE63H qpS//qpjfrtsIcHAjnKDaDtL1LYCtHoweI+DOpKKULSAYp/JE6F8LNibPQ0/P3S5ZIJNC4QZ uESjFOalJwFIqGQdkQB7ltRNJENLrHc+2jKGOuyFHm/Sbvp5EMGdaeQ0+u8CY0P+y6oXenwx 7WrJz/GvbNoFhJoJ6RzxCMQrFgxrssVZ7w5HcUj94lbnJ6osdYE/WpSd50B6jet6LKh5revg u9XI9CoqsPQ1V4wKYYdllPuogCye7KNYNKuiiuSNpaF4gHq1ZWGArwZtWHjgc2v3LegOpRQF SwOskMKmWsUyHIRMG1p8RpkBQTqY2rGSeUqPSvaqjT0nq+SUEM6qxEXD/2Wqri/X6bamuPDb S0PkBvFD2+0zr5Bc2YkMGPBYPNGZiTp3UjmZlLfn3TiBKIC92jherY563CULjSsiBEJCOSvv 4VPLn5aAcfbCXJnE3IGCp/hPl50iQqu7BPOYBbWXeb9ptDjGCAThNxSz0WAXkmcjAFE8gdE6 Znk9 Message-ID: Date: Mon, 16 Dec 2019 11:39:05 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 In-Reply-To: <65f5f247-15e7-ac0a-183e-8a66193f426f@intel.com> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [10.17.10.39] X-ClientProxiedBy: ocex03.SolarFlarecom.com (10.20.40.36) To ukex01.SolarFlarecom.com (10.17.10.4) X-TM-AS-Product-Ver: SMEX-12.5.0.1300-8.5.1010-25106.003 X-TM-AS-Result: No-18.285400-8.000000-10 X-TMASE-MatchedRID: 8HTFlOrbAtHmLzc6AOD8DfHkpkyUphL9YhoV2WlukSXfUZT83lbkEAoi fCpSIssp+8k49QHMhKiX9RBfHVAigTs6RIuDSecTnMQdNQ64xff/T7YBBUajEDqm8DqBF6Dn4a8 tjcE0lxwoipFUtVmu9Z09pgURDiYVzroGAhCVDDUtMfCdg6KRDTrZR4dvqaRP33Nl3elSfspAGS nh5QXxvvgmFcBlNYL78hYOgsqT2ysevt/ebtDuDaOknopQLzTuGIMg4+U4kbUZSz1vvG+0mtBOL d4mlBAvUbxopzQbTufX4tZESnn3A+5mL/Sp2zN370a2q2D7J4BUzrW1jLbY0aRea8wUAdQ6Xnmu BT+Y9/LZoYLwcV5H6PRWZfxrpZenmUh0Uc9nt3Ekp7iSXiinhH316REeU5CRj4jFvt2M5IKg0hW zUrznJ4QgVGpvZnGnH5ED9mQDsMh7rjItHwarhYph1hAtvKZNp1Pjcaldww2ZfDRE1uqSgrBZsz Sz1qeiLrAmCaN0WuL/QpXmI/FJnNbCHM2gsS6O5venhychcY1jCt269TGh+E/EoM7ZtRRVYbjY3 v109cHj7ZFQBVaLnKF+b4H8EwufKTk1ptLkcSvB7F9jxUX48i1CkRb13IZunlRCZ/VFW62LVFtk 3Um5KEqna2uFlro8LA+Fru9cSckoT+BOd7erTD2T5GsqAvLYV0QSZ/pNFUGfcVJ8cL8ZO8iq4e+ BcPI5hTn9JwTKjEuXaG8P3vv/xS4YlXCNHMx/cFe8RUQnnIerfCU2+xgE2M8DuAI4aSPIFHYQ/2 BuKxs0nQ9dUhhgoYjS9HlY6vhU7eqO+VpGDY2eAiCmPx4NwLTrdaH1ZWqC1B0Hk1Q1KyLUZxEAl FPo846HM5rqDwqtcGh3g99w1Xdm70bCH3z81ZOyKHnZ/Ifax/FEvUY2q3BtLxmo8MtEZg== X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--18.285400-8.000000 X-TMASE-Version: SMEX-12.5.0.1300-8.5.1010-25106.003 X-MDID: 1576485557-SlUN1yYEvrXq Subject: Re: [dpdk-dev] questions about new offload ethdev api X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 12/10/19 9:07 PM, Ferruh Yigit wrote: > On 1/23/2018 2:34 PM, Shahaf Shuler wrote: >> Tuesday, January 23, 2018 3:53 PM, Olivier Matz: > > <...> > >>> >>> 2/ meaning of rxmode.jumbo_frame, rxmode.enable_scatter, >>> rxmode.max_rx_pkt_len >>> >>> While it's not related to the new API, it is probably a good opportunity >>> to clarify the meaning of these flags. I'm not able to find a good >>> documentation about them. >>> >>> Here is my understanding, the configuration only depends on: - the maximum >>> rx frame length - the amount of data available in a mbuf (minus headroom) >>> >>> Flags to set in rxmode (example): >>> +---------------+----------------+----------------+-----------------+ | >>> |mbuf_data_len=1K|mbuf_data_len=2K|mbuf_data_len=16K| >>> +---------------+----------------+----------------+-----------------+ >>> |max_rx_len=1500|enable_scatter | | | >>> +---------------+----------------+----------------+-----------------+ >>> |max_rx_len=9000|enable_scatter, |enable_scatter, |jumbo_frame | | >>> |jumbo_frame |jumbo_frame | | >>> +---------------+----------------+----------------+-----------------+ >>> >>> If this table is correct, the flag jumbo_frame would be equivalent to check >>> if max_rx_pkt_len is above a threshold. >>> >>> And enable_scatter could be deduced from the mbuf size of the given rxq >>> (which is a bit harder but maybe doable). >> >> I glad you raised this subject. We had a lot of discussion on it internally >> in Mellanox. >> >> I fully agree. All application needs is to specify the maximum packet size it >> wants to receive. >> >> I think also the lack of documentation is causing PMDs to use those flags >> wrongly. For example - some PMDs set the jumbo_frame flag internally without >> it being set by the application. >> >> I would like to add one more item : MTU. What is the relation (if any) >> between setting MTU and the max_rx_len ? I know MTU stands for Max Transmit >> Unit, however at least in Linux it is the same for the Send and the receive. >> >> > > (Resurrecting the thread after two years, I will reply again with latest > understanding.) > > Thanks Olivier for above summary and table, and unfortunately usage still not > consistent between PMDs. According my understanding: > > 'max_rx_pkt_len' is user configuration value, to limit the size packet that is > shared with host, but this doesn't limit the size of packet that NIC receives. Also comment in lib/librte_ethdev/rte_ethdev.h says that the rxmode field is used if (and I think only if) JUMBO_FRAME is enabled. So, if user wants to set it on device configure stage, device *must* support JUMBO_FRAME offload which mean that driver code handles rxmode.max_rx_pkt_len and either accept it and configures HW appropriately or return an error if specified value is wrong. Basically it is written in jumbo frame feature definition in features.rst. User has max_rx_pktlen in dev_info to find out maximum supported value for max_rx_pkt_len. > Like if the mbuf size of the mempool used by a queue is 1024 bytes, we don't > want packets bigger than buffer size, but if NIC supports it is possible receive > 6000 bytes packet and split data into multiple buffers, and we can use multi > segment packets to represent it. > So what we need is NIC ability to limit the size of data to share to host and > scattered Rx support (device + driver). It requires RX_SCATTER offload enabled and it must be controlled by the user only (not PMD) since it basically mean if the application is ready to handle multi-segment packets (have code which takes a look at the number of segments and next pointers etc). Moreover, application may disable MULTI_SEG Tx offload (and drivers may ignore number of segments and next pointer as well). > But MTU limits the size of the packet that NIC receives. Personally I've never treated it this way. For me the only difference between max_rx_pkt_len and MTU is: - max_rx_pkt_len is entire packet with all L2 headers and even FCS (basically everything which could be provided to application in mbuf) - MTU does not cover L2 (and VLANs, I'm not sure about MPLS) > Assuming above are correct J, > > Using mbuf data size as 'max_rx_pkt_len' without asking from user is an option, > but perhaps user has different reason to limit packet size, so I think better to > keep as different config option. > > I think PMD itself enabling "jumbo frame" offload is not too bad, and > acceptable, since providing a large MTU already implies it. Yes > But not sure about PMD enabling scattered Rx, application may want to force to > receive single segment mbufs, for that case PMD enabling this config on its own > looks like a problem. Yes > But user really needs this when a packet doesn't fit to the mbuf, so providing a > MTU larger than 'max_rx_pkt_len' _may_ imply enabling scattered Rx, I assume > this is the logic in some PMDs which looks acceptable. I don't think so. IMO auto enabling Rx scatter from PMD is a breakage of a contract between application and driver. As stated above the application may be simply not ready to handle multi-segment packets correctly. I think that providing an MTU larger than 'max_rx_pkt_len' is simply a change of max_rx_pkt_len = (MTU plus space for L2+). > And PMD behavior should be according for mentioned configs: > > 1) Don't change user provided 'max_rx_pkt_len' value I have no strong opinion. However, it is important to clarify which understanding of max_rx_pkt_len vs MTU is the right one. > 2) If jumbo frame is not enabled, don't limit the size of packets to the host (I > think this is based on assumption that mbuf size always will be > 1514) I think that JUMBO_FRAME is not relevant here. It is just a promise to take a look at max_rx_pkt_len on configure or start stage. > 3) When user request to set the MTU bigger than ETH_MAX, PMD enable jumbo frame > support (if it is not enabled by user already and supported by HW). If HW > doesn't support if of course it should fail. I'm not sure which ETH_MAX is mentioned above. #define ETH_MAX_MTU 0xFFFFU /* 65535, same as IP_MAX_MTU */ or do you mean #define ETH_FRAME_LEN 1514 /* Max. octets in frame sans FCS */ or even #define ETH_DATA_LEN 1500 /* Max. octets in payload */ We should be careful when we talk about Ethernet lengths and MTU. > 4) When user request to set MTU bigger than 'max_rx_pkt_len' I think the second parameter to consider here is not max_rx_pkt_len, but amount of space for data in single mbuf (for all Rx queues). > 4a) if "scattered Rx" is enabled, configure the MTU and limit packet size to > host to 'max_rx_pkt_len' Yes and no. IMO configure the MTU and bump max_rx_pkt_len. > 4b) if "scattered Rx" is not enabled but HW supports it, enable "scattered Rx" > by PMD, configure the MTU and limit packet size to host to 'max_rx_pkt_len' No, I think it is wrong to enable Rx scatter from PMD. > 4c) if "scattered Rx" is not enabled and not supported by HW, fail MTU set. Yes, regardless support in HW. > 4d) if HW doesn't support to limit the packet size to host, but requested MTU > bigger than 'max_rx_pkt_len' it should fail. I would rephrase it as impossibility to disable Rx scatter. If so, it must be driver responsibility to drop scattered packets if Rx scatter offload is not enabled. > Btw, I am aware of that some PMDs have a larger MTU by default and can't limit > the packet size to host to 'max_rx_pkt_len' value, I don't know what to do in > that case, fail in configure? Or at least be sure configured mempool's mbuf size > is big enough? See above. Thanks for reminder about the topic.