* [dpdk-users] Mellanox rss hash information
@ 2016-12-28 8:28 Saleh, Omran
2016-12-28 10:59 ` Shahaf Shuler
0 siblings, 1 reply; 6+ messages in thread
From: Saleh, Omran @ 2016-12-28 8:28 UTC (permalink / raw)
To: users
Hi All,
Currently, I am using DPDK 16.11 with Mellanox NIC adapters (mlx5). I encounter a problem that the RSS hash value from mbuf (mbug.hash->rss) is not the same as calculated from rte_softrss(). It is completely different.
I am using the default hash key from mlx5
uint8_t rss_hash_default_key[] = {
0x2c, 0xc6, 0x81, 0xd1,
0x5b, 0xdb, 0xf4, 0xf7,
0xfc, 0xa2, 0x83, 0x19,
0xdb, 0x1a, 0x3e, 0x94,
0x6b, 0x9e, 0x38, 0xd9,
0x2c, 0x9c, 0x03, 0xd1,
0xad, 0x99, 0x44, 0xa7,
0xd9, 0x56, 0x3d, 0x59,
0x06, 0x3c, 0x25, 0xf3,
0xfc, 0x1f, 0xdc, 0x2a,
};
And the following call:
auto hash = rte_softrss(&tuple.v4.src_addr, RTE_THASH_V4_L4_LEN, rss_hash_default_key);
with RSS configurations as the following:
port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
port_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_NONFRAG_IPV4_TCP;
port_conf.rx_adv_conf.rss_conf.rss_key= rss_hash_default_key;
port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
This is working perfectly with ixgbe but Mellanox not. Can you help me to figure out what is the problem? What is the default Toeplitz function for mlx5?!
My NIC adapters are Mellanox Technologies MT27700 Family [ConnectX-4]
With firmware version 12.16.1006 and ofed version 3.3-2.0.0
uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux
Thank you
Best regards,
Omran Saleh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] Mellanox rss hash information
2016-12-28 8:28 [dpdk-users] Mellanox rss hash information Saleh, Omran
@ 2016-12-28 10:59 ` Shahaf Shuler
2016-12-28 14:18 ` Saleh, Omran
0 siblings, 1 reply; 6+ messages in thread
From: Shahaf Shuler @ 2016-12-28 10:59 UTC (permalink / raw)
To: Saleh, Omran, users
Have you tried you use rte_softrss_be function?
I suspect you have an endianness issue here as the mlx5 rss_hash_default_key is already in big-endian format.
/**
* Optimized implementation.
* If you want the calculated hash value matches NIC RSS value
* you have to use special converted key with rte_convert_rss_key() fn.
* @param input_tuple
* Pointer to input tuple
* @param input_len
* Length of input_tuple in 4-bytes chunks
* @param *rss_key
* Pointer to RSS hash key.
* @return
* Calculated hash value.
*/
static inline uint32_t
rte_softrss_be(uint32_t *input_tuple, uint32_t input_len,
const uint8_t *rss_key)
{
...
}
--Shahaf
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Saleh, Omran
Sent: Wednesday, December 28, 2016 10:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Mellanox rss hash information
Hi All,
Currently, I am using DPDK 16.11 with Mellanox NIC adapters (mlx5). I encounter a problem that the RSS hash value from mbuf (mbug.hash->rss) is not the same as calculated from rte_softrss(). It is completely different.
I am using the default hash key from mlx5
uint8_t rss_hash_default_key[] = {
0x2c, 0xc6, 0x81, 0xd1,
0x5b, 0xdb, 0xf4, 0xf7,
0xfc, 0xa2, 0x83, 0x19,
0xdb, 0x1a, 0x3e, 0x94,
0x6b, 0x9e, 0x38, 0xd9,
0x2c, 0x9c, 0x03, 0xd1,
0xad, 0x99, 0x44, 0xa7,
0xd9, 0x56, 0x3d, 0x59,
0x06, 0x3c, 0x25, 0xf3,
0xfc, 0x1f, 0xdc, 0x2a,
};
And the following call:
auto hash = rte_softrss(&tuple.v4.src_addr, RTE_THASH_V4_L4_LEN, rss_hash_default_key); with RSS configurations as the following:
port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS; port_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_NONFRAG_IPV4_TCP; port_conf.rx_adv_conf.rss_conf.rss_key= rss_hash_default_key; port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
This is working perfectly with ixgbe but Mellanox not. Can you help me to figure out what is the problem? What is the default Toeplitz function for mlx5?!
My NIC adapters are Mellanox Technologies MT27700 Family [ConnectX-4] With firmware version 12.16.1006 and ofed version 3.3-2.0.0
uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux
Thank you
Best regards,
Omran Saleh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] Mellanox rss hash information
2016-12-28 10:59 ` Shahaf Shuler
@ 2016-12-28 14:18 ` Saleh, Omran
2016-12-29 8:05 ` Saleh, Omran
2017-01-03 12:30 ` Shahaf Shuler
0 siblings, 2 replies; 6+ messages in thread
From: Saleh, Omran @ 2016-12-28 14:18 UTC (permalink / raw)
To: Shahaf Shuler, users
I am still getting the same problem with the optimized function, the result value from rte_softrss_be is different than mbuf.hash->rss.
any idea about the problem, can be an old kernel produce this problem?
-----Original Message-----
From: Shahaf Shuler [mailto:shahafs@mellanox.com]
Sent: Mittwoch, 28. Dezember 2016 12:00
To: Saleh, Omran <omran.saleh@sap.com>; users@dpdk.org
Subject: RE: Mellanox rss hash information
Have you tried you use rte_softrss_be function?
I suspect you have an endianness issue here as the mlx5 rss_hash_default_key is already in big-endian format.
/**
* Optimized implementation.
* If you want the calculated hash value matches NIC RSS value
* you have to use special converted key with rte_convert_rss_key() fn.
* @param input_tuple
* Pointer to input tuple
* @param input_len
* Length of input_tuple in 4-bytes chunks
* @param *rss_key
* Pointer to RSS hash key.
* @return
* Calculated hash value.
*/
static inline uint32_t
rte_softrss_be(uint32_t *input_tuple, uint32_t input_len,
const uint8_t *rss_key)
{
...
}
--Shahaf
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Saleh, Omran
Sent: Wednesday, December 28, 2016 10:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Mellanox rss hash information
Hi All,
Currently, I am using DPDK 16.11 with Mellanox NIC adapters (mlx5). I encounter a problem that the RSS hash value from mbuf (mbug.hash->rss) is not the same as calculated from rte_softrss(). It is completely different.
I am using the default hash key from mlx5
uint8_t rss_hash_default_key[] = {
0x2c, 0xc6, 0x81, 0xd1,
0x5b, 0xdb, 0xf4, 0xf7,
0xfc, 0xa2, 0x83, 0x19,
0xdb, 0x1a, 0x3e, 0x94,
0x6b, 0x9e, 0x38, 0xd9,
0x2c, 0x9c, 0x03, 0xd1,
0xad, 0x99, 0x44, 0xa7,
0xd9, 0x56, 0x3d, 0x59,
0x06, 0x3c, 0x25, 0xf3,
0xfc, 0x1f, 0xdc, 0x2a,
};
And the following call:
auto hash = rte_softrss(&tuple.v4.src_addr, RTE_THASH_V4_L4_LEN, rss_hash_default_key); with RSS configurations as the following:
port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS; port_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_NONFRAG_IPV4_TCP; port_conf.rx_adv_conf.rss_conf.rss_key= rss_hash_default_key; port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
This is working perfectly with ixgbe but Mellanox not. Can you help me to figure out what is the problem? What is the default Toeplitz function for mlx5?!
My NIC adapters are Mellanox Technologies MT27700 Family [ConnectX-4] With firmware version 12.16.1006 and ofed version 3.3-2.0.0
uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux
Thank you
Best regards,
Omran Saleh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] Mellanox rss hash information
2016-12-28 14:18 ` Saleh, Omran
@ 2016-12-29 8:05 ` Saleh, Omran
2016-12-29 10:38 ` Saleh, Omran
2017-01-03 12:30 ` Shahaf Shuler
1 sibling, 1 reply; 6+ messages in thread
From: Saleh, Omran @ 2016-12-29 8:05 UTC (permalink / raw)
To: Shahaf Shuler, users
Another thing I have noticed that the mbuf.hash->rss is equal in both sides , server and client,
For example,
I am sending a packet which has the following info from a client to a server:
src_ip =172.31.134.146, dest_ip = 172.31.134.150, src_port=49154, dest_port = 9
The rss value (mbuf.hash->rss) for this packet on the server was 4272763546.
Sending a reply from the server to the client
src_ip =172.31.134.150, dest_ip = 172.31.134.146, src_port=9, dest_port = 49154.
Produces the same rss value (mbuf.hash->rss) on the client (4272763546).
On the other hand, rte_softrss_be and rte_softrss functions (both generate the same value) on the client output 3752309715
and on the server 1141129430.
Any idea ? can be a kernel problem (i.e., modules) ?
Can you please provide me how mbuf.hash->rss (i.e., Toeplitz function) is calculated ?
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Saleh, Omran
Sent: Mittwoch, 28. Dezember 2016 15:19
To: Shahaf Shuler <shahafs@mellanox.com>; users@dpdk.org
Subject: Re: [dpdk-users] Mellanox rss hash information
I am still getting the same problem with the optimized function, the result value from rte_softrss_be is different than mbuf.hash->rss.
any idea about the problem, can be an old kernel produce this problem?
-----Original Message-----
From: Shahaf Shuler [mailto:shahafs@mellanox.com]
Sent: Mittwoch, 28. Dezember 2016 12:00
To: Saleh, Omran <omran.saleh@sap.com>; users@dpdk.org
Subject: RE: Mellanox rss hash information
Have you tried you use rte_softrss_be function?
I suspect you have an endianness issue here as the mlx5 rss_hash_default_key is already in big-endian format.
/**
* Optimized implementation.
* If you want the calculated hash value matches NIC RSS value
* you have to use special converted key with rte_convert_rss_key() fn.
* @param input_tuple
* Pointer to input tuple
* @param input_len
* Length of input_tuple in 4-bytes chunks
* @param *rss_key
* Pointer to RSS hash key.
* @return
* Calculated hash value.
*/
static inline uint32_t
rte_softrss_be(uint32_t *input_tuple, uint32_t input_len,
const uint8_t *rss_key)
{
...
}
--Shahaf
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Saleh, Omran
Sent: Wednesday, December 28, 2016 10:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Mellanox rss hash information
Hi All,
Currently, I am using DPDK 16.11 with Mellanox NIC adapters (mlx5). I encounter a problem that the RSS hash value from mbuf (mbug.hash->rss) is not the same as calculated from rte_softrss(). It is completely different.
I am using the default hash key from mlx5
uint8_t rss_hash_default_key[] = {
0x2c, 0xc6, 0x81, 0xd1,
0x5b, 0xdb, 0xf4, 0xf7,
0xfc, 0xa2, 0x83, 0x19,
0xdb, 0x1a, 0x3e, 0x94,
0x6b, 0x9e, 0x38, 0xd9,
0x2c, 0x9c, 0x03, 0xd1,
0xad, 0x99, 0x44, 0xa7,
0xd9, 0x56, 0x3d, 0x59,
0x06, 0x3c, 0x25, 0xf3,
0xfc, 0x1f, 0xdc, 0x2a,
};
And the following call:
auto hash = rte_softrss(&tuple.v4.src_addr, RTE_THASH_V4_L4_LEN, rss_hash_default_key); with RSS configurations as the following:
port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS; port_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_NONFRAG_IPV4_TCP; port_conf.rx_adv_conf.rss_conf.rss_key= rss_hash_default_key; port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
This is working perfectly with ixgbe but Mellanox not. Can you help me to figure out what is the problem? What is the default Toeplitz function for mlx5?!
My NIC adapters are Mellanox Technologies MT27700 Family [ConnectX-4] With firmware version 12.16.1006 and ofed version 3.3-2.0.0
uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux
Thank you
Best regards,
Omran Saleh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] Mellanox rss hash information
2016-12-29 8:05 ` Saleh, Omran
@ 2016-12-29 10:38 ` Saleh, Omran
0 siblings, 0 replies; 6+ messages in thread
From: Saleh, Omran @ 2016-12-29 10:38 UTC (permalink / raw)
To: Shahaf Shuler, users
Something else, I changed the hash key to another key, but, still getting the same rss value for the packets in both sides (4272763546), server and client, is this related to OFED version and firmware version?
Again I am using
Mellanox Technologies MT27700 Family [ConnectX-4]
With firmware version 12.16.1006 and OFED version 3.3-2.0.0
uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux
Best regards,
Omran
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Saleh, Omran
Sent: Donnerstag, 29. Dezember 2016 09:06
To: Shahaf Shuler <shahafs@mellanox.com>; users@dpdk.org
Subject: Re: [dpdk-users] Mellanox rss hash information
Another thing I have noticed that the mbuf.hash->rss is equal in both sides , server and client, For example, I am sending a packet which has the following info from a client to a server:
src_ip =172.31.134.146, dest_ip = 172.31.134.150, src_port=49154, dest_port = 9 The rss value (mbuf.hash->rss) for this packet on the server was 4272763546.
Sending a reply from the server to the client src_ip =172.31.134.150, dest_ip = 172.31.134.146, src_port=9, dest_port = 49154.
Produces the same rss value (mbuf.hash->rss) on the client (4272763546).
On the other hand, rte_softrss_be and rte_softrss functions (both generate the same value) on the client output 3752309715 and on the server 1141129430.
Any idea ? can be a kernel problem (i.e., modules) ?
Can you please provide me how mbuf.hash->rss (i.e., Toeplitz function) is calculated ?
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Saleh, Omran
Sent: Mittwoch, 28. Dezember 2016 15:19
To: Shahaf Shuler <shahafs@mellanox.com>; users@dpdk.org
Subject: Re: [dpdk-users] Mellanox rss hash information
I am still getting the same problem with the optimized function, the result value from rte_softrss_be is different than mbuf.hash->rss.
any idea about the problem, can be an old kernel produce this problem?
-----Original Message-----
From: Shahaf Shuler [mailto:shahafs@mellanox.com]
Sent: Mittwoch, 28. Dezember 2016 12:00
To: Saleh, Omran <omran.saleh@sap.com>; users@dpdk.org
Subject: RE: Mellanox rss hash information
Have you tried you use rte_softrss_be function?
I suspect you have an endianness issue here as the mlx5 rss_hash_default_key is already in big-endian format.
/**
* Optimized implementation.
* If you want the calculated hash value matches NIC RSS value
* you have to use special converted key with rte_convert_rss_key() fn.
* @param input_tuple
* Pointer to input tuple
* @param input_len
* Length of input_tuple in 4-bytes chunks
* @param *rss_key
* Pointer to RSS hash key.
* @return
* Calculated hash value.
*/
static inline uint32_t
rte_softrss_be(uint32_t *input_tuple, uint32_t input_len,
const uint8_t *rss_key)
{
...
}
--Shahaf
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Saleh, Omran
Sent: Wednesday, December 28, 2016 10:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Mellanox rss hash information
Hi All,
Currently, I am using DPDK 16.11 with Mellanox NIC adapters (mlx5). I encounter a problem that the RSS hash value from mbuf (mbug.hash->rss) is not the same as calculated from rte_softrss(). It is completely different.
I am using the default hash key from mlx5
uint8_t rss_hash_default_key[] = {
0x2c, 0xc6, 0x81, 0xd1,
0x5b, 0xdb, 0xf4, 0xf7,
0xfc, 0xa2, 0x83, 0x19,
0xdb, 0x1a, 0x3e, 0x94,
0x6b, 0x9e, 0x38, 0xd9,
0x2c, 0x9c, 0x03, 0xd1,
0xad, 0x99, 0x44, 0xa7,
0xd9, 0x56, 0x3d, 0x59,
0x06, 0x3c, 0x25, 0xf3,
0xfc, 0x1f, 0xdc, 0x2a,
};
And the following call:
auto hash = rte_softrss(&tuple.v4.src_addr, RTE_THASH_V4_L4_LEN, rss_hash_default_key); with RSS configurations as the following:
port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS; port_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_NONFRAG_IPV4_TCP; port_conf.rx_adv_conf.rss_conf.rss_key= rss_hash_default_key; port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
This is working perfectly with ixgbe but Mellanox not. Can you help me to figure out what is the problem? What is the default Toeplitz function for mlx5?!
My NIC adapters are Mellanox Technologies MT27700 Family [ConnectX-4] With firmware version 12.16.1006 and ofed version 3.3-2.0.0
uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux
Thank you
Best regards,
Omran Saleh
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-users] Mellanox rss hash information
2016-12-28 14:18 ` Saleh, Omran
2016-12-29 8:05 ` Saleh, Omran
@ 2017-01-03 12:30 ` Shahaf Shuler
1 sibling, 0 replies; 6+ messages in thread
From: Shahaf Shuler @ 2017-01-03 12:30 UTC (permalink / raw)
To: Saleh, Omran, users
Hi Saleh,
Looking into the kernel, it turns out that the kernel sets by default the Mellanox symmetric RSS hash. This is a proprietary hash function which performs operation on the tuple before it enters the Toeplitz function.
The next kernel drivers will use the spec-compliant RSS function, which will match the rte_softrss output.
--Shahaf
-----Original Message-----
From: Saleh, Omran [mailto:omran.saleh@sap.com]
Sent: Wednesday, December 28, 2016 4:19 PM
To: Shahaf Shuler <shahafs@mellanox.com>; users@dpdk.org
Subject: RE: Mellanox rss hash information
I am still getting the same problem with the optimized function, the result value from rte_softrss_be is different than mbuf.hash->rss.
any idea about the problem, can be an old kernel produce this problem?
-----Original Message-----
From: Shahaf Shuler [mailto:shahafs@mellanox.com]
Sent: Mittwoch, 28. Dezember 2016 12:00
To: Saleh, Omran <omran.saleh@sap.com>; users@dpdk.org
Subject: RE: Mellanox rss hash information
Have you tried you use rte_softrss_be function?
I suspect you have an endianness issue here as the mlx5 rss_hash_default_key is already in big-endian format.
/**
* Optimized implementation.
* If you want the calculated hash value matches NIC RSS value
* you have to use special converted key with rte_convert_rss_key() fn.
* @param input_tuple
* Pointer to input tuple
* @param input_len
* Length of input_tuple in 4-bytes chunks
* @param *rss_key
* Pointer to RSS hash key.
* @return
* Calculated hash value.
*/
static inline uint32_t
rte_softrss_be(uint32_t *input_tuple, uint32_t input_len,
const uint8_t *rss_key)
{
...
}
--Shahaf
-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Saleh, Omran
Sent: Wednesday, December 28, 2016 10:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Mellanox rss hash information
Hi All,
Currently, I am using DPDK 16.11 with Mellanox NIC adapters (mlx5). I encounter a problem that the RSS hash value from mbuf (mbug.hash->rss) is not the same as calculated from rte_softrss(). It is completely different.
I am using the default hash key from mlx5
uint8_t rss_hash_default_key[] = {
0x2c, 0xc6, 0x81, 0xd1,
0x5b, 0xdb, 0xf4, 0xf7,
0xfc, 0xa2, 0x83, 0x19,
0xdb, 0x1a, 0x3e, 0x94,
0x6b, 0x9e, 0x38, 0xd9,
0x2c, 0x9c, 0x03, 0xd1,
0xad, 0x99, 0x44, 0xa7,
0xd9, 0x56, 0x3d, 0x59,
0x06, 0x3c, 0x25, 0xf3,
0xfc, 0x1f, 0xdc, 0x2a,
};
And the following call:
auto hash = rte_softrss(&tuple.v4.src_addr, RTE_THASH_V4_L4_LEN, rss_hash_default_key); with RSS configurations as the following:
port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS; port_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_NONFRAG_IPV4_TCP; port_conf.rx_adv_conf.rss_conf.rss_key= rss_hash_default_key; port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;
This is working perfectly with ixgbe but Mellanox not. Can you help me to figure out what is the problem? What is the default Toeplitz function for mlx5?!
My NIC adapters are Mellanox Technologies MT27700 Family [ConnectX-4] With firmware version 12.16.1006 and ofed version 3.3-2.0.0
uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux
Thank you
Best regards,
Omran Saleh
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-01-03 12:30 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-12-28 8:28 [dpdk-users] Mellanox rss hash information Saleh, Omran
2016-12-28 10:59 ` Shahaf Shuler
2016-12-28 14:18 ` Saleh, Omran
2016-12-29 8:05 ` Saleh, Omran
2016-12-29 10:38 ` Saleh, Omran
2017-01-03 12:30 ` Shahaf Shuler
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).