From: "Williams, Christopher" <Christopher_Williams@McAfee.com>
To: "users@dpdk.org" <users@dpdk.org>
Cc: "Neal, Peter" <Peter_Neal@McAfee.com>
Subject: [dpdk-users] rte_eth_dev_rss_reta_update appears to have no effect with net_ixgbe driver
Date: Mon, 17 Jul 2017 11:00:39 +0000 [thread overview]
Message-ID: <DM3PR16MB07339FA2288FF6E0973AAC049CA00@DM3PR16MB0733.namprd16.prod.outlook.com> (raw)
We're using receive side scaling with Intel NICs that use the net_ixgbe PMD.
Our application needs to be able to follow entire conversations in a higher-level protocol, such as HTTP or SMTP, which requires us to set a hashing key such that (a, b) hashes to the same value as (b, a) (i.e. client->server packets and server->client packets must end up in the same queue). The requirement to be able to follow FTP conversations limits us to hashing over IP addresses only, since any FTP conversation will use multiple TCP ports.
Hashing IP addresses only results in extremely uneven distribution of packets into the rx queues. We *should* be able to fix this by setting a redirection table to bias packets towards one table rather than another. However, configuring a redirection table appears to have no effect.
The following C++ function is a slight simplification of what we're actually doing:
#define MAX_RETA_SIZE (ETH_RSS_RETA_SIZE_512 / RTE_RETA_GROUP_SIZE)
void config_eth_dev(uint8_t port_id, uint16_t nRx)
{
rte_eth_conf eth_config;
memset(ð_config, 0, sizeof(eth_config));
struct rte_eth_dev_info inf;
rte_eth_dev_info_get(port_id, &inf);
eth_config.rxmode.mq_mode = ETH_MQ_RX_RSS; // Enable receive-side scaling
// Set a hashing key such that frames from a->b and b->a are inserted into
// the same hardware queue
static uint8_t hashKey[] = {
0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A, 0x6D, 0x5A,
};
eth_config.rx_adv_conf.rss_conf.rss_key = hashKey;
eth_config.rx_adv_conf.rss_conf.rss_key_len = sizeof(hashKey);
// Hash IP addresses only so that, e.g., FTP packets go to the same rx queue
eth_config.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IPV4 | ETH_RSS_IPV6;
if (rte_eth_dev_configure(port_id, nRx, 1, ð_config))
{
throw std::runtime_error("rte_eth_dev_configure");
}
rte_eth_dev_info_get(port_id, &inf);
// Set a redirection table - this should distribute traffic more-or-less evenly
// to each queue
uint16_t rSize = inf.reta_size;
// RX queue distribution for two queues
static uint16_t dist2[] = {
0, 0, 0, 0, 1, 1, 1,
};
// RX queue distribution for four queues
static uint16_t dist4[] = {
0, 0, 1, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3,
};
uint16_t *pd = nullptr, i;
size_t nd = 0;
switch (nRx) {
case 4:
pd = dist4;
nd = sizeof(dist4) / sizeof(uint16_t);
break;
case 2:
pd = dist2;
nd = sizeof(dist2) / sizeof(uint16_t);
break;
default:
break;
}
if (rSize && rSize <= ETH_RSS_RETA_SIZE_512 && pd)
{
struct rte_eth_rss_reta_entry64 rConf[MAX_RETA_SIZE];
memset(rConf, 0, sizeof(rConf));
for (i = 0; i < rSize; i++)
{
rConf[i / RTE_RETA_GROUP_SIZE].mask = UINT64_MAX;
}
for (i = 0; i < rSize; i++)
{
uint16_t reta_id = i / RTE_RETA_GROUP_SIZE;
uint16_t reta_pos = i % RTE_RETA_GROUP_SIZE;
uint16_t rss_qs_pos = i % nd;
rConf[reta_id].reta[reta_pos] = pd[rss_qs_pos];
}
if (rte_eth_dev_rss_reta_update(port_id, rConf, rSize))
{
throw std::runtime_error("rte_eth_dev_rss_reta_update");
}
}
}
The code for configuring the redirection table is lifted from the ip_pipeline sample application.
Is there something we're doing wrong here (or something we're failing to do)?
Christopher Williams.
________________________________
McAfee Security UK Limited is registered in England and Wales with its registered address at C/O Skadden, Arps, Slate, Meagher & Flom (UK) LLP, 40 Bank Street, Canary Wharf, London, United Kingdom, E14 5DS, Company No. 10472868
reply other threads:[~2017-07-17 11:00 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DM3PR16MB07339FA2288FF6E0973AAC049CA00@DM3PR16MB0733.namprd16.prod.outlook.com \
--to=christopher_williams@mcafee.com \
--cc=Peter_Neal@McAfee.com \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).