From: Ruslan Nikolaev <ruslan@purestorage.com>
To: dev@dpdk.org
Subject: [dpdk-dev] MSI-X vector #1 seems to be stalled sometimes after VF reset (ixgbe)
Date: Mon, 9 Jan 2017 18:42:10 -0800 [thread overview]
Message-ID: <56D73D38-A0F0-4289-BEF2-A46620461E94@purestorage.com> (raw)
Attached are 2 patches, and the discussion below is related to the slightly modified version of the dpdk 16.07 library: interrupts_excerpt.patch and dpdk_vfreset.patch
1. We use a single-shot interrupt mechanism for the RX queue (vector #1, intr_handle.efds[0]file descriptor).
When we receive first interrupt, we start a polling thread. When the polling thread becomes idle again, we enable interrupts.
(to enable interrupts, we use rte_eth_dev_rx_intr_enable, queue_id = 0)
2. We enable interrupts right away for mailbox, reset adapter notifications (vector #0, intr_handle.fd file descriptor)
(to enable interrupts, we use rte_eth_dev_rx_intr_enable, queue_id = UINT16_MAX which we reserved for non-RX interrupts)
3. Changes related to interrupt setup and enabling/disabling are in interrupt_excerpt.patch
Changes: Seems like writing to the register already implies OR-semantic in interrupt enabling, so it does not seem be necessary to read previous value of the register (especially that now we have to have 2 vectors and want to avoid any race condition between reading and writing the register). Also, rte_intr_enable is going to write the same configuration to VFIO which does not seem to be necessary. Could you confirm and/or clarify that?
For disabling interrupt, it seems we have to use a different register.
4. Changes related to resetting devices are in dpdk_vfreset.patch
We used an unofficial patch from http://dpdk.org/dev/patchwork/patch/14009/ as the model. The patch is doing pretty the same thing but just maintains a state machine for our convenience, so that we can have a loop outside the reset function.
5. We see an intermittent stall of interrupt vector #1 when links are toggled. It does not always happen but only intermittently. Vector #0 still seems to work fine because we are able to get mailbox interrupts (when the adapter is reset).
Our current suspicion is that it may have something to do to the reset adapter handling in the unofficial patch (which, in turns, relies on dev_stop/dev_start functions). It appears that vector #1 (RX) interrupts are stalled intermittently only after the adapter reset takes place.
Please give your advice / suggestions.
next reply other threads:[~2017-01-10 2:42 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-01-10 2:42 Ruslan Nikolaev [this message]
2017-01-10 2:47 ` Ruslan Nikolaev
2017-01-10 2:48 ` Ruslan Nikolaev
2017-01-10 10:39 ` Mcnamara, John
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56D73D38-A0F0-4289-BEF2-A46620461E94@purestorage.com \
--to=ruslan@purestorage.com \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).