From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f172.google.com (mail-pf0-f172.google.com [209.85.192.172]) by dpdk.org (Postfix) with ESMTP id 22C95152A for ; Tue, 10 Jan 2017 03:42:17 +0100 (CET) Received: by mail-pf0-f172.google.com with SMTP id f144so28429351pfa.2 for ; Mon, 09 Jan 2017 18:42:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=purestorage.com; s=google; h=from:subject:message-id:date:to:mime-version; bh=u8wwrEvLZlB4oE/AehXKEFCWE2HoehxOZEob5Z3dKu4=; b=aSkZzteYsWETSqUdfUAkgB32QuJ+nRPHZFOzYK8fmJhOUZlxlD8gQwOCYJcTxcZxLR cmp1OV8xNrVEUjFXXJgh/zkhRL7OxYgAjIcHJYN4UtdFWA1ub5CxJkkT6fOeDzl3p5NC aTHfYaR5yJmi1P++yJl60cNDPQLymGI4m/qGw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:message-id:date:to:mime-version; bh=u8wwrEvLZlB4oE/AehXKEFCWE2HoehxOZEob5Z3dKu4=; b=nq5hcmSTLYO7l9TyXwv4AlvTaWNpBT4pQ45nzMh8R661ioTdJMhWEmHZahb2GskPcT TXBuzV1ux4rknGTh+GyNJvu5/5yGCAB2KhWYC/4GeZgAMUO/pLtnCo2PHAW4+hGc0VFn RcL4Jk8IEQWzoaSxdlAu3e36VourMzoXTjkjpnW0Fnvz8i+Vl+EU4/kP4O5/XICaXt9a ipB38b7Cbq4kQVP/uZZ8UDdHAH0byksOyBfVLdRiRMrVAjHfGOdS3fb1EV+DN2v2Ua/a GLtmuUJmK0ab5MFTriUiDuioJ7NRhj0cjNimaZyCFFYrxuPpLMHn7uewbHaBWJ+Zz/Ub H4IA== X-Gm-Message-State: AIkVDXL+b4gZImH1vrsmw4Atq3XkN4l2QkdLpG/SyiAKabvESHrbc8v8SKSrFMB4SFXVimsh X-Received: by 10.98.29.74 with SMTP id d71mr1076694pfd.144.1484016137073; Mon, 09 Jan 2017 18:42:17 -0800 (PST) Received: from ruslan-mbp.purestorage.com ([64.84.70.252]) by smtp.gmail.com with ESMTPSA id s5sm562718pgj.19.2017.01.09.18.42.16 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 09 Jan 2017 18:42:16 -0800 (PST) From: Ruslan Nikolaev Message-Id: <56D73D38-A0F0-4289-BEF2-A46620461E94@purestorage.com> Date: Mon, 9 Jan 2017 18:42:10 -0800 To: dev@dpdk.org Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-dev] MSI-X vector #1 seems to be stalled sometimes after VF reset (ixgbe) X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jan 2017 02:42:18 -0000 Attached are 2 patches, and the discussion below is related to the = slightly modified version of the dpdk 16.07 library: = interrupts_excerpt.patch and dpdk_vfreset.patch 1. We use a single-shot interrupt mechanism for the RX queue (vector #1, = intr_handle.efds[0]file descriptor). When we receive first interrupt, we start a polling thread. When the = polling thread becomes idle again, we enable interrupts. (to enable interrupts, we use rte_eth_dev_rx_intr_enable, queue_id =3D = 0) 2. We enable interrupts right away for mailbox, reset adapter = notifications (vector #0, intr_handle.fd file descriptor) (to enable interrupts, we use rte_eth_dev_rx_intr_enable, queue_id =3D = UINT16_MAX which we reserved for non-RX interrupts) 3. Changes related to interrupt setup and enabling/disabling are in = interrupt_excerpt.patch Changes: Seems like writing to the register already implies OR-semantic = in interrupt enabling, so it does not seem be necessary to read previous = value of the register (especially that now we have to have 2 vectors and = want to avoid any race condition between reading and writing the = register). Also, rte_intr_enable is going to write the same = configuration to VFIO which does not seem to be necessary. Could you = confirm and/or clarify that? For disabling interrupt, it seems we have to use a different register. 4. Changes related to resetting devices are in dpdk_vfreset.patch We used an unofficial patch from = http://dpdk.org/dev/patchwork/patch/14009/ as the model. The patch is = doing pretty the same thing but just maintains a state machine for our = convenience, so that we can have a loop outside the reset function. 5. We see an intermittent stall of interrupt vector #1 when links are = toggled. It does not always happen but only intermittently. Vector #0 = still seems to work fine because we are able to get mailbox interrupts = (when the adapter is reset). Our current suspicion is that it may have something to do to the reset = adapter handling in the unofficial patch (which, in turns, relies on = dev_stop/dev_start functions). It appears that vector #1 (RX) interrupts = are stalled intermittently only after the adapter reset takes place. Please give your advice / suggestions.