From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <t.yoshimura8869@gmail.com>
Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com
[209.85.215.193]) by dpdk.org (Postfix) with ESMTP id 707D9160
for <dev@dpdk.org>; Fri, 20 Jul 2018 10:13:55 +0200 (CEST)
Received: by mail-pg1-f193.google.com with SMTP id y5-v6so6224004pgv.1
for <dev@dpdk.org>; Fri, 20 Jul 2018 01:13:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025;
h=from:to:cc:subject:date:message-id:in-reply-to:references;
bh=Xb/L4C444IcS15/vG3E+6h6KT41mB/7ErkQyVQi0gvM=;
b=mAz2hhPVIBq+1/D8eyqMq0pcULMRcGkSFUpxruVbHARqWQfHFoXgNr8K1dhgVKQWRo
NiPS+WLkTvLU5SBArUmHGw8WipSxGKN8wE3BCwYsh59oMEjwW3fSAcyJXXThtR9pZUS5
wFFF60hEcY2/OJN6icqNi8ToWX9vjCtqB8gIVputtUzZJiGEBh5mz+Nx4RDZq1sfigWr
wlXslra+zUixwHQrh8rbgAgEZmykrzAXWENDEOUe7pQX2x8VQoFHb23m3Sl5fvZS6etj
lgQLKFNUS3idMoRYajefk+ESrjuSRaEvdjTk1Xbn5xg5CF6dZyjYyyZJVLrwaZfrta8+
E60w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to
:references;
bh=Xb/L4C444IcS15/vG3E+6h6KT41mB/7ErkQyVQi0gvM=;
b=QJjIg+KK/H3+/kXA5rsLm1Xu5pitMO0/Sf3FZ0QufUET4lBdCOH+TuXpBjTxLv3RZa
dPJ4HXy45MmjUmId6p8BL/9InERPcFOsudOSx+KRXS/MD4GhOzWsxn9xULx8Fp+Sdp0o
sFSCS+/i4z5I8X/UncS4iwpZB1MVZ9L7gwb62tuSvHt7a6SKBLJg1VgkCRLx1ZftdpGg
P2W73bvv5nnr+vwFUD7h5d36X/hN4gjZz21rhD5s5nfbVsTxbLo0lPUJ9CJk1RppOZM6
9Mb5QJOGRncs8ayKgpbjXji4527VKTnTKGfdjOLbLTkzHRb8ThH86a6KzSn9yP1dRT06
pl0w==
X-Gm-Message-State: AOUpUlE0yF14lBh5DBjm37MFmx0kBuyqc28bcHQaYUFFLjymS0Wc8h2l
/PcWkL95q+4HMEX3JwKF30s=
X-Google-Smtp-Source: AAOMgpc59Q1ipnLhOfDp+SDwd2+BT4KS1tO3L/Fytmrx0wzh/tN48UawixyCZV66Dkc6prFAH6I2cA==
X-Received: by 2002:a63:8042:: with SMTP id
j63-v6mr1126659pgd.230.1532074434534;
Fri, 20 Jul 2018 01:13:54 -0700 (PDT)
Received: from takeshi-no-air.dhcp.hakozaki.ibm.com
(sg-fw-ice-redblue-p3.sagamino.jp.ibm.com. [203.141.91.12])
by smtp.gmail.com with ESMTPSA id a77-v6sm3336212pfj.38.2018.07.20.01.13.52
(version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128);
Fri, 20 Jul 2018 01:13:53 -0700 (PDT)
From: Takeshi Yoshimura <t.yoshimura8869@gmail.com>
To: Anatoly Burakov <anatoly.burakov@intel.com>
Cc: dev@dpdk.org,
Takeshi Yoshimura <t.yoshimura8869@gmail.com>
Date: Fri, 20 Jul 2018 17:13:47 +0900
Message-Id: <20180720081347.6123-1-t.yoshimura8869@gmail.com>
X-Mailer: git-send-email 2.15.1
In-Reply-To: <20180712030833.4887-1-t.yoshimura8869@gmail.com>
References: <20180712030833.4887-1-t.yoshimura8869@gmail.com>
Subject: [dpdk-dev] [PATCH v5] vfio: fix workaround of BAR mapping
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Fri, 20 Jul 2018 08:13:55 -0000
Currently, VFIO will try to map around MSI-X table in the BARs. When
MSI-X table (page-aligned) size is equal to (page-aligned) size of BAR,
VFIO will just skip the BAR.
Recent kernel versions will allow VFIO to map the entire BAR containing
MSI-X tables (*), so instead of trying to map around the MSI-X vector
or skipping the BAR entirely if it's not possible, we can now try
mapping the entire BAR first. If mapping the entire BAR doesn't
succeed, fall back to the old behavior of mapping around MSI-X table or
skipping the BAR.
(*): "vfio-pci: Allow mapping MSIX BAR",
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/
commit/?id=a32295c612c57990d17fb0f41e7134394b2f35f6
Fixes: 90a1633b2347 ("eal/linux: allow to map BARs with MSI-X tables")
Signed-off-by: Takeshi Yoshimura <t.yoshimura8869@gmail.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---
Thanks, Anatoly.
I updated the code with munmap in an error path.
I also fixed the message and the wrong link.
Regards,
Takeshi
drivers/bus/pci/linux/pci_vfio.c | 93 ++++++++++++++++++++++------------------
1 file changed, 52 insertions(+), 41 deletions(-)
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index aeeaa9ed8..07188c071 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -332,50 +332,59 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,
void *bar_addr;
struct pci_msix_table *msix_table = &vfio_res->msix_table;
struct pci_map *bar = &vfio_res->maps[bar_index];
+ bool again = false;
if (bar->size == 0)
/* Skip this BAR */
return 0;
- if (msix_table->bar_index == bar_index) {
- /*
- * VFIO will not let us map the MSI-X table,
- * but we can map around it.
- */
- uint32_t table_start = msix_table->offset;
- uint32_t table_end = table_start + msix_table->size;
- table_end = (table_end + ~PAGE_MASK) & PAGE_MASK;
- table_start &= PAGE_MASK;
-
- if (table_start == 0 && table_end >= bar->size) {
- /* Cannot map this BAR */
- RTE_LOG(DEBUG, EAL, "Skipping BAR%d\n", bar_index);
- bar->size = 0;
- bar->addr = 0;
- return 0;
- }
-
- memreg[0].offset = bar->offset;
- memreg[0].size = table_start;
- memreg[1].offset = bar->offset + table_end;
- memreg[1].size = bar->size - table_end;
-
- RTE_LOG(DEBUG, EAL,
- "Trying to map BAR%d that contains the MSI-X "
- "table. Trying offsets: "
- "0x%04lx:0x%04lx, 0x%04lx:0x%04lx\n", bar_index,
- memreg[0].offset, memreg[0].size,
- memreg[1].offset, memreg[1].size);
- } else {
- memreg[0].offset = bar->offset;
- memreg[0].size = bar->size;
- }
-
/* reserve the address using an inaccessible mapping */
bar_addr = mmap(bar->addr, bar->size, 0, MAP_PRIVATE |
MAP_ANONYMOUS | additional_flags, -1, 0);
- if (bar_addr != MAP_FAILED) {
+ if (bar_addr == MAP_FAILED) {
+ RTE_LOG(ERR, EAL,
+ "Failed to create inaccessible mapping for BAR%d\n",
+ bar_index);
+ return -1;
+ }
+
+ memreg[0].offset = bar->offset;
+ memreg[0].size = bar->size;
+ do {
void *map_addr = NULL;
+ if (again) {
+ /*
+ * VFIO did not let us map the MSI-X table,
+ * but we can map around it.
+ */
+ uint32_t table_start = msix_table->offset;
+ uint32_t table_end = table_start + msix_table->size;
+ table_end = (table_end + ~PAGE_MASK) & PAGE_MASK;
+ table_start &= PAGE_MASK;
+
+ if (table_start == 0 && table_end >= bar->size) {
+ /* Cannot map this BAR */
+ RTE_LOG(DEBUG, EAL, "Skipping BAR%d\n",
+ bar_index);
+ munmap(bar_addr, bar->size);
+ bar->size = 0;
+ bar->addr = 0;
+ return 0;
+ }
+
+ memreg[0].offset = bar->offset;
+ memreg[0].size = table_start;
+ memreg[1].offset = bar->offset + table_end;
+ memreg[1].size = bar->size - table_end;
+
+ RTE_LOG(DEBUG, EAL,
+ "Trying to map BAR%d that contains the MSI-X "
+ "table. Trying offsets: "
+ "0x%04lx:0x%04lx, 0x%04lx:0x%04lx\n", bar_index,
+ memreg[0].offset, memreg[0].size,
+ memreg[1].offset, memreg[1].size);
+ }
+
if (memreg[0].size) {
/* actual map of first part */
map_addr = pci_map_resource(bar_addr, vfio_dev_fd,
@@ -384,6 +393,12 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,
MAP_FIXED);
}
+ if (map_addr == MAP_FAILED &&
+ msix_table->bar_index == bar_index && !again) {
+ again = true;
+ continue;
+ }
+
/* if there's a second part, try to map it */
if (map_addr != MAP_FAILED
&& memreg[1].offset && memreg[1].size) {
@@ -404,12 +419,8 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,
bar_index);
return -1;
}
- } else {
- RTE_LOG(ERR, EAL,
- "Failed to create inaccessible mapping for BAR%d\n",
- bar_index);
- return -1;
- }
+ break;
+ } while (again);
bar->addr = bar_addr;
return 0;
--
2.15.1