From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 47953A057B; Thu, 2 Apr 2020 14:43:15 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 2B3E02BAF; Thu, 2 Apr 2020 14:43:15 +0200 (CEST) Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id C5553FFA for ; Thu, 2 Apr 2020 14:43:12 +0200 (CEST) IronPort-SDR: q3TOvOS2OSocZf5m77G1PF6xCpbuWVIaH9825h4gFZSH6hk2a/n7npscLEUuYLKP7X4A+uIpYI xWQfGPTTVQgA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Apr 2020 05:43:11 -0700 IronPort-SDR: 0ymw2nj7oLotiy5+Th4N7EjsB34Mngqn+4z+n8E3Mo7fDWUJxdRWE0Wo76ca/JHyWA35gG1i04 I8hoMrFF+I6w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.72,335,1580803200"; d="scan'208";a="396351831" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by orsmga004.jf.intel.com with ESMTP; 02 Apr 2020 05:43:11 -0700 Received: from shsmsx602.ccr.corp.intel.com (10.109.6.142) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 2 Apr 2020 05:43:10 -0700 Received: from shsmsx603.ccr.corp.intel.com (10.109.6.143) by SHSMSX602.ccr.corp.intel.com (10.109.6.142) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 2 Apr 2020 20:43:08 +0800 Received: from shsmsx603.ccr.corp.intel.com ([10.109.6.143]) by SHSMSX603.ccr.corp.intel.com ([10.109.6.143]) with mapi id 15.01.1713.004; Thu, 2 Apr 2020 20:43:08 +0800 From: "Wang, Haiyue" To: "vattunuru@marvell.com" , "dev@dpdk.org" CC: "jerinj@marvell.com" , "alex.williamson@redhat.com" , "thomas@monjalon.net" , "david.marchand@redhat.com" Thread-Topic: [dpdk-dev] [RFC v1 1/1] vfio: set vf token and gain vf device access Thread-Index: AQHV8qdJrs8OwxlGCEq3pxKcvWBkzqhl8aIg Date: Thu, 2 Apr 2020 12:43:08 +0000 Message-ID: <28a7b43ac5704c9781a035ab4a14d352@intel.com> References: <20200305043311.17065-1-vattunuru@marvell.com> In-Reply-To: <20200305043311.17065-1-vattunuru@marvell.com> Accept-Language: zh-CN, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-reaction: no-action dlp-version: 11.2.0.6 x-originating-ip: [10.239.127.36] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [RFC v1 1/1] vfio: set vf token and gain vf device access X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi A Vamsi, > -----Original Message----- > From: dev On Behalf Of vattunuru@marvell.com > Sent: Thursday, March 5, 2020 12:33 > To: dev@dpdk.org > Cc: jerinj@marvell.com; alex.williamson@redhat.com; thomas@monjalon.net; = david.marchand@redhat.com; > Vamsi Attunuru > Subject: [dpdk-dev] [RFC v1 1/1] vfio: set vf token and gain vf device ac= cess >=20 > From: Vamsi Attunuru >=20 > vfio-pci driver enables virtual function access from > the DPDK applications when those vf device's physical > function is also bound to vfio driver. >=20 > Patch adds the required configuration and checks to > enable DPDK applications to access both pf and it's > vf devices through vfio-pci driver. >=20 > See background on vf token scheme in linux vfio driver. > http://patches.dpdk.org/cover/65915/ >=20 > When a physical function is enabled with non-zero > virtual functions, patch sets the UUID using > VFIO_DEVICE_FEATURE ioctl from physical function's > file descriptor. Same UUID is used to gain the access > for the virtual functions on those physical function. >=20 > Following changes required on top of this DPDK patch > * Kernel version check for VFIO_DEVICE_FEATURE ioctl > * Use uuid gen API to generate UUID. >=20 > Signed-off-by: Vamsi Attunuru >=20 > diff --git a/lib/librte_eal/linux/eal/eal_vfio.c b/lib/librte_eal/linux/e= al/eal_vfio.c > index 01b5ef3..e2fdd35 100644 > --- a/lib/librte_eal/linux/eal/eal_vfio.c > +++ b/lib/librte_eal/linux/eal/eal_vfio.c > @@ -12,6 +12,7 @@ > #include > #include > #include > +#include > #include >=20 > #include "eal_filesystem.h" > @@ -50,6 +51,9 @@ struct vfio_config { > struct user_mem_maps mem_maps; > }; >=20 > +rte_uuid_t uuid_token =3D RTE_UUID_INIT(0xf8615163, 0xdf3e, 0x46c5, > + 0x913f, 0xf2d2f965ed0eULL); > + > /* per-process VFIO config */ > static struct vfio_config vfio_cfgs[VFIO_MAX_CONTAINERS]; > static struct vfio_config *default_vfio_cfg =3D &vfio_cfgs[0]; > @@ -657,6 +661,102 @@ rte_vfio_clear_group(int vfio_group_fd) > return 0; > } >=20 > +static bool > +rte_vfio_dev_is_physfn(const char *sysfs_base, const char *dev_addr) > +{ > + char linkname[PATH_MAX]; > + char filename[PATH_MAX]; > + int ret; > + > + memset(linkname, 0, sizeof(linkname)); > + memset(filename, 0, sizeof(filename)); > + > + /* check if physfn directory exist for this device */ > + snprintf(linkname, sizeof(linkname), > + "%s/%s/physfn", sysfs_base, dev_addr); > + > + ret =3D readlink(linkname, filename, sizeof(filename)); > + > + /* For PFs, physfn directory does not exist */ > + if (ret < 0) > + return true; > + > + return false; > +} > + > +static int > +is_vf_token_required(const char *sysfs_base, const char *dev_addr) > +{ > + char *tok[16], *physfn, *physfn_drv; > + char linkname[PATH_MAX]; > + char filename[PATH_MAX]; > + int ret; > + > + memset(linkname, 0, sizeof(linkname)); > + memset(filename, 0, sizeof(filename)); > + > + snprintf(linkname, sizeof(linkname), > + "%s/%s/physfn", sysfs_base, dev_addr); > + > + ret =3D readlink(linkname, filename, sizeof(filename)); > + if (ret < 0) > + return -1; > + > + ret =3D rte_strsplit(filename, sizeof(filename), > + tok, RTE_DIM(tok), '/'); > + if (ret <=3D 0) { > + RTE_LOG(ERR, EAL, " %s cannot get it's physfn\n", dev_addr); > + return -1; > + } > + > + physfn =3D tok[ret - 1]; > + > + snprintf(linkname, sizeof(linkname), > + "/sys/bus/pci/devices/%s/driver", physfn); > + ret =3D readlink(linkname, filename, sizeof(filename)); > + if (ret < 0) > + return -1; > + > + ret =3D rte_strsplit(filename, sizeof(filename), > + tok, RTE_DIM(tok), '/'); > + if (ret <=3D 0) { > + RTE_LOG(ERR, EAL, " %s cannot get it's physfn driver info\n", > + dev_addr); > + return -1; > + } > + > + physfn_drv =3D tok[ret - 1]; > + > + if (strncmp(physfn_drv, "vfio-pci", sizeof("vfio-pci"))) > + return 1; > + > + /* physfn is bound to vfio-pci */ > + return 0; > +} > + Based on Alex's vfio for qemu patch (https://lore.kernel.org/lkml/202002041= 61737.34696b91@w520.home/), and understand the SR-IOV design by reading your RFC, I scratched a simple = design. 1. ./usertools/dpdk-devbind.py -b vfio-pci 0000:87:00.0 2. echo 2 > /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs 3. ./x86_64-native-linux-gcc/app/testpmd -l 22-25 -n 4 -w 87:00.0,vf_token= =3D2ab74924-c335-45f4-9b16-8569e5b08258 --file-prefix=3Dpf -- -i If no token in VF: ./x86_64-native-linux-gcc/app/testpmd -l 26-29 -n 4 -w 87:02.0 --file-prefi= x=3Dvf1 -- -i Kernel error: vfio-pci 0000:87:02.0: VF token required to access device ./x86_64-native-linux-gcc/app/testpmd -l 26-29 -n 4 -w 87:02.0,vf_token=3D2= ab74924-c335-45f4-9b16-8569e5b08258 --file-prefix=3Dvf1 -- -i static int vfio_pci_vf_token_arg_handler(__rte_unused const char *key, const char *value, void *opaque) { if (rte_uuid_parse(value, opaque)) { RTE_LOG(ERR, EAL, "The VF token is not a valid uuid : %s\n", value); return -1; } return 0; } static int vfio_pci_vf_token_arg(struct rte_devargs *devargs, rte_uuid_t uu) { const char *key =3D "vf_token"; struct rte_kvargs *kvlist; int ret =3D 0; if (devargs =3D=3D NULL) return 0; kvlist =3D rte_kvargs_parse(devargs->args, NULL); if (kvlist =3D=3D NULL) return 0; if (!rte_kvargs_count(kvlist, key)) goto exit; if (rte_kvargs_process(kvlist, key, vfio_pci_vf_token_arg_handler, uu) < 0) goto exit; ret =3D 1; exit: rte_kvargs_free(kvlist); return ret; } static int pci_vfio_map_resource_primary(struct rte_pci_device *dev) { struct vfio_device_info device_info =3D { .argsz =3D sizeof(device_info) = }; + rte_uuid_t vf_token =3D RTE_UUID_INIT(0, 0, 0, 0, 0ULL); char pci_addr[PATH_MAX] =3D {0}; int vfio_dev_fd; struct rte_pci_addr *loc =3D &dev->addr; @@ -668,8 +712,9 @@ pci_vfio_map_resource_primary(struct rte_pci_device *de= v) snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT, loc->domain, loc->bus, loc->devid, loc->function); =20 + vfio_pci_vf_token_arg(dev->device.devargs, vf_token); ret =3D rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, - &vfio_dev_fd, &device_info); + &vfio_dev_fd, &device_info, vf_token); if (ret) return ret; int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr, - int *vfio_dev_fd, struct vfio_device_info *device_info) + int *vfio_dev_fd, struct vfio_device_info *device_info, + rte_uuid_t vf_token) { struct vfio_group_status group_status =3D { .argsz =3D sizeof(group_status) @@ -712,6 +713,7 @@ rte_vfio_setup_device(const char *sysfs_base, const cha= r *dev_addr, int vfio_container_fd; int vfio_group_fd; int iommu_group_num; + char dev[PATH_MAX]; int i, ret; =20 /* get group number */ @@ -895,8 +897,19 @@ rte_vfio_setup_device(const char *sysfs_base, const ch= ar *dev_addr, t->type_id, t->name); } =20 + if (!rte_uuid_is_null(vf_token)) { + char vf_token_str[PATH_MAX]; + + rte_uuid_unparse(vf_token, vf_token_str, sizeof(vf_token_str)); + snprintf(dev, sizeof(dev), + "%s vf_token=3D%s", dev_addr, vf_token_str); + } else { + snprintf(dev, sizeof(dev), + "%s", dev_addr); + } + /* get a file descriptor for the device */ - *vfio_dev_fd =3D ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_addr)= ; + *vfio_dev_fd =3D ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev); > +static bool > +rte_vfio_dev_has_nonzero_numvfs(const char *sysfs_base, const char *dev_= addr) > +{ > + char linkname[PATH_MAX]; > + unsigned long num_vfs; > + int ret; > + > + if (!rte_vfio_dev_is_physfn(sysfs_base, dev_addr)) > + return false; > + > + memset(linkname, 0, sizeof(linkname)); > + > + snprintf(linkname, sizeof(linkname), > + "%s/%s/sriov_numvfs", sysfs_base, dev_addr); > + > + ret =3D eal_parse_sysfs_value(linkname, &num_vfs); > + > + if ((ret < 0) || (num_vfs =3D=3D 0)) > + return false; > + > + return true; > +} > + > int > rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr, > int *vfio_dev_fd, struct vfio_device_info *device_info) > @@ -669,6 +769,7 @@ rte_vfio_setup_device(const char *sysfs_base, const c= har *dev_addr, > int vfio_container_fd; > int vfio_group_fd; > int iommu_group_num; > + char dev[PATH_MAX]; > int i, ret; >=20 > /* get group number */ > @@ -683,6 +784,29 @@ rte_vfio_setup_device(const char *sysfs_base, const = char *dev_addr, > if (ret < 0) > return -1; >=20 > + snprintf(dev, sizeof(dev), "%s", dev_addr); > + > + if (!rte_vfio_dev_is_physfn(sysfs_base, dev_addr)) { > + char vf_token[PATH_MAX]; > + /* > + * Check if vf_token is required or not, > + * vf_token is required when the VF's physfn is > + * binded with vfio-pci driver > + */ > + ret =3D is_vf_token_required(sysfs_base, dev_addr); > + /* if negative, something failed */ > + if (ret < 0) > + return -1; > + > + if (ret =3D=3D 0) { > + /* vf_token required to open device file descriptor */ > + rte_uuid_unparse(uuid_token, > + vf_token, sizeof(vf_token)); > + snprintf(dev, sizeof(dev), > + "%s vf_token=3D%s", dev_addr, vf_token); > + } > + } > + > /* get the actual group fd */ > vfio_group_fd =3D rte_vfio_get_group_fd(iommu_group_num); > if (vfio_group_fd < 0) > @@ -853,7 +977,7 @@ rte_vfio_setup_device(const char *sysfs_base, const c= har *dev_addr, > } >=20 > /* get a file descriptor for the device */ > - *vfio_dev_fd =3D ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev_add= r); > + *vfio_dev_fd =3D ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, dev); > if (*vfio_dev_fd < 0) { > /* if we cannot get a device fd, this implies a problem with > * the VFIO group or the container not having IOMMU configured. > @@ -877,6 +1001,31 @@ rte_vfio_setup_device(const char *sysfs_base, const= char *dev_addr, > rte_vfio_clear_group(vfio_group_fd); > return -1; > } > + > + if (rte_vfio_dev_has_nonzero_numvfs(sysfs_base, dev_addr)) { > +#define VF_TOKEN (sizeof(struct vfio_device_feature) + sizeof(rte_uuid_= t)) > + > + struct vfio_device_feature *vf_token; > + uint8_t local[VF_TOKEN]; > + > + memset(local, 0, VF_TOKEN); > + vf_token =3D (struct vfio_device_feature *)local; > + vf_token->argsz =3D VF_TOKEN; > + vf_token->flags =3D VFIO_DEVICE_FEATURE_SET | > + VFIO_DEVICE_FEATURE_PCI_VF_TOKEN; > + > + memcpy(local + sizeof(struct vfio_device_feature), > + &uuid_token, sizeof(uuid_token)); > + > + ret =3D ioctl(*vfio_dev_fd, VFIO_DEVICE_FEATURE, vf_token); > + if (ret) { > + RTE_LOG(ERR, EAL, " Failed to set UUID on %s " > + "error %i (%s)\n", dev_addr, errno, > + strerror(errno)); > + return -1; > + } > + } > + > vfio_group_device_get(vfio_group_fd); >=20 > return 0; > -- > 2.8.4