From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1879AA059A; Fri, 10 Apr 2020 15:02:53 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 525C41D55B; Fri, 10 Apr 2020 15:02:52 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id ACA3F1BF57 for ; Fri, 10 Apr 2020 15:02:50 +0200 (CEST) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 03ACtlZw010981; Fri, 10 Apr 2020 06:02:49 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : subject : date : message-id : references : in-reply-to : content-type : content-transfer-encoding : mime-version; s=pfpt0818; bh=NqdWfJGEjPGiBTAHylPe78aJpi5H68HGn2RF53evY+0=; b=thJWLlg1PoV8CPPuQ/9pBbfVDhAlZW92/QIaQBc/dhkjgzq7UfNYb88eL+W0tD71beu8 J8yaVK7XktAA8s5ReEIdBjF6ALJsLf+csK4gxfp58BQG6gJgKB3tp5NYEzGL/A040j0X +X0CV53WPhKFgxFpaB7Ak5qcQ2KcR8C8n3MwWETdFO67quKzNi2ejVRdhCARhkGc9kDw LcDWRDSb6WC+rkFFAnGZ9KL89KEStjeE0IJnXkh6MgwTm5Cu9kVUBIEPKuKL6Pc+unGv HuxwdKceiO4ehb28gu7D0gQhVS3T7OpBaGUlXchw98962geyc62SwWwqf/waUiwGxkQx gA== Received: from sc-exch04.marvell.com ([199.233.58.184]) by mx0a-0016f401.pphosted.com with ESMTP id 3091jwn89j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Fri, 10 Apr 2020 06:02:48 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 10 Apr 2020 06:02:47 -0700 Received: from SC-EXCH04.marvell.com (10.93.176.84) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 10 Apr 2020 06:02:46 -0700 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (104.47.55.176) by SC-EXCH04.marvell.com (10.93.176.84) with Microsoft SMTP Server (TLS) id 15.0.1497.2 via Frontend Transport; Fri, 10 Apr 2020 06:02:46 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=f6GlI6TMod3hUbiPLNieV9Ydh3TDgVPYFg+YA/645s9/APG/0xORmW+TFuZ1V5AwQH2f7ydljrLP9rNfbx6IH+1oELiwW7ffhVCpbtpMqiyY4ooDE3WGvjN2VFkDtXpxEMkpDclQWg+3UHoqtUHDi6LB+0oK6XY5TrPdQh+Up1RXop8MJV7hOXi7U+CD8K5y27he9BE59Uk5TzIcceEG+JfjQzEqFu+e5p34lN5ifAsCE3Ycawl1NFUnZ9KJcWrWQtaMUwecIWeAZxMjFtt4u+mjnDu802WdRjoWw90R+VajnoBP0XIxDgKdhnhEdlff1p6VYpfaeluxiHqTjh/S6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NqdWfJGEjPGiBTAHylPe78aJpi5H68HGn2RF53evY+0=; b=FuCxrzu6TD+dTA89e877aRSm2aNnS7nP3Z4tFncw3V/R7I9N7VGY8x+zwSQ9jgwg5F0fv+c+LayRrTFi8QKgJjOni9jeay7Hnb7ZoLKTWgDqUvRHpDEPI9MfVEi40heCqybQJ+oI9DO273cjYVFq4UsNUlJDnYPA1om4Wh2LDofOZOn3clegDfwxkqTtTeXz8iU/x8ctn/hgEqv6Mn3Uk2zncz7Tt7EJuuFCAfxEafnx1NTNVG/7uuJnqKsMARv6Mzf0RvoXkSMPnEwdqxGsZopANlHbw237lB3qJbpGagy0PSp+nchqGzorUUbfdCXTIsanoniKwfrhGJGkB36kFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=marvell.com; dmarc=pass action=none header.from=marvell.com; dkim=pass header.d=marvell.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.onmicrosoft.com; s=selector1-marvell-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NqdWfJGEjPGiBTAHylPe78aJpi5H68HGn2RF53evY+0=; b=Oys/eTazeGtfiA6ADfjHEciHLeN72uC7mJKGhSFgGHalvCLjflfIRgl29ZI0sxxknTHtU/POtGrAsnjmvzObfqmsAFLdksGwmWRRYtgGV8BO1vaEHSN2CSbxm4c/afoekD8FQKNGZIE/dyfVWpsP8JkmUqPqSFVS1KOA4UfCil0= Received: from MWHPR18MB1645.namprd18.prod.outlook.com (2603:10b6:300:ca::9) by MWHPR18MB1582.namprd18.prod.outlook.com (2603:10b6:300:cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.16; Fri, 10 Apr 2020 13:02:44 +0000 Received: from MWHPR18MB1645.namprd18.prod.outlook.com ([fe80::34df:bd9:ffe5:ff77]) by MWHPR18MB1645.namprd18.prod.outlook.com ([fe80::34df:bd9:ffe5:ff77%8]) with mapi id 15.20.2878.023; Fri, 10 Apr 2020 13:02:44 +0000 From: Vamsi Krishna Attunuru To: Haiyue Wang , "dev@dpdk.org" , "thomas@monjalon.net" , Jerin Jacob Kollanukkaran , "alex.williamson@redhat.com" , "david.marchand@redhat.com" Thread-Topic: [EXT] [RFC v2] eal: add VFIO-PCI SR-IOV support Thread-Index: AQHWDwtuV4P3CHd910aaJrrhlXAa0ahyUlqg Date: Fri, 10 Apr 2020 13:02:43 +0000 Message-ID: References: <20200305043311.17065-1-vattunuru@marvell.com> <20200410073254.34905-1-haiyue.wang@intel.com> In-Reply-To: <20200410073254.34905-1-haiyue.wang@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [157.47.119.179] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a349a24b-c2b0-43e8-0225-08d7dd4f759a x-ms-traffictypediagnostic: MWHPR18MB1582: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:449; x-forefront-prvs: 0369E8196C x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:MWHPR18MB1645.namprd18.prod.outlook.com; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(4636009)(376002)(39860400002)(346002)(136003)(366004)(396003)(66476007)(26005)(53546011)(478600001)(71200400001)(8676002)(5660300002)(6506007)(7696005)(33656002)(9686003)(52536014)(2906002)(81156014)(8936002)(186003)(64756008)(966005)(66556008)(19627235002)(86362001)(110136005)(66446008)(76116006)(66946007)(55016002)(316002); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: marvell.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: qRub3hUfUDk7Q+ZjWyKAjv4HpalN99fJIGSgmlnwtVIl0j/Xm70YsMO/oZOLLsmA5rkKDd8HI1OfX1rpLiElABJnSza1WT5NdhM8o/5RH4z1LLXkOPH2fSnep5eEN7a/LbpDwCRs1cr3ctwait8Vo9ZcMVmLLsJGPVLCI5rme9c+GB8E8tfifJmKubHKnA0lzO3KqPVMQJgT6a6UDNLBL14vYXQMKie7QS6xtGiOxQfSlfGTreNd2OcRjHCTtG0NQzIdMacWL2p59Ttu9obR2iYo75fjqzN7Sa+dvy3A5brPQYYbPyVWdt7CXu0FKa8CAMMl2PnE2XUlYzThIyGuro3zqS1mZ8JuOAAT9XQFkk86zMBpaTGr9uhFS1guaCr6BCDP4uqJ36CB8SZ+KviqSIPRLOJjq6q8ph1GnvX89cMSMroY1EeVHyA+77prRF3EsKE79udkU6bH3a+fLFuIR1rGR4AUU0dtsbEF2x5qTuBeTdVVgn62LTHPRqu/J86tQwQmN/LH5LR4gyOIZcqZqQ== x-ms-exchange-antispam-messagedata: iT3qS5VftbALMJZH5uzFLa5Qiw8o82M2gj858ehG/EQT7Y1WOr2ByMX+bWs/AYGPRQ1vKiD037BOi8ynR2G24aH8MsokeRThwxz/vEQtBeSQHB9LRUfhDbtatUBp8qxEZHZN0oyf/gBo583nrBCRig== Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: a349a24b-c2b0-43e8-0225-08d7dd4f759a X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Apr 2020 13:02:43.9577 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: vxJTU5Tt9JFK4fqG1GqcMxJ/F7iEHcCoPdPHZD4Ogjy5PP5I8oT2l2MNDwCJK06FM2KttesfwKEfZtPF2zrd1A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR18MB1582 X-OriginatorOrg: marvell.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.676 definitions=2020-04-10_04:2020-04-09, 2020-04-10 signatures=0 Subject: Re: [dpdk-dev] [EXT] [RFC v2] eal: add VFIO-PCI SR-IOV support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Haiyue Wang > Sent: Friday, April 10, 2020 1:03 PM > To: dev@dpdk.org; thomas@monjalon.net; Vamsi Krishna Attunuru > ; Jerin Jacob Kollanukkaran ; > alex.williamson@redhat.com; david.marchand@redhat.com > Cc: Haiyue Wang > Subject: [EXT] [RFC v2] eal: add VFIO-PCI SR-IOV support >=20 > External Email >=20 > ---------------------------------------------------------------------- > The kernel module vfio-pci introduces the VF token to enable SR-IOV > support. >=20 > The VF token can be set by a vfio-pci based PF driver and must be known b= y > the vfio-pci based VF driver in order to gain access to the device. >=20 > An example VF token option would take this form: >=20 > 1. ./usertools/dpdk-devbind.py -b vfio-pci 0000:87:00.0 >=20 > 2. echo 2 > /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs >=20 > 3. Start the PF: > ./x86_64-native-linux-gcc/app/testpmd -l 22-25 -n 4 \ > -w 87:00.0,vf_token=3D2ab74924-c335-45f4-9b16-8569e5b08258 \ > --file-prefix=3Dpf -- -i >=20 > 4. Start the VF: > ./x86_64-native-linux-gcc/app/testpmd -l 26-29 -n 4 \ > -w 87:02.0,vf_token=3D2ab74924-c335-45f4-9b16-8569e5b08258 \ > --file-prefix=3Dvf1 -- -i >=20 > Test based on vfio-pci patch: > https://urldefense.proofpoint.com/v2/url?u=3Dhttps- > 3A__patchwork.ozlabs.org_cover_1253222_&d=3DDwIDAg&c=3DnKjWec2b6R0m > OyPaz7xtfQ&r=3DWllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=3DSFD0o > TfFgnU88wT2X7qMWRDen3KFV8oSOlqz3WQm3nI&s=3DCU2jxKUsy4oiI26apob > DF8BJebDrheBiu3eKlF3e76E&e=3D >=20 > Signed-off-by: Haiyue Wang > --- > Based on RFC v1: https://urldefense.proofpoint.com/v2/url?u=3Dhttps- > 3A__patchwork.dpdk.org_patch_66281_&d=3DDwIDAg&c=3DnKjWec2b6R0mOyP > az7xtfQ&r=3DWllrYaumVkxaWjgKto6E_rtDQshhIhik2jkvzFyRhW8&m=3DSFD0oTfFg > nU88wT2X7qMWRDen3KFV8oSOlqz3WQm3nI&s=3D3dHnLcXpkRVOZs6wdZQylc > Je0oNNBs77hXE6SOuLO7o&e=3D > --- > drivers/bus/pci/linux/pci_vfio.c | 56 +++++++++++++++++++++++++++++-- > lib/librte_eal/freebsd/eal.c | 3 +- > lib/librte_eal/include/rte_vfio.h | 8 ++++- > lib/librte_eal/linux/eal_vfio.c | 20 +++++++++-- > 4 files changed, 80 insertions(+), 7 deletions(-) >=20 > diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci= _vfio.c > index 64cd84a68..7f99337c7 100644 > --- a/drivers/bus/pci/linux/pci_vfio.c > +++ b/drivers/bus/pci/linux/pci_vfio.c > @@ -11,6 +11,7 @@ > #include > #include >=20 > +#include > #include > #include > #include > @@ -644,11 +645,59 @@ pci_vfio_msix_is_mappable(int vfio_dev_fd, int > msix_region) > return ret; > } >=20 > +static void > +vfio_pci_vf_token_arg(struct rte_devargs *devargs, rte_uuid_t uu) { > +#define VF_TOKEN_ARG "vf_token=3D" > + char c, *p, *vf_token; > + > + if (devargs =3D=3D NULL) > + return; > + > + p =3D strstr(devargs->args, VF_TOKEN_ARG); > + if (!p) > + return; > + > + vf_token =3D p + strlen(VF_TOKEN_ARG); > + if (strlen(vf_token) < (RTE_UUID_STRLEN - 1)) > + return; > + > + c =3D vf_token[RTE_UUID_STRLEN - 1]; > + if (c !=3D '\0' && c !=3D ',') > + return; > + > + vf_token[RTE_UUID_STRLEN - 1] =3D '\0'; > + if (rte_uuid_parse(vf_token, uu)) { > + RTE_LOG(ERR, EAL, > + "The VF token is not a valid uuid : %s\n", vf_token); > + vf_token[RTE_UUID_STRLEN - 1] =3D c; > + return; > + } > + > + RTE_LOG(DEBUG, EAL, > + "The VF token is found : %s\n", vf_token); > + > + vf_token[RTE_UUID_STRLEN - 1] =3D c; > + > + /* Purge this vfio-pci specific token from the device arguments */ > + if (c !=3D '\0') { > + /* 1. Handle the case : 'vf_token=3Duuid,arg1=3Dval1' */ > + memmove(p, vf_token + RTE_UUID_STRLEN, > + strlen(vf_token + RTE_UUID_STRLEN) + 1); > + } else { > + /* 2. Handle the case : 'arg1=3Dval1,vf_token=3Duuid' */ > + if (p !=3D devargs->args) > + p--; > + > + *p =3D '\0'; > + } > +} >=20 > static int > pci_vfio_map_resource_primary(struct rte_pci_device *dev) { > struct vfio_device_info device_info =3D { .argsz =3D sizeof(device_info= ) }; > + rte_uuid_t vf_token =3D RTE_UUID_INIT(0, 0, 0, 0, 0ULL); > char pci_addr[PATH_MAX] =3D {0}; > int vfio_dev_fd; > struct rte_pci_addr *loc =3D &dev->addr; @@ -668,8 +717,9 @@ > pci_vfio_map_resource_primary(struct rte_pci_device *dev) > snprintf(pci_addr, sizeof(pci_addr), PCI_PRI_FMT, > loc->domain, loc->bus, loc->devid, loc->function); >=20 > + vfio_pci_vf_token_arg(dev->device.devargs, vf_token); > ret =3D rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, > - &vfio_dev_fd, &device_info); > + &vfio_dev_fd, &device_info, > vf_token); > if (ret) > return ret; >=20 > @@ -797,6 +847,7 @@ static int > pci_vfio_map_resource_secondary(struct rte_pci_device *dev) { > struct vfio_device_info device_info =3D { .argsz =3D sizeof(device_info= ) }; > + rte_uuid_t vf_token =3D RTE_UUID_INIT(0, 0, 0, 0, 0ULL); > char pci_addr[PATH_MAX] =3D {0}; > int vfio_dev_fd; > struct rte_pci_addr *loc =3D &dev->addr; @@ -830,8 +881,9 @@ > pci_vfio_map_resource_secondary(struct rte_pci_device *dev) > return -1; > } >=20 > + vfio_pci_vf_token_arg(dev->device.devargs, vf_token); > ret =3D rte_vfio_setup_device(rte_pci_get_sysfs_path(), pci_addr, > - &vfio_dev_fd, &device_info); > + &vfio_dev_fd, &device_info, > vf_token); > if (ret) > return ret; >=20 > diff --git a/lib/librte_eal/freebsd/eal.c b/lib/librte_eal/freebsd/eal.c = index > 6ae37e7e6..cbb4c400e 100644 > --- a/lib/librte_eal/freebsd/eal.c > +++ b/lib/librte_eal/freebsd/eal.c > @@ -995,7 +995,8 @@ rte_eal_vfio_intr_mode(void) int > rte_vfio_setup_device(__rte_unused const char *sysfs_base, > __rte_unused const char *dev_addr, > __rte_unused int *vfio_dev_fd, > - __rte_unused struct vfio_device_info *device_info) > + __rte_unused struct vfio_device_info *device_info > + __rte_unused rte_uuid_t vf_token) > { > return -1; > } > diff --git a/lib/librte_eal/include/rte_vfio.h b/lib/librte_eal/include/r= te_vfio.h > index 20ed8c45a..1f9e22d82 100644 > --- a/lib/librte_eal/include/rte_vfio.h > +++ b/lib/librte_eal/include/rte_vfio.h > @@ -16,6 +16,8 @@ extern "C" { >=20 > #include >=20 > +#include > + > /* > * determine if VFIO is present on the system > */ > @@ -102,13 +104,17 @@ struct vfio_device_info; > * @param device_info > * Device information. > * > + * @param vf_token > + * VF token. > + * > * @return > * 0 on success. > * <0 on failure. > * >1 if the device cannot be managed this way. > */ > int rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr, > - int *vfio_dev_fd, struct vfio_device_info *device_info); > + int *vfio_dev_fd, struct vfio_device_info *device_info, > + rte_uuid_t vf_token); >=20 > /** > * Release a device mapped to a VFIO-managed I/O MMU group. > diff --git a/lib/librte_eal/linux/eal_vfio.c b/lib/librte_eal/linux/eal_v= fio.c > index 4502aefed..8c0ad04e5 100644 > --- a/lib/librte_eal/linux/eal_vfio.c > +++ b/lib/librte_eal/linux/eal_vfio.c > @@ -702,7 +702,8 @@ rte_vfio_clear_group(int vfio_group_fd) >=20 > int > rte_vfio_setup_device(const char *sysfs_base, const char *dev_addr, > - int *vfio_dev_fd, struct vfio_device_info *device_info) > + int *vfio_dev_fd, struct vfio_device_info *device_info, > + rte_uuid_t vf_token) > { > struct vfio_group_status group_status =3D { > .argsz =3D sizeof(group_status) > @@ -712,6 +713,7 @@ rte_vfio_setup_device(const char *sysfs_base, const > char *dev_addr, > int vfio_container_fd; > int vfio_group_fd; > int iommu_group_num; > + char dev[PATH_MAX]; > int i, ret; >=20 > /* get group number */ > @@ -895,8 +897,19 @@ rte_vfio_setup_device(const char *sysfs_base, > const char *dev_addr, > t->type_id, t->name); > } >=20 > + if (!rte_uuid_is_null(vf_token)) { > + char vf_token_str[RTE_UUID_STRLEN]; > + > + rte_uuid_unparse(vf_token, vf_token_str, > sizeof(vf_token_str)); > + snprintf(dev, sizeof(dev), > + "%s vf_token=3D%s", dev_addr, vf_token_str); > + } else { > + snprintf(dev, sizeof(dev), > + "%s", dev_addr); > + } > + > /* get a file descriptor for the device */ > - *vfio_dev_fd =3D ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, > dev_addr); > + *vfio_dev_fd =3D ioctl(vfio_group_fd, VFIO_GROUP_GET_DEVICE_FD, > dev); > if (*vfio_dev_fd < 0) { > /* if we cannot get a device fd, this implies a problem with > * the VFIO group or the container not having IOMMU > configured. > @@ -2081,7 +2094,8 @@ int > rte_vfio_setup_device(__rte_unused const char *sysfs_base, > __rte_unused const char *dev_addr, > __rte_unused int *vfio_dev_fd, > - __rte_unused struct vfio_device_info *device_info) > + __rte_unused struct vfio_device_info *device_info > + __rte_unused rte_uuid_t vf_token) > { > return -1; > } > -- > 2.26.0 Acked-by: Vamsi Attunuru