From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from col0-omc4-s3.col0.hotmail.com (col0-omc4-s3.col0.hotmail.com [65.55.34.205]) by dpdk.org (Postfix) with ESMTP id A0A9F58DD for ; Thu, 27 Mar 2014 23:05:17 +0100 (CET) Received: from COL131-DS5 ([65.55.34.201]) by col0-omc4-s3.col0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Thu, 27 Mar 2014 15:06:50 -0700 X-TMN: [e9Puzel9/qrxewoIcEk+v89L+PpfI58Z] X-Originating-Email: [fredhps10@hotmail.com] Message-ID: From: Fred Pedrisa To: "'Carew, Alan'" , "'Masaru Oki'" References: <035401cf49f9$2b410700$81c31500$@intel.com> In-Reply-To: Date: Thu, 27 Mar 2014 19:06:55 -0300 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0368_01CF49EF.B8FEEE40" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQFyzF3+QaTJhJhcLinoDlvHNu2iOwKCrqacm5pjSmA= Content-Language: pt-br X-OriginalArrivalTime: 27 Mar 2014 22:06:50.0315 (UTC) FILETIME=[DA8189B0:01CF4A08] Cc: dev@dpdk.org Subject: [dpdk-dev] RES: RES: RES: RES: hw.nic_uio.bdfs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Mar 2014 22:05:18 -0000 ------=_NextPart_000_0368_01CF49EF.B8FEEE40 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi ! I've attached my contribution (the fixed source) by changing the = behavior the way Alan wanted :) It is working and if you like to use it, this would be cool. Sincerely, Fred Pedrisa -----Mensagem original----- De: Fred Pedrisa [mailto:fredhps10@hotmail.com]=20 Enviada em: quinta-feira, 27 de mar=E7o de 2014 17:28 Para: 'Carew, Alan'; 'Masaru Oki' Cc: 'dev@dpdk.org' Assunto: RES: [dpdk-dev] RES: RES: RES: hw.nic_uio.bdfs Hello, It just requires a small code change :), and it can work in the expected way. So you mean bdfs is to 'select' the only wanted devices yes ? May I = change my code proposition for you ? Sincerely, Fred Pedrisa -----Mensagem original----- De: Carew, Alan [mailto:alan.carew@intel.com] Enviada em: quinta-feira, = 27 de mar=E7o de 2014 17:12 Para: Fred Pedrisa; Masaru Oki Cc: dev@dpdk.org Assunto: [dpdk-dev] RES: RES: RES: hw.nic_uio.bdfs Hi folks, Just to clarify, the purpose of hw.nic_uio.bdfs is to remove NICs from drivers not owned by nic_uio and add them to a pool of unbound devices. As with the Linux equivalent driver(igb_uio), nic_uio will then attempt = to bind any unbound NICs and make them available for DPDK. However, the Linux OS also has the ability to selectively bind/unbind devices via sysfs, this is something we lack on FreeBSD. I can see where it would be a problem when only a subset of unbound = devices is required for nic_uio and currently the only option would be to ensure = the original driver is loaded first. The change below changes the purpose of hw.nic_uio.bdfs, i.e. blacklist devices so that they are not used by nic_uio. A better approach would be to make the variable an exclusive list of = devices to be used, this would then require all users to specify the exact = devices to be used. Thanks, Alan -----Mensagem original----- De: dev [mailto:dev-bounces at dpdk.org] Em nome de Fred Pedrisa Enviada = em: quarta-feira, 26 de mar=E7o de 2014 04:22 Para: 'Masaru Oki' Cc: dev at dpdk.org Assunto: [dpdk-dev] RES: RES: hw.nic_uio.bdfs Hello, Here is my fix for probe code : static int nic_uio_probe (device_t dev) { int i, len; char *remaining; long bus =3D 0, device =3D 0, function =3D 0; remaining =3D bdf_str; len =3D strlen(remaining); for (i =3D 0; remaining && len >=3D 5 && i < len;i+=3D6) { if ( remaining[i + 1] =3D=3D ':' && remaining[i + 3] = =3D=3D ':' ) { bus =3D strtol(&remaining[i + 0],NULL,0); device =3D strtol(&remaining[i + 2],NULL,0); function =3D strtol(&remaining[i + 4],NULL,0); if (dev !=3D NULL) { if (pci_get_bus(dev) =3D=3D bus && pci_get_slot(dev) =3D=3D device && pci_get_function(dev) =3D=3D = function) { printf("nic_uio: success = blocking probe of : %ld:%ld:%ld!\n", bus, device, function); return (ENXIO); } } } } for (i =3D 0; i < NUM_DEVICES; i++) if (pci_get_vendor(dev) =3D=3D devices[i].vend && pci_get_device(dev) =3D=3D devices[i].dev) { device_set_desc(dev, "Intel(R) DPDK PCI = Device"); return (BUS_PROBE_SPECIFIC); } return (ENXIO); } Now it is working as intended ;) -----Mensagem original----- De: dev [mailto:dev-bounces at dpdk.org] Em nome de Fred Pedrisa Enviada = em: quarta-feira, 26 de mar=E7o de 2014 04:16 Para: 'Masaru Oki' Cc: dev at dpdk.org Assunto: [dpdk-dev] RES: hw.nic_uio.bdfs Hello, =20 Yes, I am writing a fix for this too ;) =20 De: Masaru Oki [mailto:m-oki at stratosphere.co.jp] Enviada em: quarta-feira, 26 de mar=E7o de 2014 04:08 Para: Fred Pedrisa Cc: dev at dpdk.org Assunto: Re: [dpdk-dev] hw.nic_uio.bdfs =20 > By default nic_uio takes all the NICs for itself Yes. I think nic_uio_probe should check hw.nic_uio.bdfs. =20 =20 2014-03-26 15:49 GMT+09:00 Fred Pedrisa : Hello, =20 By default nic_uio takes all the NICs for itself =20 So in theory, you needed an option to reserve some NIC ports to your = system, without DPDK taking it for itself =20 De: Masaru Oki [mailto:m-oki at stratosphere.co.jp] Enviada em: quarta-feira, 26 de mar=E7o de 2014 03:43 Para: Fred Pedrisa Cc: dev at dpdk.org Assunto: Re: [dpdk-dev] hw.nic_uio.bdfs =20 avoid??? want you hw.nic_uio.avoid_bdfs? nic_uio behavior I guessed 1. detach kernel driver specified by hw.nic_uio.bdfs 2. attach nic_uio driver for all NICs not attached. but 2. is not correct, I think. =20 =20 2014-03-26 15:20 GMT+09:00 Fred Pedrisa : Hello, =20 You did not understand the purpose of that parameter, it is made to = 'avoid' nic_uio from pursuing the wanted NICs... so they are free to be used in = the system :) =20 Right now the code to handle it is wrong and I am trying to fix it = myself. =20 De: Masaru Oki [mailto:m-oki at stratosphere.co.jp] Enviada em: quarta-feira, 26 de mar=E7o de 2014 03:16 Para: Fred Pedrisa Cc: dev at dpdk.org Assunto: Re: [dpdk-dev] hw.nic_uio.bdfs =20 Hi, I tried with Intel version 1.6.0 and FreeBSD 9.2-RELEASE on VMware = Player. kldload nic_uio by hand, works fine. But kldunload nic_uio only detach uio driver, don't re-attach kernel = driver. [oki@ ~]$ cat /boot/loader.conf ############################################################## ### User settings ########################################## ############################################################## hw.contigmem.num_buffers=3D64 hw.contigmem.buffer_size=3D2097152 hw.nic_uio.bdfs=3D"2:5:0,2:6:0" contigmem_load=3D"YES" #nic_uio_load=3D"YES" [oki@ ~]$ pciconf -l | egrep '(em|uio)' em0 at pci0:2:1:0: class=3D0x020000 card=3D0x075015ad chip=3D0x100f8086 = rev=3D0x01 hdr=3D0x00 em1 at pci0:2:5:0: class=3D0x020000 card=3D0x075015ad chip=3D0x100f8086 = rev=3D0x01 hdr=3D0x00 em2 at pci0:2:6:0: class=3D0x020000 card=3D0x075015ad chip=3D0x100f8086 = rev=3D0x01 hdr=3D0x00 [oki@ ~]$ kenv hw.nic_uio.bdfs 2:5:0,2:6:0 [oki@ ~]$ sudo kldload nic_uio Password: [oki@ ~]$ pciconf -l | egrep '(em|uio)' em0 at pci0:2:1:0: class=3D0x020000 card=3D0x075015ad chip=3D0x100f8086 = rev=3D0x01 hdr=3D0x00 nic_uio0 at pci0:2:5:0: class=3D0x020000 card=3D0x075015ad = chip=3D0x100f8086 rev=3D0x01 hdr=3D0x00 nic_uio1 at pci0:2:6:0: class=3D0x020000 card=3D0x075015ad = chip=3D0x100f8086 rev=3D0x01 hdr=3D0x00 [oki@ ~]$ sudo kldunload nic_uio [oki@ ~]$ pciconf -l | egrep '(em|uio)' em0 at pci0:2:1:0: class=3D0x020000 card=3D0x075015ad chip=3D0x100f8086 = rev=3D0x01 hdr=3D0x00 [oki@ ~]$ =20 2014-03-26 14:35 GMT+09:00 Fred Pedrisa : Hi, guys. This variable is not working as intended for FreeBSD :( It does not dettach nic_uio from the wanted ports :/ =20 =20 ------=_NextPart_000_0368_01CF49EF.B8FEEE40 Content-Type: text/plain; name="nic_uio.c" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="nic_uio.c" /* -=0A= * BSD LICENSE=0A= * =0A= * Copyright(c) 2010-2014 Intel Corporation. All rights reserved.=0A= * All rights reserved.=0A= * =0A= * Redistribution and use in source and binary forms, with or without=0A= * modification, are permitted provided that the following conditions=0A= * are met:=0A= * =0A= * * Redistributions of source code must retain the above copyright=0A= * notice, this list of conditions and the following disclaimer.=0A= * * Redistributions in binary form must reproduce the above = copyright=0A= * notice, this list of conditions and the following disclaimer in=0A= * the documentation and/or other materials provided with the=0A= * distribution.=0A= * * Neither the name of Intel Corporation nor the names of its=0A= * contributors may be used to endorse or promote products derived=0A= * from this software without specific prior written permission.=0A= * =0A= * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS=0A= * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT=0A= * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS = FOR=0A= * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT=0A= * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, = INCIDENTAL,=0A= * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT=0A= * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF = USE,=0A= * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON = ANY=0A= * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT=0A= * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE = USE=0A= * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.=0A= */=0A= #include =0A= __FBSDID("$FreeBSD$");=0A= =0A= #include /* defines used in kernel.h */=0A= #include =0A= #include /* types used in module initialization */=0A= #include /* cdevsw struct */=0A= #include /* structs, prototypes for pci bus stuff and = DEVMETHOD */=0A= #include =0A= #include =0A= #include =0A= =0A= #include =0A= #include /* For pci_get macros! */=0A= #include /* The softc holds our per-instance data. */=0A= #include =0A= #include =0A= #include =0A= #include =0A= #include =0A= =0A= #define MAX_BARS (PCIR_MAX_BAR_0 + 1)=0A= =0A= =0A= struct nic_uio_softc {=0A= device_t dev_t;=0A= struct cdev *my_cdev;=0A= int bar_id[MAX_BARS];=0A= struct resource *bar_res[MAX_BARS];=0A= u_long bar_start[MAX_BARS];=0A= u_long bar_size[MAX_BARS];=0A= };=0A= =0A= /* Function prototypes */=0A= static d_open_t nic_uio_open;=0A= static d_close_t nic_uio_close;=0A= static d_mmap_t nic_uio_mmap;=0A= static d_mmap_single_t nic_uio_mmap_single;=0A= static int nic_uio_probe(device_t dev);=0A= static int nic_uio_attach(device_t dev);=0A= static int nic_uio_detach(device_t dev);=0A= static int nic_uio_shutdown(void);=0A= static int nic_uio_modevent(module_t mod, int type, void = *arg);=0A= =0A= static struct cdevsw uio_cdevsw =3D {=0A= .d_name =3D "nic_uio",=0A= .d_version =3D D_VERSION,=0A= .d_open =3D nic_uio_open,=0A= .d_close =3D nic_uio_close,=0A= .d_mmap =3D nic_uio_mmap,=0A= .d_mmap_single =3D nic_uio_mmap_single,=0A= };=0A= =0A= static device_method_t nic_uio_methods[] =3D {=0A= DEVMETHOD(device_probe, nic_uio_probe),=0A= DEVMETHOD(device_attach, nic_uio_attach),=0A= DEVMETHOD(device_detach, nic_uio_detach),=0A= DEVMETHOD_END=0A= };=0A= =0A= struct device {=0A= int vend;=0A= int dev;=0A= };=0A= =0A= struct pci_bdf {=0A= uint32_t bus;=0A= uint32_t devid;=0A= uint32_t function;=0A= };=0A= =0A= =0A= #define RTE_PCI_DEV_ID_DECL_EM(vend, dev) {vend, dev},=0A= #define RTE_PCI_DEV_ID_DECL_IGB(vend, dev) {vend, dev},=0A= #define RTE_PCI_DEV_ID_DECL_IGBVF(vend, dev) {vend, dev},=0A= #define RTE_PCI_DEV_ID_DECL_IXGBE(vend, dev) {vend, dev},=0A= #define RTE_PCI_DEV_ID_DECL_IXGBEVF(vend, dev) {vend, dev},=0A= #define RTE_PCI_DEV_ID_DECL_VIRTIO(vend, dev) {vend, dev},=0A= =0A= const struct device devices[] =3D {=0A= #include =0A= };=0A= #define NUM_DEVICES (sizeof(devices)/sizeof(devices[0]))=0A= =0A= =0A= static devclass_t nic_uio_devclass;=0A= =0A= DEFINE_CLASS_0(nic_uio, nic_uio_driver, nic_uio_methods, sizeof(struct = nic_uio_softc));=0A= DRIVER_MODULE(nic_uio, pci, nic_uio_driver, nic_uio_devclass, = nic_uio_modevent, 0);=0A= =0A= static char bdf_str[1024];=0A= =0A= static int=0A= nic_uio_mmap(struct cdev *cdev, vm_ooffset_t offset, vm_paddr_t *paddr,=0A= int prot, vm_memattr_t *memattr)=0A= {=0A= *paddr =3D offset;=0A= return (0);=0A= }=0A= =0A= static int=0A= nic_uio_mmap_single(struct cdev *cdev, vm_ooffset_t *offset, vm_size_t = size,=0A= struct vm_object **obj, int nprot)=0A= {=0A= /*=0A= * The BAR index is encoded in the offset. Divide the offset by=0A= * PAGE_SIZE to get the index of the bar requested by the user=0A= * app.=0A= */=0A= unsigned bar =3D *offset/PAGE_SIZE;=0A= struct nic_uio_softc *sc =3D cdev->si_drv1;=0A= =0A= if (bar >=3D MAX_BARS)=0A= return EINVAL;=0A= =0A= if (sc->bar_res[bar] =3D=3D NULL) {=0A= sc->bar_id[bar] =3D PCIR_BAR(bar);=0A= =0A= if (PCI_BAR_IO(pci_read_config(sc->dev_t, sc->bar_id[bar], 4)))=0A= sc->bar_res[bar] =3D bus_alloc_resource_any(sc->dev_t, SYS_RES_IOPORT,=0A= &sc->bar_id[bar], RF_ACTIVE);=0A= else=0A= sc->bar_res[bar] =3D bus_alloc_resource_any(sc->dev_t, SYS_RES_MEMORY,=0A= &sc->bar_id[bar], RF_ACTIVE);=0A= }=0A= if (sc->bar_res[bar] =3D=3D NULL)=0A= return ENXIO;=0A= =0A= sc->bar_start[bar] =3D rman_get_start(sc->bar_res[bar]);=0A= sc->bar_size[bar] =3D rman_get_size(sc->bar_res[bar]);=0A= =0A= device_printf(sc->dev_t, "Bar %u @ %lx, size %lx\n", bar,=0A= sc->bar_start[bar], sc->bar_size[bar]);=0A= =0A= *offset =3D sc->bar_start[bar];=0A= *obj =3D vm_pager_allocate(OBJT_DEVICE, cdev, size, nprot, *offset,=0A= curthread->td_ucred);=0A= return 0;=0A= }=0A= =0A= =0A= int=0A= nic_uio_open(struct cdev *dev, int oflags, int devtype, d_thread_t *td)=0A= {=0A= return 0;=0A= }=0A= =0A= int=0A= nic_uio_close(struct cdev *dev, int fflag, int devtype, d_thread_t *td)=0A= {=0A= return 0;=0A= }=0A= =0A= static int=0A= nic_uio_probe (device_t dev)=0A= {=0A= int i, len;=0A= char *remaining;=0A= long bus =3D 0, device =3D 0, function =3D 0;=0A= remaining =3D bdf_str;=0A= len =3D strlen(remaining);=0A= =0A= for (i =3D 0; remaining && len >=3D 5 && i < len;i+=3D6) {=0A= if ( remaining[i + 1] =3D=3D ':' && remaining[i + 3] = =3D=3D ':' ) {=0A= bus =3D strtol(&remaining[i + 0],NULL,0);=0A= device =3D strtol(&remaining[i + 2],NULL,0);=0A= function =3D strtol(&remaining[i + 4],NULL,0);=0A= if (dev !=3D NULL) {=0A= if (pci_get_bus(dev) =3D=3D bus && = pci_get_slot(dev) =3D=3D device && pci_get_function(dev) =3D=3D = function) {=0A= for (i =3D 0; i < NUM_DEVICES; i++) {=0A= if (pci_get_vendor(dev) =3D=3D devices[i].vend && = pci_get_device(dev) =3D=3D devices[i].dev) {=0A= printf("nic_uio: success probing the wanted device : = %ld:%ld:%ld!\n", bus, device, function);=0A= device_set_desc(dev, "Intel(R) DPDK PCI = Device");=0A= return (BUS_PROBE_SPECIFIC);=0A= }=0A= }=0A= }=0A= }=0A= }=0A= }=0A= =0A= return (ENXIO);=0A= }=0A= =0A= static int=0A= nic_uio_attach(device_t dev)=0A= {=0A= int i;=0A= struct nic_uio_softc *sc;=0A= =0A= sc =3D device_get_softc(dev);=0A= sc->dev_t =3D dev;=0A= sc->my_cdev =3D make_dev(&uio_cdevsw, device_get_unit(dev),=0A= UID_ROOT, GID_WHEEL, 0600, "uio@pci:%u:%u:%u",=0A= pci_get_bus(dev), pci_get_slot(dev), pci_get_function(dev));=0A= if (sc->my_cdev =3D=3D NULL)=0A= return ENXIO;=0A= sc->my_cdev->si_drv1 =3D sc;=0A= =0A= for (i =3D 0; i < MAX_BARS; i++)=0A= sc->bar_res[i] =3D NULL;=0A= =0A= pci_enable_busmaster(dev);=0A= =0A= return 0;=0A= }=0A= =0A= static int=0A= nic_uio_detach(device_t dev)=0A= {=0A= int i;=0A= struct nic_uio_softc *sc;=0A= sc =3D device_get_softc(dev);=0A= =0A= for (i =3D 0; i < MAX_BARS; i++)=0A= if (sc->bar_res[i] !=3D NULL) {=0A= =0A= if (PCI_BAR_IO(pci_read_config(dev, sc->bar_id[i], 4)))=0A= bus_release_resource(dev, SYS_RES_IOPORT, sc->bar_id[i],=0A= sc->bar_res[i]);=0A= else =0A= bus_release_resource(dev, SYS_RES_MEMORY, sc->bar_id[i],=0A= sc->bar_res[i]);=0A= }=0A= =0A= if (sc->my_cdev !=3D NULL)=0A= destroy_dev(sc->my_cdev);=0A= return 0;=0A= }=0A= =0A= static void=0A= nic_uio_load(void)=0A= {=0A= char *remaining;=0A= long bus =3D 0, device =3D 0, function =3D 0;=0A= int i, j, len;=0A= device_t dev;=0A= =0A= memset(bdf_str, 0, sizeof(bdf_str));=0A= TUNABLE_STR_FETCH("hw.nic_uio.bdfs", bdf_str, sizeof(bdf_str));=0A= remaining =3D bdf_str;=0A= len =3D strlen(remaining);=0A= =0A= for (i =3D 0; remaining && len >=3D 5 && i < len;i+=3D6) {=0A= if ( remaining[i + 1] =3D=3D ':' && remaining[i + 3] = =3D=3D ':' ) {=0A= bus =3D strtol(&remaining[i + 0],NULL,0);=0A= device =3D strtol(&remaining[i + 2],NULL,0);=0A= function =3D strtol(&remaining[i + 4],NULL,0);=0A= =0A= printf("nic_uio: trying to dettach %ld:%ld:%ld...\n", bus, device, = function);=0A= dev =3D pci_find_bsf(bus, device, function);=0A= if (dev !=3D NULL) {=0A= for (j =3D 0; j < NUM_DEVICES; j++) {=0A= if (pci_get_vendor(dev) =3D=3D = devices[j].vend && pci_get_device(dev) =3D=3D devices[j].dev) {=0A= printf("nic_uio: success in dettachment of : %ld:%ld:%ld!\n", = bus, device, function);=0A= = device_detach(dev); }=0A= }=0A= }=0A= }=0A= }=0A= }=0A= =0A= static void=0A= nic_uio_unload(void)=0A= {=0A= }=0A= =0A= static int=0A= nic_uio_shutdown(void)=0A= {=0A= return (0);=0A= }=0A= =0A= static int=0A= nic_uio_modevent(module_t mod, int type, void *arg)=0A= {=0A= =0A= switch (type) {=0A= case MOD_LOAD:=0A= nic_uio_load();=0A= break;=0A= case MOD_UNLOAD:=0A= nic_uio_unload();=0A= break;=0A= case MOD_SHUTDOWN:=0A= nic_uio_shutdown();=0A= break;=0A= default:=0A= break;=0A= }=0A= =0A= return (0);=0A= }=0A= ------=_NextPart_000_0368_01CF49EF.B8FEEE40--