From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lj1-f182.google.com (mail-lj1-f182.google.com [209.85.208.182]) by dpdk.org (Postfix) with ESMTP id ECB5D1B3E8 for ; Wed, 26 Sep 2018 17:44:30 +0200 (CEST) Received: by mail-lj1-f182.google.com with SMTP id y71-v6so9232107lje.9 for ; Wed, 26 Sep 2018 08:44:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=centripetalnetworks.com; s=google; h=mime-version:from:date:message-id:subject:to; bh=6MV6jHRdEok/kon/8Q9FXmB8AU+7n/B8i1Rlv8X/gPw=; b=T3Y3dU5NYLnIRuKvFjO7d/rkuaDz7FmYkL0/p+HhIL2aS9XFilbgR3ubbjMRWeMb/D 2RXMwUJecUOw6B3n8YBS4unXdqMTg/F12nVCydbIYSEq9BfPD57wT4N10LTlxfCzhjuC +VdiYySBueYTSvxCF5KrW4TY+U+cqGhxOVS/U= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=6MV6jHRdEok/kon/8Q9FXmB8AU+7n/B8i1Rlv8X/gPw=; b=LaGMywEXFbixJCAWHc2v4jADXYsEgcC3bhCyI5Eo8TfvFM7N+3hf03sLrEByZNHJX9 Eum66QS4LaSR0sFdedDoFSy9vfyAaZkONpSCTBFCCyG/8S+x7rxeJ2dOYeBfzozkuVLc Ewt6ucs4ruDzydjS/DOGZ7mOlq/D5Y6qQeCQIoYl9aS8bLBKB64h5y5kiz5hLaGLKmLT VEmGK8nFrgL2Wcjp30UkUmSvIwr35ED3ihcjZu++lL8V6mtJmRje8CZkC0LQJtcSSjh/ mNrqArLPhCXSj987EHI9JnAkewzgSHmlJO33c4BxrsB/ejjl9DU5OJhaoKZSdciAU8Gf OWmw== X-Gm-Message-State: ABuFfoj3fjOKnngEy6cOEyZt5crtxMZuvYNbQA2KZYfoPOmJxhICTy47 4zmylGKeOLLjoFZ2G01BOZI2DhbDcjWq3oJUg49WTmHR X-Google-Smtp-Source: ACcGV62MiBXDNTtwclBbbeABQ7Lx+4kxPPE+70nWz/eKOA7E+vVUN4pt4MNe88G318dEg2nHEBTZiv1mm6zp+rNWATw= X-Received: by 2002:a2e:9d45:: with SMTP id y5-v6mr3452026ljj.136.1537976670294; Wed, 26 Sep 2018 08:44:30 -0700 (PDT) MIME-Version: 1.0 From: Paul Sprague Date: Wed, 26 Sep 2018 11:44:19 -0400 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Error when running DPDK in ESXi Centos guest. X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Sep 2018 15:44:31 -0000 Hello! We're trying to run DPDK example apps in a guest machine running Centos 7.5. The host is ESXi version 6.5. I'm building dpdk on the guest machine where I'm trying to run it. I've tried both DPDK versions 18.05 and 18.08. We have created five interfaces on esxi for connection to our guest. One management port and four data ports. We're binding theses four data ports to DPDK. The ports are all VMXNET3 interfaces. They are basically setup like the VMXNET3 interfaces in [ https://doc.dpdk.org/guides/nics/vmxnet3.html], using a vswitch to connect to a physical interface. However note that we do not have any VF interfaces as shown in this document, only VMXNET3 interfaces. Fundamentally, the roadblock we are hitting is that the VMXNET3 interfaces are failing initialization when starting the DPDK example app. Here is what we see: [root@rg-vm ~]# ./dpdk-18.08/examples/packet_ordering/build/packet_ordering -c 0x0e0 -- -p 0xf EAL: Detected 24 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Probing VFIO support... EAL: PCI device 0000:04:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 15ad:7b0 net_vmxnet3 *eth_vmxnet3_dev_init(): Incompatible hardware version: 0* *EAL: Requested device 0000:04:00.0 cannot be used* We see this for all four interfaces that we are trying to bind to DPDK. However, strangely, *sometimes* after a reboot, the first two interfaces initialize correctly. But after that first attempt, all four interfaces then fail the same way. Here are the commands we're using to setup DPDK. modprobe uio insmod ./dpdk-18.08/build/build/kernel/linux/igb_uio/igb_uio.ko ./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 04:00.0 ./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 0c:00.0 ./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 13:00.0 ./dpdk-18.08/usertools/dpdk-devbind.py --bind=igb_uio 1b:00.0 Note that we have also tried using the uio_pci_generic with the same results. We have not been able to get the vfio-pci driver to bind to the VMXNET3 interfaces. I'm not sure it matters, but the physical interfaces on the other side of the vswitch that we're connecting to are: 17:00.0 Ethernet controller: Intel Corporation I350 Gigabit Fiber Network Connection (rev 01) We are running on a Dell R540. Also note that when we run Centos 7.5 with DPDK on this hardware without VMWare, everything works fine. Also if we run in VMWare, but "passthrough" the i350 interfaces to the VM (instead of using vswitch and vmxnet) everything also works fine in that case. I've tried updating the kernel (3.10) to the latests (4.18) but still get the same error. If I try to read the version register (VSSR) (the one that causes this error) in the vmxnet3 pci bar registers (before I bind to DPDK) using ethtool, it looks fine (non zero). I've googled around a lot but can't seem to find much help on this. Any help would be greatly appreciated. Thanks! Paul Sprague Centripetal Networks