Re: [dpdk-dev] virtio pmd failed in pci probing

[Vincent Li <vincent.mc.li@gmail.com>]

sorry for the noise, it turned out I need to add the virtio pmd in the
mTCP app Makefile as below

LIBS += -m64 -g -O3 -pthread -lrt -march=native -Wl,-export-dynamic
${MTCP_FLD}/lib/libmtcp.a -L../../dpdk/lib -Wl,-lnuma -Wl,-lmtcp
-Wl,-lpthread -Wl,-lrt -Wl,-ldl -Wl,--whole-archive
-Wl,-lrte_distributor -Wl,-lrte_kni -Wl,-lrte_pipeline -Wl,-lrte_table
-Wl,-lrte_port -Wl,-lrte_timer -Wl,-lrte_hash -Wl,-lrte_lpm
-Wl,-lrte_power -Wl,-lrte_acl -Wl,-lrte_meter -Wl,-lrte_sched -Wl,-lm
-Wl,-lrt -Wl,--start-group -Wl,-lrte_kvargs -Wl,-lrte_mbuf
-Wl,-lrte_ip_frag -Wl,-lethdev -Wl,-lrte_mempool -Wl,-lrte_ring
-Wl,-lrte_eal -Wl,-lrte_cmdline -Wl,-lrte_cfgfile -Wl,-lrte_pmd_bond
-Wl,-lrte_pmd_vmxnet3_uio -Wl,-lrte_pmd_i40e -Wl,-lrte_pmd_ixgbe
-Wl,-lrte_pmd_e1000 -Wl,-lrte_pmd_virtio -Wl,-lrte_pmd_ring -Wl,-lrt
-Wl,-lm -Wl,-ldl -Wl,--end-group -Wl,--no-whole-archive

On Thu, May 12, 2016 at 1:10 PM, Vincent Li <vincent.mc.li@gmail.com> wrote:
> I add a debug log line in dpdk-16.04/lib/librte_ether/rte_ethdev.c
> rte_eth_driver_register
>
> void
> rte_eth_driver_register(struct eth_driver *eth_drv)
> {
>         eth_drv->pci_drv.devinit = rte_eth_dev_init;
>         eth_drv->pci_drv.devuninit = rte_eth_dev_uninit;
>         rte_eal_pci_register(&eth_drv->pci_drv);
>         RTE_LOG(DEBUG, EAL, "  register pmd driver %s\n",
> eth_drv->pci_drv.name);
> }
>
> then run the mTCP app, rte_virtio_pmd is missing:
>
> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> unreliable clock cycles !
> EAL: Master lcore 0 is ready (tid=6ec31900;cpuset=[0])
> EAL:   register pmd driver rte_vmxnet3_pmd
> EAL:   register pmd driver rte_i40e_pmd
> EAL:   register pmd driver rte_i40evf_pmd
> EAL:   register pmd driver rte_ixgbe_pmd
> EAL:   register pmd driver rte_ixgbevf_pmd
> EAL:   register pmd driver rte_igb_pmd
> EAL:   register pmd driver rte_igbvf_pmd
> EAL:   register pmd driver rte_em_pmd
> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_vmxnet3_pmd
> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_i40e_pmd
> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_i40evf_pmd
> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_ixgbe_pmd
> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_ixgbevf_pmd
> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_igb_pmd
> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_igbvf_pmd
> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_em_pmd
> pci_probe_all_drivers ret: 1 devargs not NULL and whitelisted
> EAL: Error - exiting with code: 1
>   Cause: No Ethernet port!
>
> any clue?
>
>
> On Thu, May 12, 2016 at 9:34 AM, Vincent Li <vincent.mc.li@gmail.com> wrote:
>> Hi
>>
>> I am testing mTCP https://github.com/eunyoung14/mtcp  with dpdk-16.04
>> on KVM guest and it appears the virtio pmd fail to load, detail info
>> below:
>>
>>
>> # ./tools/dpdk_nic_bind.py --status
>>
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:00:07.0 'Virtio network device' drv=igb_uio unused=
>>
>> Network devices using kernel driver
>> ===================================
>> 0000:00:03.0 'Virtio network device' if= drv=virtio-pci unused=igb_uio
>> 0000:00:08.0 'Virtio network device' if= drv=virtio-pci unused=igb_uio
>>
>> in mtcp/src/io_module.c, I have (I hard code the whiltelist to make it
>> easy to test for me):
>>
>>
>>
>>                sprintf(whitelist, "%s", "0000:00:07.0");
>>                /* initialize the rte env first, what a waste of
>> implementation effort!  */
>>                 char *argv[] = {"",
>>                                 "-c",
>>                                 cpumaskbuf,
>>                                 "-w",
>>                                 whitelist,
>>                                 "-n",
>>                                 mem_channels,
>>                                 "--proc-type=auto",
>>                                 ""
>>                 };
>>                 const int argc = 8;
>>
>>                 /*
>>                  * re-set getopt extern variable optind.
>>                  * this issue was a bitch to debug
>>                  * rte_eal_init() internally uses getopt() syscall
>>                  * mtcp applications that also use an `external' getopt
>>                  * will cause a violent crash if optind is not reset to zero
>>                  * prior to calling the func below...
>>                  * see man getopt(3) for more details
>>                  */
>>                 optind = 0;
>>
>>                 /* initialize the dpdk eal env */
>>                 ret = rte_eal_init(argc, argv);
>>                 if (ret < 0)
>>                         rte_exit(EXIT_FAILURE, "Invalid EAL args!\n");
>>                 /* give me the count of 'detected' ethernet ports */
>>                 num_devices = rte_eth_dev_count();
>>                 if (num_devices == 0) {
>>                         rte_exit(EXIT_FAILURE, "No Ethernet port!\n");
>>                 }
>>
>> in dpdk-16.04/lib/librte_eal/common/eal_common_pci.c
>> rte_eal_pci_probe_one_driver, I added debug line below:
>>
>> /*
>>  * If vendor/device ID match, call the devinit() function of the
>>  * driver.
>>  */
>> static int
>> rte_eal_pci_probe_one_driver(struct rte_pci_driver *dr, struct
>> rte_pci_device *dev)
>> {
>>         int ret;
>>         const struct rte_pci_id *id_table;
>>
>>         RTE_LOG(DEBUG, EAL, "  dev->id.vendor_id:dev->id.device_id
>> dr->name %x:%x %s\n", dev->id.vendor_id,
>>                                 dev->id.device_id, dr->name);
>>
>>
>> when I run mTCP app as below, it says "No Ethernet port!", please note
>> that the debug line did not show rte_virtio_pmd, why? is it because
>> virtio pmd lack of vendor_id/device_id implementation?
>>
>>
>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
>> unreliable clock cycles !
>> EAL: Master lcore 0 is ready (tid=dca0e900;cpuset=[0])
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_vmxnet3_pmd
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_i40e_pmd
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_i40evf_pmd
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_ixgbe_pmd
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_ixgbevf_pmd
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_igb_pmd
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_igbvf_pmd
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_em_pmd
>> pci_probe_all_drivers ret: 1 devargs not NULL and whitelisted
>> EAL: Error - exiting with code: 1
>>   Cause: No Ethernet port!
>>
>>
>> dpdk testpmd app detects the ethernet port fine, here is the testpmd
>> output (Note here the debug line shows rte_virtio_pmd):
>>
>> ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xf -n 4  -w 0000:00:07.0 -- -i
>>
>> EAL: Master lcore 0 is ready (tid=e9c98900;cpuset=[0])
>> EAL: lcore 3 is ready (tid=a69f6700;cpuset=[3])
>> EAL: lcore 2 is ready (tid=a71f7700;cpuset=[2])
>> EAL: lcore 1 is ready (tid=a79f8700;cpuset=[1])
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000 rte_vmxnet3_pmd
>> EAL:   dev->id.vendor_id:dev->id.device_id dr->name 1af4:1000
>> rte_virtio_pmd
>> EAL: PCI device 0000:00:07.0 on NUMA socket -1
>> EAL:   probe driver: 1af4:1000 rte_virtio_pmd
>> EAL:   PCI memory mapped at 0x7fabe8a00000
>>
>>
>> I am able to workaround this by compiling dpdk as shared library for
>> mTCP and use '-d' to load librte_pmd_virtio.so explicitly.
>>
>> it looks to me there isn't much difference between how testpmd and
>> mTCP invokes dpdk, it is straightforward rte_eal_init(argc, argv) and
>> rte_eth_dev_count()
>>
>> is there any potential bug in the implementation of virtio pmd in pci probing ?
>>
>> Thanks!