Re: [dpdk-dev] [PATCH 3/7] eal: move virtual device probing into a bus

[Jerin Jacob <jerin.jacob@caviumnetworks.com>]

On Wed, Feb 15, 2017 at 02:27:47PM +0000, Shreyansh Jain wrote:
> > -----Original Message-----
> > From: Wiles, Keith [mailto:keith.wiles@intel.com]
> > Sent: Wednesday, February 15, 2017 7:53 PM
> > To: Shreyansh Jain <shreyansh.jain@nxp.com>
> > Cc: Jan Blunck <jblunck@infradead.org>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH 3/7] eal: move virtual device probing into a
> > bus
> > 
> > 
> > > On Feb 15, 2017, at 8:15 AM, Shreyansh Jain <shreyansh.jain@nxp.com> wrote:
> > >
> > > On Wednesday 15 February 2017 07:41 PM, Shreyansh Jain wrote:
> > >> On Wednesday 15 February 2017 03:32 PM, Jan Blunck wrote:
> > >>> This is a refactoring of the virtual device probing which moves into into
> > >>> a proper bus structure.
> > >>>
> > >>> Signed-off-by: Jan Blunck <jblunck@infradead.org>
> > >>> ---
> > >>> lib/librte_eal/common/eal_common_dev.c  | 22 -----------------
> > >>> lib/librte_eal/common/eal_common_vdev.c | 44
> > >>> +++++++++++++++++++++++++++++++++
> > >>> 2 files changed, 44 insertions(+), 22 deletions(-)
> > >>>
> > >>
> > >> [...]
> > >>
> > >>>
> > >>> diff --git a/lib/librte_eal/common/eal_common_vdev.c
> > >>> b/lib/librte_eal/common/eal_common_vdev.c
> > >>> index 7d6e54f..523a3d6 100644
> > >>> --- a/lib/librte_eal/common/eal_common_vdev.c
> > >>> +++ b/lib/librte_eal/common/eal_common_vdev.c
> > >>> @@ -37,8 +37,10 @@
> > >>> #include <stdint.h>
> > >>> #include <sys/queue.h>
> > >>>
> > >> [...]
> > >>
> > >>> +
> > >>> +static struct rte_bus rte_vdev_bus = {
> > >>> +    .scan = vdev_scan,
> > >>> +    .probe = vdev_probe,
> > >>> +};
> > >>> +
> > >>> +RTE_REGISTER_BUS_LATE(virtual, rte_vdev_bus);
> > >>>
> > >>
> > >> Does it matter if VDEV buses are registered before or after other
> > >> buses? Either way, the callbacks would be called in the order specified
> > >> in EAL.
> > >>
> > >>
> > >
> > > Just ignore this comment - I am misunderstood something.
> > >
> > > But another question: Is there specific reason VDEV should be
> > registered/scanned *after* other devices? Is there some specific problem if
> > we do otherwise? (I think this is should be done, but I don't have a specific
> > reason).
> > 
> > Does the bonding driver which uses physical devices need to be registered
> > after physical ones? In Pktgen I noticed the vdev after the physical ports
> > and I could not blacklist them as the bonding driver needed them, which
> > caused the bonding ports to have a greater port number. In the case of pktgen
> > the bonding ports were up around 8 or 10 and caused the display to not show
> > the bonding ports. This is really just a usability problem for the developer
> > using Pktgen. I would like to see the vdev devices first, but as long as the
> > drivers (like bonding) are fine with them being first.
> 
> Ah, now I remember - there was a patch from Jerin for this.
> Probably he is the best person to comment here.
> (I don't have much insight here).

commit f4ce209a8ce5f416b61c76cee773bc54749e2048
Author: Jerin Jacob <jerin.jacob@caviumnetworks.com>
Date:   Sun Nov 20 13:30:50 2016 +0530

    eal: postpone vdev initialization

    Some platform like octeontx may use pci and
    vdev based combined device to represent a logical
    dpdk functional device.In such case, postponing the
    vdev initialization after pci device
    initialization will provide the better view of
    the pci device resources in the system in
    vdev's probe function, and it allows better
    functional subsystem registration in vdev probe
    function.

    As a bonus, This patch fixes a bond device
    initialization use case.

    example command to reproduce the issue:
    ./testpmd -c 0x2  --vdev 'eth_bond0,mode=0,
    slave=0000:02:00.0,slave=0000:03:00.0' --
    --port-topology=chained

    root cause:
    In existing case(vdev initialization and then pci
    initialization), creates three Ethernet ports with
    following port ids
    0 - Bond device
    1 - PCI device 0
    2 - PCI devive 1

    Since testpmd, calls the configure/start on all the ports on
    start up,it will translate to following illegal setup sequence

    1)bond device configure/start
    1.1) pci device0 stop/configure/start
    1.2) pci device1 stop/configure/start
    2)pci device 0 configure(illegal setup case,
    as device in start state)

    The fix changes the initialization sequence and
    allow initialization in following valid setup order
    1) pcie device 0 configure/start
    2) pcie device 1 configure/start
    3) bond device 2 configure/start
    3.1) pcie device 0/stop/configure/start
    3.2) pcie device 1/stop/configure/start