Re: [dpdk-dev] [PATCH v4 00/11] Port Hotplug Framework

[Tetsuya Mukawa <mukawa@igel.co.jp>]

On 2015/01/27 14:50, Qiu, Michael wrote:
> On 1/27/2015 1:02 PM, Tetsuya Mukawa wrote:
>> On 2015/01/27 12:00, Qiu, Michael wrote:
>>> On 1/19/2015 6:42 PM, Tetsuya Mukawa wrote:
>>>> This patch series adds a dynamic port hotplug framework to DPDK.
>>>> With the patches, DPDK apps can attach or detach ports at runtime.
>>>>
>>>> The basic concept of the port hotplug is like followings.
>>>> - DPDK apps must have responsibility to manage ports.
>>>>   DPDK apps only know which ports are attached or detached at the moment.
>>>>   The port hotplug framework is implemented to allow DPDK apps to manage ports.
>>>>   For example, when DPDK apps call port attach function, attached port number
>>>>   will be returned. Also DPDK apps can detach port by port number.
>>>> - Kernel support is needed for attaching or detaching physical device ports.
>>>>   To attach new device, the device will be recognized by kernel at first and
>>>>   controlled by kernel driver. Then user can bind the device to igb_uio
>>> Here does it really need native kernel driver here? As it will be
>>> controlled by igb_uio.
>>> I think even if the device has no kernel driver is also OK.
>> Thanks for correcting. Yes, it should be.
>> How about following.
>>
>> - Kernel support is needed for attaching or detaching physical device ports.
>>   To attach a new device, the device will be recognized by kernel PCI
>> hotplug feature at first.
> No, here should not explain as "kernel PCI hotplug feature" which is
> stand for removing or adding a PCI device from system level, it is
> devices related not driver.
>
> What about:
>
> - Kernel support is needed for attaching or detaching physical device
> ports. To attach a new physical device port, the device will be
> recognized by userspace directly I/O framework in kernel at first.

Thanks. I will replace like above.

Thanks,
Tetsuya

> Thanks,
> Michael
>>   Then user can bind the device to igb_uio.
>>
>>> Also I have finished initial patch of passthrough driver flag in
>>> "struct rte_pci_device"
>>>
>>> I will send to you after I do some basic test on that, then I will send to
>>> you, and you can give some comments on that.
>> I appreciate for your implementing.
>>
>> Thanks,
>> Tetsuya
>>
>>
>>> Thanks,
>>> Michael
>>>
>>>>   by 'dpdk_nic_bind.py'. Finally, DPDK apps can call the port hotplug
>>>>   functions to attach ports.
>>>>   For detaching, steps are vice versa.
>>>> - Before detach ports, ports must be stopped and closed.
>>>>   DPDK application must call rte_eth_dev_stop() and rte_eth_dev_close() before
>>>>   detaching ports. These function will call finalization codes of PMDs.
>>>>   But so far, no PMD frees all resources allocated by initialization.
>>>>   It means PMDs are needed to be fixed to support the port hotplug.
>>>>   'RTE_PCI_DRV_DETACHABLE' is a new flag indicating a PMD supports detaching.
>>>>   Without this flag, detaching will be failed.
>>>> - Mustn't affect legacy DPDK apps.
>>>>   No DPDK EAL behavior is changed, if the port hotplug functions are't called.
>>>>   So all legacy DPDK apps can still work without modifications.
>>>>
>>>> And a few limitations.
>>>> - The port hotplug functions are not thread safe.
>>>>   DPDK apps should handle it.
>>>> - Only support Linux and igb_uio so far.
>>>>   BSD and VFIO is not supported. I will send VFIO patches at least, but I don't
>>>>   have a plan to submit BSD patch so far.
>>>>
>>>>
>>>> Here is port hotplug APIs.
>>>> -------------------------------------------------------------------------------
>>>> /**
>>>>  * Attach a new device.
>>>>  *
>>>>  * @param devargs
>>>>  *   A pointer to a strings array describing the new device
>>>>  *   to be attached. The strings should be a pci address like
>>>>  *   '0000:01:00.0' or virtual device name like 'eth_pcap0'.
>>>>  * @param port_id
>>>>  *  A pointer to a port identifier actually attached.
>>>>  * @return
>>>>  *  0 on success and port_id is filled, negative on error
>>>>  */
>>>> int rte_eal_dev_attach(const char *devargs, uint8_t *port_id);
>>>>
>>>> /**
>>>>  * Detach a device.
>>>>  *
>>>>  * @param port_id
>>>>  *   The port identifier of the device to detach.
>>>>  * @param addr
>>>>  *  A pointer to a device name actually detached.
>>>>  * @return
>>>>  *  0 on success and devname is filled, negative on error
>>>>  */
>>>> int rte_eal_dev_detach(uint8_t port_id, char *devname);
>>>> -------------------------------------------------------------------------------
>>>>
>>>> This patch series are for DPDK EAL. To use port hotplug function by DPDK apps,
>>>> each PMD should be fixed to support 'RTE_PCI_DRV_DETACHABLE' flag. Please check
>>>> a patch for pcap PMD.
>>>>
>>>> Also please check testpmd patch. It will show you how to fix your legacy
>>>> applications to support port hotplug feature.
>>>>
>>>>
>>>> PATCH v4 changes
>>>>  - Merge patches to review easier.
>>>>  - Fix indent of 'if' statement.
>>>>  - Fix calculation method of eal_compare_pci_addr().
>>>>  - Fix header file declaration.
>>>>  - Add header file to determine if hotplug can be enabled.
>>>>    (Thanks to Qiu, Michael)
>>>>  - Use braces with 'for' loop.
>>>>  - Add paramerter checking.
>>>>  - Fix sanity check code
>>>>  - Fix comments of rte_eth_dev_type.
>>>>  - Change function names.
>>>>    (Thanks to Iremonger, Bernard)
>>>>
>>>> PATCH v3 changes:
>>>>  - Fix enum definition used in rte_ethdev.c.
>>>>    (Thanks to Zhang, Helin)
>>>>
>>>> PATCH v2 changes:
>>>>  - Replace rte_eal_dev_attach_pdev(), rte_eal_dev_detach_pdev,
>>>>    rte_eal_dev_attach_vdev() and rte_eal_dev_detach_vdev() to
>>>>    rte_eal_dev_attach() and rte_eal_dev_detach().
>>>>  - Add parameter values checking.
>>>>  - Refashion a few functions.
>>>>    (Thanks to Iremonger, Bernard)
>>>>
>>>> PATCH v1 Changes:
>>>>  - Fix error checking code of librte_eth APIs.
>>>>  - Fix issue that port from pcap PMD cannot be detached correctly.
>>>>  - Fix issue that testpmd could hang after forwarding, if attaching and detaching
>>>>    is repeatedly.
>>>>  - Fix if-condition of rte_eth_dev_get_port_by_addr().
>>>>    (Thanks to Mark Enright)
>>>>
>>>> RFC PATCH v2 Changes:
>>>> - remove 'rte_eth_dev_validate_port()', and cleanup codes.
>>>>
>>>>
>>>> Tetsuya Mukawa (11):
>>>>   eal/pci,ethdev: Remove assumption that port will not be detached
>>>>   eal/pci: Consolidate pci address comparison APIs
>>>>   ethdev: Add rte_eth_dev_free to free specified device
>>>>   eal,ethdev: Add a function and function pointers to close ether device
>>>>   ethdev: Add functions that will be used by port hotplug functions
>>>>   eal/linux/pci: Add functions for unmapping igb_uio resources
>>>>   eal/pci: Add a function to remove the entry of devargs list
>>>>   eal/pci: Cleanup pci driver initialization code
>>>>   ethdev: Add one dev_type paramerter to rte_eth_dev_allocate
>>>>   eal/pci: Add rte_eal_dev_attach/detach() functions
>>>>   eal: Enable port hotplug framework in Linux
>>>>
>>>>  app/test/virtual_pmd.c                          |   2 +-
>>>>  config/common_linuxapp                          |   5 +
>>>>  lib/librte_eal/bsdapp/eal/eal_pci.c             |  16 +-
>>>>  lib/librte_eal/common/Makefile                  |   1 +
>>>>  lib/librte_eal/common/eal_common_dev.c          | 273 +++++++++++++++
>>>>  lib/librte_eal/common/eal_common_devargs.c      |  57 ++++
>>>>  lib/librte_eal/common/eal_common_pci.c          |  92 ++++-
>>>>  lib/librte_eal/common/eal_private.h             |  36 ++
>>>>  lib/librte_eal/common/include/rte_dev.h         |  33 ++
>>>>  lib/librte_eal/common/include/rte_dev_hotplug.h |  44 +++
>>>>  lib/librte_eal/common/include/rte_devargs.h     |  18 +
>>>>  lib/librte_eal/common/include/rte_pci.h         |  76 +++++
>>>>  lib/librte_eal/linuxapp/eal/Makefile            |   1 +
>>>>  lib/librte_eal/linuxapp/eal/eal_pci.c           | 136 ++++++--
>>>>  lib/librte_eal/linuxapp/eal/eal_pci_init.h      |   8 +
>>>>  lib/librte_eal/linuxapp/eal/eal_pci_uio.c       |  67 +++-
>>>>  lib/librte_ether/rte_ethdev.c                   | 424 ++++++++++++++++++------
>>>>  lib/librte_ether/rte_ethdev.h                   | 145 +++++++-
>>>>  lib/librte_pmd_af_packet/rte_eth_af_packet.c    |   2 +-
>>>>  lib/librte_pmd_bond/rte_eth_bond_api.c          |   2 +-
>>>>  lib/librte_pmd_pcap/rte_eth_pcap.c              |   2 +-
>>>>  lib/librte_pmd_ring/rte_eth_ring.c              |   2 +-
>>>>  lib/librte_pmd_xenvirt/rte_eth_xenvirt.c        |   2 +-
>>>>  23 files changed, 1283 insertions(+), 161 deletions(-)
>>>>  create mode 100644 lib/librte_eal/common/include/rte_dev_hotplug.h
>>>>
>>