DPDK patches and discussions
 help / color / Atom feed
From: Ferruh Yigit <ferruh.yigit@intel.com>
To: Matan Azrad <matan@mellanox.com>,
	"Yigit, Ferruh" <ferruh.yigit@linux.intel.com>,
	"dev@dpdk.org" <dev@dpdk.org>,
	Bernard Iremonger <bernard.iremonger@intel.com>
Cc: Gaetan Rivet <gaetan.rivet@6wind.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	"stable@dpdk.org" <stable@dpdk.org>,
	David Marchand <david.marchand@redhat.com>,
	Jeff Guo <jia.guo@intel.com>, Qi Zhang <qi.z.zhang@intel.com>
Subject: Re: [dpdk-dev] [dpdk-stable] [PATCH 2/2] app/testpmd: fix invalid port detaching
Date: Mon, 3 Feb 2020 15:58:31 +0000
Message-ID: <33a6aa15-28e4-e770-204c-25ca230ca653@intel.com> (raw)
In-Reply-To: <AM0PR0502MB40190732C5655D5425D2BF6DD2090@AM0PR0502MB4019.eurprd05.prod.outlook.com>

On 1/25/2020 6:56 PM, Matan Azrad wrote:
> Hi Ferruh
> 
> From: Ferruh Yigit
>> On 1/23/2020 7:25 PM, Matan Azrad wrote:
>>> Hi
>>>
>>> From: Ferruh Yigit
>>>> On 1/23/2020 3:29 PM, Matan Azrad wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> From: Ferruh Yigit
>>>>>> On 1/23/2020 2:05 PM, Matan Azrad wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> From: Yigit, Ferruh
>>>>>>>> On 11/12/2019 8:47 AM, Matan Azrad wrote:
>>>>>>>>> The port was not validated before detaching.
>>>>>>>>>
>>>>>>>>> Ignore port detach operation when the port is not valid.
>>>>>>>>>
>>>>>>>>> Fixes: f8e5baa2662d ("app/testpmd: check not detaching device
>>>>>>>>> twice")
>>>>>>>>> Cc: thomas@monjalon.net
>>>>>>>>> Cc: stable@dpdk.org
>>>>>>>>>
>>>>>>>>> Signed-off-by: Matan Azrad <matan@mellanox.com>
>>>>>>>>> ---
>>>>>>>>>  app/test-pmd/testpmd.c | 3 +++
>>>>>>>>>  1 file changed, 3 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c
>>>>>>>>> index 4444346..370eefe 100644
>>>>>>>>> --- a/app/test-pmd/testpmd.c
>>>>>>>>> +++ b/app/test-pmd/testpmd.c
>>>>>>>>> @@ -2545,6 +2545,9 @@ struct extmem_param {
>>>>>>>>>
>>>>>>>>>  	printf("Removing a device...\n");
>>>>>>>>>
>>>>>>>>> +	if (port_id_is_invalid(port_id, ENABLED_WARN))
>>>>>>>>> +		return;
>>>>>>>>> +
>>>>>>>>>  	dev = rte_eth_devices[port_id].device;
>>>>>>>>>  	if (dev == NULL) {
>>>>>>>>>  		printf("Device already removed\n");
>>>>>>>>>
>>>>>>>>
>>>>>>>> The patch is already in 19.11 [1] but it is breaking the testpmd
>>>>>>>> hotplug support.
>>>>>>>> Before 'detach_port_device()' called, the port has been stopped
>>>>>>>> and closed [2], which will make port fail from 'port_id_is_invalid()'
>>>>>>>> check and the device removal path never fully called.
>>>>>>>> The implication is, since device not detached, vfio request
>>>>>>>> interrupt keeps triggered continuously and re-starts the detach
>>>>>>>> path, but because of the half cleaned device it fails and app
>>>>>>>> gets stuck with a
>>>>>> continuous log [3].
>>>>>>>>
>>>>>>>> I wonder if the actual hotplug has been tested with this patch,
>>>>>>>> the commit log is not clear about the motivation and implication
>>>>>>>> of the patch, I am not clear why this check is added but I am
>>>>>>>> sending a patch soon to remove it back.
>>>>>>>
>>>>>>> The motivation of this patch was to prevent double detach on same
>>>>>>> port,
>>>>>> so the user cannot call detach of invalid port.
>>>>>>
>>>>>> What is the definition of the 'invalid port', if you mean device
>>>>>> already detached case, in the second call of the function "if (dev
>>>>>> == NULL)" check should prevent it going forward.
>>>>>
>>>>> No, ethdev doesn't zero the device pointer when it release a port.
>>>>
>>>> As far as I can see it does, please see below.
>>>
>>> The code below is problematic because:
>>>
>>> 1. It is very bad that the application changing ethdev structure directly.
>>
>> Where the application is changing the ethdev structure?
> 
> See it in the function we talk on:
> rte_eth_devices[sibling].device = NULL;
> 
> The application shouldn't do it - it should be done only by ethdev lib or by the PMDs.
> 
> Are you agree here?

This is really no fun :(

It is not done by application, I already provided the call trace. This is done
by the path of driver .remove().

> 
>> Application calls the 'rte_dev_remove()' API, which does the job.
> 
> Agree, This function is freeing(rte_free) the rte_device (actually makes the rte_eth_devices[sibling].device pointer dangled) 
> and releases its related resources what makes the device detached.

No it doesn't, I provided full call stack, and showed where the value set to NULL.

> 
>>> 2. The below code run over valid port only, not on invalid port(UNUSED
>> state).
>>>
>>> So, the device pointer will still be valid if the port is invalid.
>>>
>>> All of this shows that this function try to detach only a valid port (probably
>> mainly because it is called by Testpmd detach command).
>>>
>>>>> So even if the port is in unused state already - means invalid, the
>>>>> device
>>>> pointer still may be valid and point to the last port that used the same id.
>>>>
>>>> If the port is closed, it is unused state, and ethdev layer resources
>>>> freed but as you said device related structures are still there,
>>>> device pointer is still valid and it is still in probed device list
>>>> etc.. We need to able to detach the device even after it is unused state.
>>>
>>> Yes, but detach is for device, not for port.
>>> The device pointer must be taken only when the port is in valid state.
>>> Why?
>>> Because if the port is in UNUSED state it is free to be allocated again by
>> ethdev layer for other device, then, the device pointer may point to other
>> device.
>>>
> 
> Do you agree on the above statement I wrote?
> 
>>>> "stop -> close -> detach" is a normal order, we shouldn't prevent it,
>>>> but your check does prevent it.
>>>
>>> Yes, this is good order, but the pointer of the device should be taken
>> before close.
>>> My patch prevent accessing invalid structure.
>>
>> The ethdev close() dev_ops, frees ethdev related resources, the rte_device
>> is still valid in that struct.
> 
> That’s exactly my concern.
> I think you wrong here, the rte_device may be invalid in that struct, especially after close():
> 
> When the port ID is closed and released, its ethdev structure moves to UNUSED state.
> When an ethdev structure is in UNUSED state it may be attached again to another rte_device - see function rte_eth_dev_allocate.
> Are you agree here?
> 
> In this case, when a new device is attached after close() and before detach_port_device() we may remove wrong rte_device and cause a lot of problems.

The problem here is re-using the ethdev structure when it is closed but not
freed completely, resulting overwriting some fields of it. This is another issue
and can be fixed in the alloc path.

> 
> Do you understand that?
> 
> One more problematic case is a user mistake by the Testpmd command which may cause segfault in the good case and memory overriding in the worst case (my patch case):
> 
> port stop all
> port detach 0
> port detach 0
> 
> detach the same port twice will cause referencing of freed pointer of rte_device.
> 
> 
> All of that is because Testpmd takes ethdev structure information from invalid ethdev structure.
> 
> My patch prevents it.

For this case I am already getting "Device already removed" message from
'detach_port_device()' function.

Your patch is doing two things:
- Hiding the fact that PMD .remove() is not setting the device pointer to null
- Breaking the hotplug functionality

>  
> 
> 
>> And yes your patch prevents accessing them and
>> prevents hotplug remove the device.
>>
> 
> Yes, my patch is not good, solved issues and caused a new one.
> 
> Agree that we need a new fix, my suggestion here is:
> 
> 1. In the Testpmd internal management for hutplug (rmv_port_callback):
> 	Call stop()
> 	Take rte_device pointer( before port close).
> 	Call close().
> 	If no other valid port for the rte_device: 
> 		call detach() by the saved rte_device pointer.

Not sure about pushing more to the application, like checking if any other port
using a device etc..

As far as I understand your concern is when multiple ethdev are using same
device, why not handle this in driver .remove() path, like detect if device
still needs to be used and if so free only ethdev resources and return error,
this error will prevent device resources to be freed:

pci_unplug()
  ret = rte_pci_detach_dev(pdev);
  if (ret == 0)
    rte_pci_remove_device(pdev);
    rte_devargs_remove(dev->devargs);
    ...

This will cause the application receive an error but this is kind of true
because all resources are not freed because they are shared.

When last ethdev detached, driver can send success causing all device resources
to be freed.

> 2. Replace the Testpmd command line for "port detach" with "detach [rte device name]":
> 	Why? 
> 	Detach by port is problematic:
> 	1. If the port is closed - Testpmd cannot get its rte_device from the related ethdev port structure.
> 	2. If the port is not closed - It is not safe to detach it.
> 	3. Attach is done by rte_device name, detach should be in same way.

Testpmd can first close() later detach().

If it is closed already, agreed that new attached devices shouldn't be able to
this struct until it is freed completely. But this is kind of edge case, because
it required new device to be attached after old one closed but before it is
detached.

>  Are you agree?
> 
> 
> I hope you understand now. 
> 
>>> And yes, Testpmd detach stays broken after my patch and after this patch
>> too.
>>>
>>>
>>>>
>>>> I am not very clear about your concern here, "point to the last port
>>>> that used the same id", can you please clarify?
>>>
>>> Yes, when ethdev layer allocates a port ID for a new device, it tries to find
>> UNUSED port.
>>> When found, the port will move to ATTACHED after the PMD finishes its
>> probing function.
>>>
>>> So, any UNUSED port may be allocated for other device and then, the
>> device pointer points to other device.
>>>
>>>>
>>>>>
>>>>>
>>>>>> But according the 'port_id_is_invalid()' API, a closed port is an
>>>>>> invalid port, I think that is wrong in this context.
>>>>>
>>>>> Why?
>>>>
>>>> Closed port is 'invalid' for using it, because ethdev resources are
>>>> freed. But it is not 'invalid' to detach it, why a port being closed
>>>> should prevent freeing its device layer resources?
>>>
>>> I didn't said that, I said that the device pointer should be taken when the
>> port is valid.
>>>
>>>
>>>>
>>>>>
>>>>> You are going to look on ethdev portid structure, don't you think we
>>>>> should
>>>> valid the port before using its structure?
>>>>
>>>> Is your main concern "rte_eth_devices[port_id].device" can be
>>>> dangling pointer?
>>>>
>>>> 1) It is not.
>>>> 2) The check you added to replace it is not correct check.
>>>>
>>> Didn't said that.
>>>
>>> It just may point to other device.
>>> It is not correct to take information from invalid structure.
>>>
>>> Don't you agree that the structure is not valid when the port is not valid?
>>>
>>>>>
>>>>>>>
>>>>>>> I agree this patch is not good and we need a fix but I think the
>>>>>>> bug is
>>>>>> conceptual.
>>>>>>>
>>>>>>> Testpmd tries to do detach by port_id which is derived by ethdev
>>>>>>> port id
>>>>>> while detach work with rte_device.
>>>>>>>
>>>>>>> For example:
>>>>>>> you can see in the line above after +++: dev =
>>>>>>> rte_eth_devices[port_id].device, Testpmd may access invalid  or
>>>>>> reallocated ethdev structure to get the device name and may even
>>>>>> detach unwanted rte_device.
>>>>>>
>>>>>> I thinks whichever function calling 'detach_port_device()' should
>>>>>> check the port validity.
>>>>>> 'detach_port_device()' doesn't know if port reallocated or not, it
>>>>>> will free the given port_id, and when freeing done
>>>>>> 'rte_eth_devices[port_id].device' will be NULL, this looks to me a
>>>>>> valid
>>>> check.
>>>>>
>>>>> Please validate me, check ethdev, I don't think so,
>>>> 'rte_eth_devices[port_id].device still valid after detach.
>>>>
>>>> This is a long stack trace, but what happens is:
>>>>
>>>> rte_dev_remove
>>>>   bus unpug
>>>>     driver remove
>>>>       rte_eth_dev_pci_release
>>>>         eth_dev->device = NULL;
>>>
>>> The last line doesn't happen here because the rte_eth_dev_pci_release
>> moves the port to UNUSED.
>>> And it is bad that application is trying to do it.
>>>
>>>>
>>>> Please check the driver you are testing remove() ops
>>>> (rte_pci_driver.remove()) does cleans the ethdev fields.
>>>>
>>>> A little more detailed stack trace for my environment:
>>>> #0  rte_eth_dev_pci_release (eth_dev=..) at  rte_ethdev_pci.h:143
>>>> #1  rte_eth_dev_pci_generic_remove (pci_dev=.., dev_uninit=..) at
>>>> rte_ethdev_pci.h:199
>>>> #2  eth_i40e_pci_remove (pci_dev=..) at i40e_ethdev.c:710
>>>> #3  rte_pci_detach_dev (dev=..) at pci_common.c:243
>>>> #4  pci_unplug (dev=..) at pci_common.c:537
>>>> #5  local_dev_remove (dev=..) at eal_common_dev.c:321
>>>> #6  rte_dev_remove (dev=..) at eal_common_dev.c:402
>>>> #7  detach_port_device (port_id=0) at testpmd.c:2663
>>>> #8  cmd_operate_detach_port_parsed (parsed_result=.., cl=..,
>>>> data=0x0) at
>>>> cmdline.c:1501
>>>> #9  cmdline_parse (cl=.., buf=.."port detach 0\n") at
>>>> cmdline_parse.c:295
>>>> #10 cmdline_valid_buffer (rdl=.., buf="port detach 0\n", size=15) at
>>>> cmdline.c:31
>>>> #11 rdline_char_in (rdl=.., c=10 '\n') at  cmdline_rdline.c:421
>>>> #12 cmdline_in (cl=.., buf=.."\n", size=1) at cmdline.c:148
>>>> #13 cmdline_interact (cl=..) at cmdline.c:227
>>>> #14 prompt () at cmdline.c:19644
>>>> #15 main (argc=3, argv=..) at testpmd.c:3617
>>>>
>>> Not all the drivers are doing it.
>>> I think it is good if we will do it by ethdev release function.
>>>
>>>
>>>>>
>>>>>> The caller of the 'detach_port_device()' should ensure correct
>>>>>> port_id passed to the function.
>>>>>
>>>>> What is correct port id, if the port was released , is it correct?
>>>>
>>>> You are right, there is no good answer for it, I was thinking
>>>> application state information can be used but no ethdev should able
>>>> to provide this information, we need 'is_freed' kind of check for it,
>>>> currently 'rte_eth_devices[port_id].device' is used for that purpose.
>>>
>>> It is wrong to take device from invalid structure. (I explained a lot above).
>>> Better way to save the rte_device in the start(before close) and call detach
>> by rte_device when we sure that all the ports of this rte_device are
>> released(mlx4 can manage 2 ports one rte_device, also any device supports
>> representors).
>>>
>>> Let's do correct fix.
>>
>> Matan,
>>
>> It become so hard to follow this discussion.The check you add is preventing
>> device hotplug, so breaking the feature, but you want to keep the check to
>> fix something which is still not clear to me.
>>
>> To simplify things, can you please clarify what error are you getting with this
>> patch, and can you please give some details how to reproduce it? So I can
>> debug the issue you are having.
> 
> Added details above, hope everything is clear when you read this line 😊 

Overall I believe this all fuss is about the PMD you are testing not cleaning
the 'rte_eth_devices[port_id].device' pointer which should be handled in driver
level but you are trying to fix this in testpmd causing it fail.


> 
>>
>>>
>>>
>>>>
>>>>>
>>>>>>>
>>>>>>> So, detach is broken with and without this patch.
>>>>>>
>>>>>> I can't see how it is broken without the check, how the problem you
>>>>>> mentioned can be reproduced? Or is it a theoretical issue?
>>>>>> But with this check hotplug support is %100 reproducible broken.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I think Testpmd should change the concept of rte_device mapping
>>>>>>> and put
>>>>>> attention to next:
>>>>>>> 1. Don't detach by ethdev port ID.
>>>>>>> 2. Multiple ethdev port IDs may related to the same rte_device.
>>>>>>>
>>>>>>> The Testpmd user should be sure that all the port IDs of the
>>>>>>> rte_device are
>>>>>> released before the detach call and Testpmd maybe need to validate it.
>>>>>>> And like attach, detach should be triggered by PCI address \
>>>>>>> rte_device
>>>>>> name.
>>>>>>>
>>>>>>
>>>>>> We need to know about port_id too to be able to stop/close it.
>>>>>> And sure no objection to improve the hotplug support but it is
>>>>>> broken now, lets fix it first.
>>>>>>
>>
>> <....>


  reply index

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-12  8:47 [dpdk-dev] [PATCH 1/2] bus/pci: fix driver detach clear Matan Azrad
2019-11-12  8:47 ` [dpdk-dev] [PATCH 2/2] app/testpmd: fix invalid port detaching Matan Azrad
2019-11-12 11:20   ` Iremonger, Bernard
2019-11-20 22:52     ` [dpdk-dev] [dpdk-stable] " David Marchand
2020-01-23 13:19   ` [dpdk-dev] " Yigit, Ferruh
2020-01-23 14:05     ` Matan Azrad
2020-01-23 14:48       ` [dpdk-dev] [dpdk-stable] " Ferruh Yigit
2020-01-23 15:29         ` Matan Azrad
2020-01-23 18:14           ` Ferruh Yigit
2020-01-23 19:25             ` Matan Azrad
2020-01-24 16:28               ` Ferruh Yigit
2020-01-25 18:56                 ` Matan Azrad
2020-02-03 15:58                   ` Ferruh Yigit [this message]
2020-02-03 17:10                     ` Matan Azrad
2020-02-12 13:49                       ` Ferruh Yigit
2020-02-13 12:37                         ` Thomas Monjalon
2020-02-13 13:36                           ` Thomas Monjalon
2020-02-13 14:00                             ` Ferruh Yigit
2019-11-19 22:40 ` [dpdk-dev] [dpdk-stable] [PATCH 1/2] bus/pci: fix driver detach clear Thomas Monjalon
2019-11-20  9:02   ` Matan Azrad
2019-11-20  9:47 ` [dpdk-dev] [PATCH v2] " Matan Azrad
2019-11-20 13:03   ` [dpdk-dev] [dpdk-stable] " David Marchand
2019-11-20 13:44     ` Matan Azrad
2019-11-20 13:51     ` Thomas Monjalon
2019-11-20 17:22       ` David Marchand
2019-11-20 22:52   ` David Marchand

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=33a6aa15-28e4-e770-204c-25ca230ca653@intel.com \
    --to=ferruh.yigit@intel.com \
    --cc=bernard.iremonger@intel.com \
    --cc=david.marchand@redhat.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@linux.intel.com \
    --cc=gaetan.rivet@6wind.com \
    --cc=jia.guo@intel.com \
    --cc=matan@mellanox.com \
    --cc=qi.z.zhang@intel.com \
    --cc=stable@dpdk.org \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

DPDK patches and discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/dev/0 dev/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dev dev/ http://inbox.dpdk.org/dev \
		dev@dpdk.org
	public-inbox-index dev


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.dev


AGPL code for this site: git clone https://public-inbox.org/ public-inbox