[dpdk-dev] [PATCH v1] ethdev: fix multi-process NULL dereference crashes

[dpdk-dev] [PATCH v1] ethdev: fix multi-process NULL dereference crashes

[Remy Horton]

Even though only primary processes should setup PMDs, secondary
processes were also blanket zeroing ethernet device memory. The
result was NULL dereference crashes in multi-process setups.

Fixes: 7f95f78a8aea ("ethdev: clear data when allocating device")

Signed-off-by: Remy Horton <remy.horton@intel.com>
---
 doc/guides/rel_notes/release_17_02.rst | 6 ++++++
 lib/librte_ether/rte_ethdev.c          | 3 ++-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_17_02.rst b/doc/guides/rel_notes/release_17_02.rst
index 180af82..20a4ced 100644
--- a/doc/guides/rel_notes/release_17_02.rst
+++ b/doc/guides/rel_notes/release_17_02.rst
@@ -72,6 +72,12 @@ Resolved Issues
 EAL
 ~~~
 
+* **ethdev: Fixed crash with multi-processing.**
+
+  Even though only primary processes should setup PMDs, secondary
+  processes were also blanket zeroing ethernet device memory. The
+  result was NULL dereference crashes in multi-process setups.
+
 
 Drivers
 ~~~~~~~
diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 9dea1f1..a681982 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -212,7 +212,8 @@ rte_eth_dev_allocate(const char *name)
 
 	eth_dev = &rte_eth_devices[port_id];
 	eth_dev->data = &rte_eth_dev_data[port_id];
-	memset(eth_dev->data, 0, sizeof(*eth_dev->data));
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		memset(eth_dev->data, 0, sizeof(*eth_dev->data));
 	snprintf(eth_dev->data->name, sizeof(eth_dev->data->name), "%s", name);
 	eth_dev->data->port_id = port_id;
 	eth_dev->data->mtu = ETHER_MTU;
-- 
2.5.5

Re: [dpdk-dev] [PATCH v1] ethdev: fix multi-process NULL dereference crashes

[Thomas Monjalon]

2017-01-11 02:42, Remy Horton:
> Even though only primary processes should setup PMDs, secondary
> processes were also blanket zeroing ethernet device memory. The
> result was NULL dereference crashes in multi-process setups.
> 
> Fixes: 7f95f78a8aea ("ethdev: clear data when allocating device")

I think it can be fixed by this patch:

http://dpdk.org/ml/archives/dev/2017-January/054220.html

Re: [dpdk-dev] [PATCH v1] ethdev: fix multi-process NULL dereference crashes

[Remy Horton]

On 11/01/2017 14:22, Thomas Monjalon wrote:
> 2017-01-11 02:42, Remy Horton:
>> Even though only primary processes should setup PMDs, secondary
>> processes were also blanket zeroing ethernet device memory. The
>> result was NULL dereference crashes in multi-process setups.
>> Fixes: 7f95f78a8aea ("ethdev: clear data when allocating device")
>
> I think it can be fixed by this patch:
>
> http://dpdk.org/ml/archives/dev/2017-January/054220.html

Close call - really depends how the (likley) merge conflict is resolved...

..Remy

Re: [dpdk-dev] [PATCH v1] ethdev: fix multi-process NULL dereference crashes

[Thomas Monjalon]

2017-01-11 02:42, Remy Horton:
> +* **ethdev: Fixed crash with multi-processing.**
> +
> +  Even though only primary processes should setup PMDs, secondary
> +  processes were also blanket zeroing ethernet device memory. The
> +  result was NULL dereference crashes in multi-process setups.
> +

3 comments here:
- it is in the wrong section (EAL instead of Drivers)
- secondary processes can setup a vdev PMD
- before Yuanhan's patch, even PCI PMD were blanking primary process data


> --- a/lib/librte_ether/rte_ethdev.c
> +++ b/lib/librte_ether/rte_ethdev.c
> @@ -212,7 +212,8 @@ rte_eth_dev_allocate(const char *name)
>  
>  	eth_dev = &rte_eth_devices[port_id];
>  	eth_dev->data = &rte_eth_dev_data[port_id];
> -	memset(eth_dev->data, 0, sizeof(*eth_dev->data));
> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> +		memset(eth_dev->data, 0, sizeof(*eth_dev->data));
>  	snprintf(eth_dev->data->name, sizeof(eth_dev->data->name), "%s", name);
>  	eth_dev->data->port_id = port_id;
>  	eth_dev->data->mtu = ETHER_MTU;
> 

I propose this rebase:

-       memset(&rte_eth_dev_data[port_id], 0, sizeof(struct rte_eth_dev_data));
        eth_dev = eth_dev_get(port_id);
+       if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+               memset(eth_dev->data, 0, sizeof(*eth_dev->data));
        snprintf(eth_dev->data->name, sizeof(eth_dev->data->name), "%s", name);
        eth_dev->data->port_id = port_id;
        eth_dev->data->mtu = ETHER_MTU;

Re: [dpdk-dev] [PATCH v1] ethdev: fix multi-process NULL dereference crashes

[Remy Horton]

On 20/01/2017 18:37, Thomas Monjalon wrote:
[..]
> 3 comments here:
> - it is in the wrong section (EAL instead of Drivers)
> - secondary processes can setup a vdev PMD
> - before Yuanhan's patch, even PCI PMD were blanking primary process data

Since the code being changed is in rte_ether rather than drivers/net it 
seemed the logical place to me.. :)

> I propose this rebase:
>
> -       memset(&rte_eth_dev_data[port_id], 0, sizeof(struct rte_eth_dev_data));
>         eth_dev = eth_dev_get(port_id);
> +       if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> +               memset(eth_dev->data, 0, sizeof(*eth_dev->data));
>         snprintf(eth_dev->data->name, sizeof(eth_dev->data->name), "%s", name);
>         eth_dev->data->port_id = port_id;
>         eth_dev->data->mtu = ETHER_MTU;

Seems OK to me, assuming Yuanhan's patch is going in as-is.

..Remy

Re: [dpdk-dev] [PATCH v1] ethdev: fix multi-process NULL dereference crashes

[Thomas Monjalon]

2017-01-24 08:16, Remy Horton:
> 
> On 20/01/2017 18:37, Thomas Monjalon wrote:
> [..]
> > 3 comments here:
> > - it is in the wrong section (EAL instead of Drivers)
> > - secondary processes can setup a vdev PMD
> > - before Yuanhan's patch, even PCI PMD were blanking primary process data
> 
> Since the code being changed is in rte_ether rather than drivers/net it 
> seemed the logical place to me.. :)

The change is in ethdev, and you put the release note in EAL.
So no, it is not logical, because ethdev is not EAL.

> > I propose this rebase:
> >
> > -       memset(&rte_eth_dev_data[port_id], 0, sizeof(struct rte_eth_dev_data));
> >         eth_dev = eth_dev_get(port_id);
> > +       if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> > +               memset(eth_dev->data, 0, sizeof(*eth_dev->data));
> >         snprintf(eth_dev->data->name, sizeof(eth_dev->data->name), "%s", name);
> >         eth_dev->data->port_id = port_id;
> >         eth_dev->data->mtu = ETHER_MTU;
> 
> Seems OK to me, assuming Yuanhan's patch is going in as-is.

Yuanhan's patch is already part of RC1.

Re: [dpdk-dev] [PATCH v1] ethdev: fix multi-process NULL dereference crashes

[Remy Horton]

On 24/01/2017 10:49, Thomas Monjalon wrote:
[..]
>> Seems OK to me, assuming Yuanhan's patch is going in as-is.
>
> Yuanhan's patch is already part of RC1.

Ah ok. I'll rebase a v2 then..

[dpdk-dev] [PATCH v2] ethdev: fix multi-process NULL dereference crashes

[Remy Horton]

Secondary processes were blanket zeroing ethernet device memory,
resulting in NULL dereference crashes in multi-process setups.

Fixes: 7f95f78a8aea ("ethdev: clear data when allocating device")

Signed-off-by: Remy Horton <remy.horton@intel.com>
---
 doc/guides/rel_notes/release_17_02.rst | 5 +++++
 lib/librte_ether/rte_ethdev.c          | 4 +++-
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/doc/guides/rel_notes/release_17_02.rst b/doc/guides/rel_notes/release_17_02.rst
index 0ecd720..1472f84 100644
--- a/doc/guides/rel_notes/release_17_02.rst
+++ b/doc/guides/rel_notes/release_17_02.rst
@@ -222,6 +222,11 @@ Drivers
   Fixed few regressions introduced in recent releases that break the virtio
   multiple process support.
 
+* **ethdev: Fixed crash with multi-processing.**
+
+  Secondary processes were blanket zeroing ethernet device memory,
+  resulting in NULL dereference crashes in multi-process setups.
+
 
 Libraries
 ~~~~~~~~~
diff --git a/lib/librte_ether/rte_ethdev.c b/lib/librte_ether/rte_ethdev.c
index 61f44e2..d911921 100644
--- a/lib/librte_ether/rte_ethdev.c
+++ b/lib/librte_ether/rte_ethdev.c
@@ -225,8 +225,10 @@ rte_eth_dev_allocate(const char *name)
 		return NULL;
 	}
 
-	memset(&rte_eth_dev_data[port_id], 0, sizeof(struct rte_eth_dev_data));
 	eth_dev = eth_dev_get(port_id);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		memset(&rte_eth_dev_data[port_id], 0,
+			sizeof(struct rte_eth_dev_data));
 	snprintf(eth_dev->data->name, sizeof(eth_dev->data->name), "%s", name);
 	eth_dev->data->port_id = port_id;
 	eth_dev->data->mtu = ETHER_MTU;
-- 
2.5.5

Re: [dpdk-dev] [PATCH v2] ethdev: fix multi-process NULL dereference crashes

[Thomas Monjalon]

2017-01-24 15:01, Remy Horton:
> Secondary processes were blanket zeroing ethernet device memory,
> resulting in NULL dereference crashes in multi-process setups.
> 
> Fixes: 7f95f78a8aea ("ethdev: clear data when allocating device")
> 
> Signed-off-by: Remy Horton <remy.horton@intel.com>
> ---
>  doc/guides/rel_notes/release_17_02.rst | 5 +++++
>  lib/librte_ether/rte_ethdev.c          | 4 +++-
>  2 files changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/doc/guides/rel_notes/release_17_02.rst b/doc/guides/rel_notes/release_17_02.rst
> index 0ecd720..1472f84 100644
> --- a/doc/guides/rel_notes/release_17_02.rst
> +++ b/doc/guides/rel_notes/release_17_02.rst
> @@ -222,6 +222,11 @@ Drivers
>    Fixed few regressions introduced in recent releases that break the virtio
>    multiple process support.
>  
> +* **ethdev: Fixed crash with multi-processing.**
> +
> +  Secondary processes were blanket zeroing ethernet device memory,
> +  resulting in NULL dereference crashes in multi-process setups.

It does not describe exactly the use-case it is fixing (same in commit message).
I guess you saw an issue when creating a vdev in the primary process and
another one in a secondary process, erasing the data of the first one.

nit: ethdev bug should be shown before PMD bugs like virtio one above.

> --- a/lib/librte_ether/rte_ethdev.c
> +++ b/lib/librte_ether/rte_ethdev.c
> @@ -225,8 +225,10 @@ rte_eth_dev_allocate(const char *name)
>  		return NULL;
>  	}
>  
> -	memset(&rte_eth_dev_data[port_id], 0, sizeof(struct rte_eth_dev_data));
>  	eth_dev = eth_dev_get(port_id);
> +	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
> +		memset(&rte_eth_dev_data[port_id], 0,
> +			sizeof(struct rte_eth_dev_data));

My previous proposal was:
	memset(eth_dev->data, 0, sizeof(*eth_dev->data))
It is better to avoid reference to the global array rte_eth_dev_data.

Anyway, the shared data are still overwritten for the name, the port id
and the MTU.
Please describe the exact case where it is working for you.

Re: [dpdk-dev] [PATCH v2] ethdev: fix multi-process NULL dereference crashes

[Remy Horton]

On 25/01/2017 11:56, Thomas Monjalon wrote:
[..]
> It does not describe exactly the use-case it is fixing (same in commit message).
> I guess you saw an issue when creating a vdev in the primary process and
> another one in a secondary process, erasing the data of the first one.

In my use-case the secondary process is proc_info, which appeared to be 
blanking the shared memory then leaving the NULL-pointer landmines for 
the primary process to land on. I'm not entirely sure why this type of 
secondary process needs to be running any ethdev startup code at all, as 
all it is doing is pulling data out of shared memory..


> My previous proposal was:
> 	memset(eth_dev->data, 0, sizeof(*eth_dev->data))
> It is better to avoid reference to the global array rte_eth_dev_data.

Git rebase screwed up, and it got lost en-route :(

..Remy

Re: [dpdk-dev] [PATCH v2] ethdev: fix multi-process NULL dereference crashes

[Remy Horton]

On 24/01/2017 15:01, Remy Horton wrote:
> Secondary processes were blanket zeroing ethernet device memory,
> resulting in NULL dereference crashes in multi-process setups.
>
> Fixes: 7f95f78a8aea ("ethdev: clear data when allocating device")
>
> Signed-off-by: Remy Horton <remy.horton@intel.com>

Self-NAK: Condition is now tautology on code path that was causing crashes

Re: [dpdk-dev] [PATCH v2] ethdev: fix multi-process NULL dereference crashes

[Thomas Monjalon]

2017-01-25 14:02, Remy Horton:
> 
> On 24/01/2017 15:01, Remy Horton wrote:
> > Secondary processes were blanket zeroing ethernet device memory,
> > resulting in NULL dereference crashes in multi-process setups.
> >
> > Fixes: 7f95f78a8aea ("ethdev: clear data when allocating device")
> >
> > Signed-off-by: Remy Horton <remy.horton@intel.com>
> 
> Self-NAK: Condition is now tautology on code path that was causing crashes

What do you mean exactly?

Re: [dpdk-dev] [PATCH v2] ethdev: fix multi-process NULL dereference crashes

[Remy Horton]

On 25/01/2017 14:31, Thomas Monjalon wrote:
> 2017-01-25 14:02, Remy Horton:
[..]
>> Self-NAK: Condition is now tautology on code path that was causing crashes
>
> What do you mean exactly?

There is an if(rte_eal_process_type() == RTE_PROC_PRIMARY) in a calling 
function, so the one my patch was introducing is now redundant.

..Remy