[dpdk-dev] [PATCH] eal/freebsd: lock memory device to prevent conflicts

[dpdk-dev] [PATCH] eal/freebsd: lock memory device to prevent conflicts

[Bruce Richardson]

Only a single DPDK process on the system can be using the /dev/contigmem
mappings at a time, but this was never explicitly enforced, e.g. when
using --in-memory flag on two processes. To prevent possible conflict
issues, we lock the dev node when it's in use, preventing other DPDK
processes from starting up and causing problems for us.

Fixes: 764bf26873b9 ("add FreeBSD support")
Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 lib/eal/freebsd/eal_hugepage_info.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/eal/freebsd/eal_hugepage_info.c b/lib/eal/freebsd/eal_hugepage_info.c
index 408f054f7a..4a8d87c23e 100644
--- a/lib/eal/freebsd/eal_hugepage_info.c
+++ b/lib/eal/freebsd/eal_hugepage_info.c
@@ -90,6 +90,10 @@ eal_hugepage_info_init(void)
 		RTE_LOG(ERR, EAL, "could not open "CONTIGMEM_DEV"\n");
 		return -1;
 	}
+	if (flock(fd, LOCK_EX) < 0) {
+		RTE_LOG(ERR, EAL, "could not lock memory. Is another DPDK process running?\n");
+		return -1;
+	}
 
 	if (buffer_size >= 1<<30)
 		RTE_LOG(INFO, EAL, "Contigmem driver has %d buffers, each of size %dGB\n",
-- 
2.30.2

Re: [dpdk-dev] [PATCH] eal/freebsd: lock memory device to prevent conflicts

[Burakov, Anatoly]

On 13-Sep-21 12:06 PM, Bruce Richardson wrote:
> Only a single DPDK process on the system can be using the /dev/contigmem
> mappings at a time, but this was never explicitly enforced, e.g. when
> using --in-memory flag on two processes. To prevent possible conflict
> issues, we lock the dev node when it's in use, preventing other DPDK
> processes from starting up and causing problems for us.
> 
> Fixes: 764bf26873b9 ("add FreeBSD support")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
>   lib/eal/freebsd/eal_hugepage_info.c | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/lib/eal/freebsd/eal_hugepage_info.c b/lib/eal/freebsd/eal_hugepage_info.c
> index 408f054f7a..4a8d87c23e 100644
> --- a/lib/eal/freebsd/eal_hugepage_info.c
> +++ b/lib/eal/freebsd/eal_hugepage_info.c
> @@ -90,6 +90,10 @@ eal_hugepage_info_init(void)
>   		RTE_LOG(ERR, EAL, "could not open "CONTIGMEM_DEV"\n");
>   		return -1;
>   	}
> +	if (flock(fd, LOCK_EX) < 0) {
> +		RTE_LOG(ERR, EAL, "could not lock memory. Is another DPDK process running?\n");
> +		return -1;
> +	}
>   
>   	if (buffer_size >= 1<<30)
>   		RTE_LOG(INFO, EAL, "Contigmem driver has %d buffers, each of size %dGB\n",
> 

This only gets triggered when regular init path is chosen, i.e. 
--no-huge still works. I'm a bit uneasy with --in-memory mode pretending 
to work on FreeBSD and Windows, but that's a separate problem :) As far 
as the patch goes, the problem it addresses does get fixed.

Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] eal/freebsd: lock memory device to prevent conflicts

[Bruce Richardson]

On Mon, Sep 13, 2021 at 02:14:55PM +0100, Burakov, Anatoly wrote:
> On 13-Sep-21 12:06 PM, Bruce Richardson wrote:
> > Only a single DPDK process on the system can be using the /dev/contigmem
> > mappings at a time, but this was never explicitly enforced, e.g. when
> > using --in-memory flag on two processes. To prevent possible conflict
> > issues, we lock the dev node when it's in use, preventing other DPDK
> > processes from starting up and causing problems for us.
> > 
> > Fixes: 764bf26873b9 ("add FreeBSD support")
> > Cc: stable@dpdk.org
> > 
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> > ---
> >   lib/eal/freebsd/eal_hugepage_info.c | 4 ++++
> >   1 file changed, 4 insertions(+)
> > 
> > diff --git a/lib/eal/freebsd/eal_hugepage_info.c b/lib/eal/freebsd/eal_hugepage_info.c
> > index 408f054f7a..4a8d87c23e 100644
> > --- a/lib/eal/freebsd/eal_hugepage_info.c
> > +++ b/lib/eal/freebsd/eal_hugepage_info.c
> > @@ -90,6 +90,10 @@ eal_hugepage_info_init(void)
> >   		RTE_LOG(ERR, EAL, "could not open "CONTIGMEM_DEV"\n");
> >   		return -1;
> >   	}
> > +	if (flock(fd, LOCK_EX) < 0) {
> > +		RTE_LOG(ERR, EAL, "could not lock memory. Is another DPDK process running?\n");
> > +		return -1;
> > +	}
> >   	if (buffer_size >= 1<<30)
> >   		RTE_LOG(INFO, EAL, "Contigmem driver has %d buffers, each of size %dGB\n",
> > 
> 
> This only gets triggered when regular init path is chosen, i.e. --no-huge
> still works.

Yes, but that is ok, I think, since no-huge doesn't use these resources or
suffer from this problem. On the other hand, except for running unit tests,
no-huge mode is pretty useless on FreeBSD as we don't have any
vfio-equivalent support, so all HW access has to use physical addresses
which can only be got using contigmem.

> I'm a bit uneasy with --in-memory mode pretending to work on
> FreeBSD and Windows, but that's a separate problem :)

Yes, it is, though one that does belong is the same area as this one. The
"fix" is probably to just print a warning when --in-memory is used,
informing the user that the flag is ignored and then continue.
Alternatively we can error out, but I think the warn+continue is better,
myself.

> As far as the patch
> goes, the problem it addresses does get fixed.
> 
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
> 
Thanks.

/Bruce

Re: [dpdk-dev] [PATCH] eal/freebsd: lock memory device to prevent conflicts

[Burakov, Anatoly]

On 13-Sep-21 2:36 PM, Bruce Richardson wrote:
> On Mon, Sep 13, 2021 at 02:14:55PM +0100, Burakov, Anatoly wrote:
>> On 13-Sep-21 12:06 PM, Bruce Richardson wrote:
>>> Only a single DPDK process on the system can be using the /dev/contigmem
>>> mappings at a time, but this was never explicitly enforced, e.g. when
>>> using --in-memory flag on two processes. To prevent possible conflict
>>> issues, we lock the dev node when it's in use, preventing other DPDK
>>> processes from starting up and causing problems for us.
>>>
>>> Fixes: 764bf26873b9 ("add FreeBSD support")
>>> Cc: stable@dpdk.org
>>>
>>> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>>> ---
>>>    lib/eal/freebsd/eal_hugepage_info.c | 4 ++++
>>>    1 file changed, 4 insertions(+)
>>>
>>> diff --git a/lib/eal/freebsd/eal_hugepage_info.c b/lib/eal/freebsd/eal_hugepage_info.c
>>> index 408f054f7a..4a8d87c23e 100644
>>> --- a/lib/eal/freebsd/eal_hugepage_info.c
>>> +++ b/lib/eal/freebsd/eal_hugepage_info.c
>>> @@ -90,6 +90,10 @@ eal_hugepage_info_init(void)
>>>    		RTE_LOG(ERR, EAL, "could not open "CONTIGMEM_DEV"\n");
>>>    		return -1;
>>>    	}
>>> +	if (flock(fd, LOCK_EX) < 0) {
>>> +		RTE_LOG(ERR, EAL, "could not lock memory. Is another DPDK process running?\n");
>>> +		return -1;
>>> +	}
>>>    	if (buffer_size >= 1<<30)
>>>    		RTE_LOG(INFO, EAL, "Contigmem driver has %d buffers, each of size %dGB\n",
>>>
>>
>> This only gets triggered when regular init path is chosen, i.e. --no-huge
>> still works.
> 
> Yes, but that is ok, I think, since no-huge doesn't use these resources or
> suffer from this problem. On the other hand, except for running unit tests,
> no-huge mode is pretty useless on FreeBSD as we don't have any
> vfio-equivalent support, so all HW access has to use physical addresses
> which can only be got using contigmem.

What i meant to say was, i've checked this against '--no-huge' which 
*should* still work with this patch, and it does :) So, the phrasing was 
unfortunate, but we agree!

> 
>> I'm a bit uneasy with --in-memory mode pretending to work on
>> FreeBSD and Windows, but that's a separate problem :)
> 
> Yes, it is, though one that does belong is the same area as this one. The
> "fix" is probably to just print a warning when --in-memory is used,
> informing the user that the flag is ignored and then continue.
> Alternatively we can error out, but I think the warn+continue is better,
> myself.

I think erroring out is better. The feature is intended to work a 
certain way, so if we can't guarantee that it does, we can't pretend it 
is "supported" or "is working". But again, irrelevant to this patch :)

> 
>> As far as the patch
>> goes, the problem it addresses does get fixed.
>>
>> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
>>
> Thanks.
> 
> /Bruce
> 


-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH] eal/freebsd: lock memory device to prevent conflicts

[David Marchand]

On Mon, Sep 13, 2021 at 4:40 PM Burakov, Anatoly
<anatoly.burakov@intel.com> wrote:
> >> I'm a bit uneasy with --in-memory mode pretending to work on
> >> FreeBSD and Windows, but that's a separate problem :)
> >
> > Yes, it is, though one that does belong is the same area as this one. The
> > "fix" is probably to just print a warning when --in-memory is used,
> > informing the user that the flag is ignored and then continue.
> > Alternatively we can error out, but I think the warn+continue is better,
> > myself.
>
> I think erroring out is better. The feature is intended to work a
> certain way, so if we can't guarantee that it does, we can't pretend it
> is "supported" or "is working". But again, irrelevant to this patch :)
>

Please, can you review
https://patchwork.dpdk.org/project/dpdk/patch/20210913140848.184928-1-bruce.richardson@intel.com/
?
Thanks.


-- 
David Marchand

Re: [dpdk-dev] [PATCH] eal/freebsd: lock memory device to prevent conflicts

[Dmitry Kozlyuk]

2021-09-13 14:14 (UTC+0100), Burakov, Anatoly:
> [...]
> I'm a bit uneasy with --in-memory mode pretending 
> to work on FreeBSD and Windows, but that's a separate problem :)

On Windows, --in-memory does not pretend to work, just the opposite,
it is enabled implicitly as it's the only working mode.

Re: [dpdk-dev] [PATCH] eal/freebsd: lock memory device to prevent conflicts

[David Marchand]

On Mon, Sep 13, 2021 at 3:15 PM Burakov, Anatoly
<anatoly.burakov@intel.com> wrote:
> On 13-Sep-21 12:06 PM, Bruce Richardson wrote:
> > Only a single DPDK process on the system can be using the /dev/contigmem
> > mappings at a time, but this was never explicitly enforced, e.g. when
> > using --in-memory flag on two processes. To prevent possible conflict
> > issues, we lock the dev node when it's in use, preventing other DPDK
> > processes from starting up and causing problems for us.
> >
> > Fixes: 764bf26873b9 ("add FreeBSD support")
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

Applied, thanks.


-- 
David Marchand

[dpdk-dev] [PATCH v2] eal/freebsd: lock memory device to prevent conflicts

[Bruce Richardson]

Only a single DPDK process on the system can be using the /dev/contigmem
mappings at a time, but this was never explicitly enforced, e.g. when
using --in-memory flag on two processes. To prevent possible conflict
issues, we lock the dev node when it's in use, preventing other DPDK
processes from starting up and causing problems for us.

Fixes: 764bf26873b9 ("add FreeBSD support")
Cc: stable@dpdk.org

Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>
---

V2: Adding missing LOCK_NB flag to make sure the process doesn't sit
    waiting for the lock to be released

 lib/eal/freebsd/eal_hugepage_info.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/eal/freebsd/eal_hugepage_info.c b/lib/eal/freebsd/eal_hugepage_info.c
index 408f054f7a..9dbe375bd3 100644
--- a/lib/eal/freebsd/eal_hugepage_info.c
+++ b/lib/eal/freebsd/eal_hugepage_info.c
@@ -90,6 +90,10 @@ eal_hugepage_info_init(void)
 		RTE_LOG(ERR, EAL, "could not open "CONTIGMEM_DEV"\n");
 		return -1;
 	}
+	if (flock(fd, LOCK_EX | LOCK_NB) < 0) {
+		RTE_LOG(ERR, EAL, "could not lock memory. Is another DPDK process running?\n");
+		return -1;
+	}

 	if (buffer_size >= 1<<30)
 		RTE_LOG(INFO, EAL, "Contigmem driver has %d buffers, each of size %dGB\n",
--
2.30.2

Re: [dpdk-dev] [PATCH v2] eal/freebsd: lock memory device to prevent conflicts

[Thomas Monjalon]

13/09/2021 16:08, Bruce Richardson:
> Only a single DPDK process on the system can be using the /dev/contigmem
> mappings at a time, but this was never explicitly enforced, e.g. when
> using --in-memory flag on two processes. To prevent possible conflict
> issues, we lock the dev node when it's in use, preventing other DPDK
> processes from starting up and causing problems for us.
> 
> Fixes: 764bf26873b9 ("add FreeBSD support")
> Cc: stable@dpdk.org
> 
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> Reviewed-by: Anatoly Burakov <anatoly.burakov@intel.com>

Looks to be applied by David.