* [dpdk-dev] eal: can not run secondary process on openstack environment
@ 2020-04-15 10:06 陈亚辉-云杉研发部
2020-04-17 13:52 ` Burakov, Anatoly
0 siblings, 1 reply; 6+ messages in thread
From: 陈亚辉-云杉研发部 @ 2020-04-15 10:06 UTC (permalink / raw)
To: dev, jiping, xiangyang
dpdk version: 18.11
Ovs-dpdk runs as openvswitch account on the openstack environment. Use the
root account to run another --proc-type=secondary program,
such as dpdk-pdump or helloword with parameter --proc-type=secondary , and
it will report an Err: Cannot open '/var/run/dpdk/rte/config' for
rte_mem_config.
Check the source code eal.c:
int
eal_create_runtime_dir(void)
{
const char *directory = default_runtime_dir;
const char *xdg_runtime_dir = getenv("XDG_RUNTIME_DIR");
const char *fallback = "/tmp";
char tmp[PATH_MAX];
int ret;
if (getuid() != 0) {
/* try XDG path first, fall back to /tmp */
if (xdg_runtime_dir != NULL)
directory = xdg_runtime_dir;
else
directory = fallback;
}
/* create DPDK subdirectory under runtime dir */
ret = snprintf(tmp, sizeof(tmp), "%s/dpdk", directory);
if (ret < 0 || ret == sizeof(tmp)) {
RTE_LOG(ERR, EAL, "Error creating DPDK runtime path
name\n");
return -1;
}
/* create prefix-specific subdirectory under DPDK runtime dir */
ret = snprintf(runtime_dir, sizeof(runtime_dir), "%s/%s",
tmp, eal_get_hugefile_prefix());
if (ret < 0 || ret == sizeof(runtime_dir)) {
RTE_LOG(ERR, EAL, "Error creating prefix-specific runtime
path name\n");
return -1;
}
The root account is corresponds to the directory /var/run/DPDK/rte, but
openvswitch account is directory /var/run/openvswitch DPDK/rte.
Then, I changed to the openvswitch account and run the process again,
errors below were reported:
EAL: Detected 40 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket
/var/run/openvswitch/dpdk/rte/mp_socket_306857_7e76690dafe702
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Could not map memory from primary process
EAL: FATAL: Cannot init memory
EAL: Cannot init memory
There seems to be insufficient permissions. So, I change the code blow and
all the things get to be OK:
int
eal_create_runtime_dir(void)
{
const char *directory = default_runtime_dir;
const char *xdg_runtime_dir = getenv("XDG_RUNTIME_DIR");
const char *fallback = "/tmp";
char tmp[PATH_MAX];
int ret;
if (getuid() != 0) {
/* try XDG path first, fall back to /tmp */
if (xdg_runtime_dir != NULL)
directory = xdg_runtime_dir;
else
directory = fallback;
}
directory = "/var/run/openvswitch"; // added by my for test.........
/* create DPDK subdirectory under runtime dir */
ret = snprintf(tmp, sizeof(tmp), "%s/dpdk", directory);
if (ret < 0 || ret == sizeof(tmp)) {
RTE_LOG(ERR, EAL, "Error creating DPDK runtime path
name\n");
return -1;
}
This is supposed to be a DPDK BUG. Considering that the primary and
secondary processes may not be the same account,
process with parameter --proc-type=secondary should get the runtime
directory from the primary process instead of
generating a directory as it does now.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [dpdk-dev] eal: can not run secondary process on openstack environment
2020-04-15 10:06 [dpdk-dev] eal: can not run secondary process on openstack environment 陈亚辉-云杉研发部
@ 2020-04-17 13:52 ` Burakov, Anatoly
0 siblings, 0 replies; 6+ messages in thread
From: Burakov, Anatoly @ 2020-04-17 13:52 UTC (permalink / raw)
To: 陈亚辉-云杉研发部,
dev, jiping, xiangyang
On 15-Apr-20 11:06 AM, 陈亚辉-云杉研发部 wrote:
> dpdk version: 18.11
>
> Ovs-dpdk runs as openvswitch account on the openstack environment. Use the
> root account to run another --proc-type=secondary program,
> such as dpdk-pdump or helloword with parameter --proc-type=secondary , and
> it will report an Err: Cannot open '/var/run/dpdk/rte/config' for
> rte_mem_config.
>
> Check the source code eal.c:
>
> int
> eal_create_runtime_dir(void)
> {
> const char *directory = default_runtime_dir;
> const char *xdg_runtime_dir = getenv("XDG_RUNTIME_DIR");
> const char *fallback = "/tmp";
> char tmp[PATH_MAX];
> int ret;
>
> if (getuid() != 0) {
> /* try XDG path first, fall back to /tmp */
> if (xdg_runtime_dir != NULL)
> directory = xdg_runtime_dir;
> else
> directory = fallback;
> }
>
> /* create DPDK subdirectory under runtime dir */
> ret = snprintf(tmp, sizeof(tmp), "%s/dpdk", directory);
> if (ret < 0 || ret == sizeof(tmp)) {
> RTE_LOG(ERR, EAL, "Error creating DPDK runtime path
> name\n");
> return -1;
> }
>
> /* create prefix-specific subdirectory under DPDK runtime dir */
> ret = snprintf(runtime_dir, sizeof(runtime_dir), "%s/%s",
> tmp, eal_get_hugefile_prefix());
> if (ret < 0 || ret == sizeof(runtime_dir)) {
> RTE_LOG(ERR, EAL, "Error creating prefix-specific runtime
> path name\n");
> return -1;
> }
>
> The root account is corresponds to the directory /var/run/DPDK/rte, but
> openvswitch account is directory /var/run/openvswitch DPDK/rte.
> Then, I changed to the openvswitch account and run the process again,
> errors below were reported:
> EAL: Detected 40 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket
> /var/run/openvswitch/dpdk/rte/mp_socket_306857_7e76690dafe702
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: Could not map memory from primary process
> EAL: FATAL: Cannot init memory
> EAL: Cannot init memory
>
> There seems to be insufficient permissions. So, I change the code blow and
> all the things get to be OK:
>
> int
> eal_create_runtime_dir(void)
> {
> const char *directory = default_runtime_dir;
> const char *xdg_runtime_dir = getenv("XDG_RUNTIME_DIR");
> const char *fallback = "/tmp";
> char tmp[PATH_MAX];
> int ret;
>
> if (getuid() != 0) {
> /* try XDG path first, fall back to /tmp */
> if (xdg_runtime_dir != NULL)
> directory = xdg_runtime_dir;
> else
> directory = fallback;
> }
>
> directory = "/var/run/openvswitch"; // added by my for test.........
> /* create DPDK subdirectory under runtime dir */
> ret = snprintf(tmp, sizeof(tmp), "%s/dpdk", directory);
> if (ret < 0 || ret == sizeof(tmp)) {
> RTE_LOG(ERR, EAL, "Error creating DPDK runtime path
> name\n");
> return -1;
> }
>
> This is supposed to be a DPDK BUG. Considering that the primary and
> secondary processes may not be the same account,
> process with parameter --proc-type=secondary should get the runtime
> directory from the primary process instead of
> generating a directory as it does now.
>
I'm not sure how would that even be accomplished, because there is no
way the initializing secondary process knows where to look for primary
processes. Do you have any suggestions as to the mechanism of finding a
primary process in such case?
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] eal: can not run secondary process on openstack environment
@ 2020-04-30 4:14 陈亚辉-云杉研发部
2020-04-30 9:14 ` Burakov, Anatoly
0 siblings, 1 reply; 6+ messages in thread
From: 陈亚辉-云杉研发部 @ 2020-04-30 4:14 UTC (permalink / raw)
To: anatoly.burakov; +Cc: dev, jiping, xiangyang
Deleting xdg_runtime_dir and fallback, runtime_dir will always be
"/var/run" defined by code:
static const char *default_runtime_dir = "/var/run"
Here is my patch:
# diff -ruN ~/dpdk-18.11.6/lib/librte_eal/linuxapp/eal/eal_orig.c eal.c
--- ~/dpdk-18.11.6/lib/librte_eal/linuxapp/eal/eal_orig.c 2020-04-14
12:16:04.774609296 +0800
+++ eal.c 2020-04-30 11:29:26.526918889 +0800
@@ -105,18 +105,9 @@
eal_create_runtime_dir(void)
{
const char *directory = default_runtime_dir;
- const char *xdg_runtime_dir = getenv("XDG_RUNTIME_DIR");
- const char *fallback = "/tmp";
char tmp[PATH_MAX];
int ret;
- if (getuid() != 0) {
- /* try XDG path first, fall back to /tmp */
- if (xdg_runtime_dir != NULL)
- directory = xdg_runtime_dir;
- else
- directory = fallback;
- }
/* create DPDK subdirectory under runtime dir */
ret = snprintf(tmp, sizeof(tmp), "%s/dpdk", directory);
if (ret < 0 || ret == sizeof(tmp)) {
Date: Fri, 17 Apr 2020 14:52:58 +0100
From: "Burakov, Anatoly" <anatoly.burakov@intel.com>
To: ???-????? <goodluckwillcomesoon@gmail.com>, dev@dpdk.org,
jiping@yunshan.net, xiangyang@yunshan.net
Subject: Re: [dpdk-dev] eal: can not run secondary process on
openstack environment
Message-ID: <8f5bb1c4-7572-0c18-ecab-89c1340430cb@intel.com>
Content-Type: text/plain; charset=utf-8; format=flowed
On 15-Apr-20 11:06 AM, ???-????? wrote:
> dpdk version: 18.11
>
> Ovs-dpdk runs as openvswitch account on the openstack environment. Use the
> root account to run another --proc-type=secondary program,
> such as dpdk-pdump or helloword with parameter --proc-type=secondary , and
> it will report an Err: Cannot open '/var/run/dpdk/rte/config' for
> rte_mem_config.
>
> Check the source code eal.c:
>
> int
> eal_create_runtime_dir(void)
> {
> const char *directory = default_runtime_dir;
> const char *xdg_runtime_dir = getenv("XDG_RUNTIME_DIR");
> const char *fallback = "/tmp";
> char tmp[PATH_MAX];
> int ret;
>
> if (getuid() != 0) {
> /* try XDG path first, fall back to /tmp */
> if (xdg_runtime_dir != NULL)
> directory = xdg_runtime_dir;
> else
> directory = fallback;
> }
>
> /* create DPDK subdirectory under runtime dir */
> ret = snprintf(tmp, sizeof(tmp), "%s/dpdk", directory);
> if (ret < 0 || ret == sizeof(tmp)) {
> RTE_LOG(ERR, EAL, "Error creating DPDK runtime path
> name\n");
> return -1;
> }
>
> /* create prefix-specific subdirectory under DPDK runtime dir */
> ret = snprintf(runtime_dir, sizeof(runtime_dir), "%s/%s",
> tmp, eal_get_hugefile_prefix());
> if (ret < 0 || ret == sizeof(runtime_dir)) {
> RTE_LOG(ERR, EAL, "Error creating prefix-specific runtime
> path name\n");
> return -1;
> }
>
> The root account is corresponds to the directory /var/run/DPDK/rte, but
> openvswitch account is directory /var/run/openvswitch DPDK/rte.
> Then, I changed to the openvswitch account and run the process again,
> errors below were reported:
> EAL: Detected 40 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket
> /var/run/openvswitch/dpdk/rte/mp_socket_306857_7e76690dafe702
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: Could not map memory from primary process
> EAL: FATAL: Cannot init memory
> EAL: Cannot init memory
>
> There seems to be insufficient permissions. So, I change the code blow and
> all the things get to be OK:
>
> int
> eal_create_runtime_dir(void)
> {
> const char *directory = default_runtime_dir;
> const char *xdg_runtime_dir = getenv("XDG_RUNTIME_DIR");
> const char *fallback = "/tmp";
> char tmp[PATH_MAX];
> int ret;
>
> if (getuid() != 0) {
> /* try XDG path first, fall back to /tmp */
> if (xdg_runtime_dir != NULL)
> directory = xdg_runtime_dir;
> else
> directory = fallback;
> }
>
> directory = "/var/run/openvswitch"; // added by my for
test.........
> /* create DPDK subdirectory under runtime dir */
> ret = snprintf(tmp, sizeof(tmp), "%s/dpdk", directory);
> if (ret < 0 || ret == sizeof(tmp)) {
> RTE_LOG(ERR, EAL, "Error creating DPDK runtime path
> name\n");
> return -1;
> }
>
> This is supposed to be a DPDK BUG. Considering that the primary and
> secondary processes may not be the same account,
> process with parameter --proc-type=secondary should get the runtime
> directory from the primary process instead of
> generating a directory as it does now.
>
I'm not sure how would that even be accomplished, because there is no
way the initializing secondary process knows where to look for primary
processes. Do you have any suggestions as to the mechanism of finding a
primary process in such case?
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [dpdk-dev] eal: can not run secondary process on openstack environment
2020-04-30 4:14 陈亚辉-云杉研发部
@ 2020-04-30 9:14 ` Burakov, Anatoly
2020-04-30 16:09 ` Stephen Hemminger
0 siblings, 1 reply; 6+ messages in thread
From: Burakov, Anatoly @ 2020-04-30 9:14 UTC (permalink / raw)
To: 陈亚辉-云杉研发部
Cc: dev, jiping, xiangyang, David Marchand
On 30-Apr-20 5:14 AM, 陈亚辉-云杉研发部 wrote:
> Deleting xdg_runtime_dir and fallback, runtime_dir will always be
> "/var/run" defined by code:
> static const char *default_runtime_dir = "/var/run"
>
I'm not sure this is a good solution. Generally, IMO, having separate
directories for DPDK processes for different users is a good thing.
Also, XDG directory exists for a reason, and i think on some distros
/var/run is not even there any more (or symlinked to /run, or...).
So, i don't think this is the way to go. David, thoughts?
--
Thanks,
Anatoly
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] eal: can not run secondary process on openstack environment
2020-04-30 9:14 ` Burakov, Anatoly
@ 2020-04-30 16:09 ` Stephen Hemminger
2020-04-30 16:36 ` Bruce Richardson
0 siblings, 1 reply; 6+ messages in thread
From: Stephen Hemminger @ 2020-04-30 16:09 UTC (permalink / raw)
To: Burakov, Anatoly
Cc: 陈亚辉-云杉研发部,
dev, jiping, xiangyang, David Marchand
On Thu, 30 Apr 2020 10:14:57 +0100
"Burakov, Anatoly" <anatoly.burakov@intel.com> wrote:
> On 30-Apr-20 5:14 AM, 陈亚辉-云杉研发部 wrote:
> > Deleting xdg_runtime_dir and fallback, runtime_dir will always be
> > "/var/run" defined by code:
> > static const char *default_runtime_dir = "/var/run"
> >
>
> I'm not sure this is a good solution. Generally, IMO, having separate
> directories for DPDK processes for different users is a good thing.
> Also, XDG directory exists for a reason, and i think on some distros
> /var/run is not even there any more (or symlinked to /run, or...).
>
> So, i don't think this is the way to go. David, thoughts?
>
The unix-domain sockets should be using the abstract socket
naming which would solve a bunch of problems like removing on program
crash, SELinux, and container namepaces.
See unix(7) for more infomation.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] eal: can not run secondary process on openstack environment
2020-04-30 16:09 ` Stephen Hemminger
@ 2020-04-30 16:36 ` Bruce Richardson
0 siblings, 0 replies; 6+ messages in thread
From: Bruce Richardson @ 2020-04-30 16:36 UTC (permalink / raw)
To: Stephen Hemminger
Cc: Burakov, Anatoly,
陈亚辉-云杉研发部,
dev, jiping, xiangyang, David Marchand
On Thu, Apr 30, 2020 at 09:09:02AM -0700, Stephen Hemminger wrote:
> On Thu, 30 Apr 2020 10:14:57 +0100
> "Burakov, Anatoly" <anatoly.burakov@intel.com> wrote:
>
> > On 30-Apr-20 5:14 AM, 陈亚辉-云杉研发部 wrote:
> > > Deleting xdg_runtime_dir and fallback, runtime_dir will always be
> > > "/var/run" defined by code:
> > > static const char *default_runtime_dir = "/var/run"
> > >
> >
> > I'm not sure this is a good solution. Generally, IMO, having separate
> > directories for DPDK processes for different users is a good thing.
> > Also, XDG directory exists for a reason, and i think on some distros
> > /var/run is not even there any more (or symlinked to /run, or...).
> >
> > So, i don't think this is the way to go. David, thoughts?
> >
>
> The unix-domain sockets should be using the abstract socket
> naming which would solve a bunch of problems like removing on program
> crash, SELinux, and container namepaces.
>
> See unix(7) for more infomation.
I disagree, I think using sockets on the filesystem is still the way to go.
Yes, with abstract sockets they disappear automatically, but on the other
hand they have the following issues:
* they are non-portable and available only on linux
* they can't be protected easily just using the filesystem permissions -
any unix sockets we create in the DPDK runtime directory are only
accessible by current user and root. For many cases in DPDK, we don't
want arbitrary users or programs able to connect to DPDK.
/Bruce
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-04-30 16:36 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-15 10:06 [dpdk-dev] eal: can not run secondary process on openstack environment 陈亚辉-云杉研发部
2020-04-17 13:52 ` Burakov, Anatoly
2020-04-30 4:14 陈亚辉-云杉研发部
2020-04-30 9:14 ` Burakov, Anatoly
2020-04-30 16:09 ` Stephen Hemminger
2020-04-30 16:36 ` Bruce Richardson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).