[dpdk-dev] Reg DPDK EAL log-level=8 options is not throwing any DEBUG messages in /var/log/messages

[dpdk-dev] Reg DPDK EAL log-level=8 options is not throwing any DEBUG messages in /var/log/messages

[Shanmugasundaram M]

Dear Team,

we are using 18.11.1-6WIND DPDK. we are trying to get EAL Debug messages
and we are not able to see any one from them in /var/log/messages. Kindly
let me know --log-level=8 is enough or --log-level=*8 is correct  OR
--log-level=eal,debug is correct ?
Appreciate your quick response.

regards,
Shanmugasundaram M

Re: [dpdk-dev] Reg DPDK EAL log-level=8 options is not throwing any DEBUG messages in /var/log/messages

[Stephen Hemminger]

On Wed, 10 Mar 2021 00:30:24 +0530
Shanmugasundaram M <shansirtt@gmail.com> wrote:

> Dear Team,
> 
> we are using 18.11.1-6WIND DPDK. we are trying to get EAL Debug messages
> and we are not able to see any one from them in /var/log/messages. Kindly
> let me know --log-level=8 is enough or --log-level=*8 is correct  OR
> --log-level=eal,debug is correct ?
> Appreciate your quick response.
> 
> regards,
> Shanmugasundaram M

DPDK now has log-level on a per-feature level. The old log level was global.
The legacy option is what you are using.

The document is missing about log-level options.

Now log-level takes three forms:
    --log-level=regex,N
    --log-level=pattern:N
    --log-level=N

The first takes a regular expressions so for example:
     -log-level='pmd\..*,8'
Would set logging at debug for Poll Mode Drivers.  Regex period
is match all, not match period.

The second form takes a shell style pattern (glob):
    --log-level='pmd.net.*:debug'
Sets debug log level for all Poll Mode Network drivers.
All forms take numeric or symbolic names for logging levels in current versions

The third form sets the global log level only.

The actual logic around what gets logged or not is a union of
the global and per-logtype argument. So if global log level is
set high (like NOTICE) then no info or debug messages will get printed.

In other words, --log-level=8 has no effect both because it only
sets the global level (and the default global level already is 8).

To turn on all logging. Just do --log-level='.*,8' (or similar)

Re: [dpdk-dev] Reg DPDK EAL log-level=8 options is not throwing any DEBUG messages in /var/log/messages

[Shanmugasundaram M]

THanks Steven for the quick response.  I want to enable  EAL module
(complete logging at debug level ) only during bootup of fastpath. Once
bootup is done, i want to move this module log-level to default/ original.
Currently I'm using     --log-level='.*',8  along with
./sources/fp/common/fp-var.h  [ #define FP_LOG_DEFAULT FP_LOG_DEBUG ]
instead of [ #define FP_LOG_DEFAULT FP_LOG_ERR ]. refer below snip
<snip>
-#define FP_LOG_DEFAULT FP_LOG_ERR
+#define FP_LOG_DEFAULT FP_LOG_DEBUG
The problem is its throwing all the other debug messages like this after
fastpath started. Any way to disable below message <snip3> and only log
bootup of EAL debug messages alone ?

REgards,
Shanmugasundaram M

<snip3>
ebug interface eth3 (port 3) is not operative
2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
packet m=0x36575b600 dropped at fp_process_input():178
2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: info
fp_ether_input(ifp=eth1 port=1)
2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
==== mbuf at [0x36866ee80]: 1 segment(s), len=60 port=1
2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
                 next=(nil) shared=0 flags=0x0
2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
    ol flags: 0x0 hash: 0x0
2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
    outer offloads:
2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
        l2: proto ETH hlen 14 flags



On Wed, Mar 10, 2021 at 1:59 AM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Wed, 10 Mar 2021 00:30:24 +0530
> Shanmugasundaram M <shansirtt@gmail.com> wrote:
>
> > Dear Team,
> >
> > we are using 18.11.1-6WIND DPDK. we are trying to get EAL Debug messages
> > and we are not able to see any one from them in /var/log/messages. Kindly
> > let me know --log-level=8 is enough or --log-level=*8 is correct  OR
> > --log-level=eal,debug is correct ?
> > Appreciate your quick response.
> >
> > regards,
> > Shanmugasundaram M
>
> DPDK now has log-level on a per-feature level. The old log level was
> global.
> The legacy option is what you are using.
>
> The document is missing about log-level options.
>
> Now log-level takes three forms:
>     --log-level=regex,N
>     --log-level=pattern:N
>     --log-level=N
>
> The first takes a regular expressions so for example:
>      -log-level='pmd\..*,8'
> Would set logging at debug for Poll Mode Drivers.  Regex period
> is match all, not match period.
>
> The second form takes a shell style pattern (glob):
>     --log-level='pmd.net.*:debug'
> Sets debug log level for all Poll Mode Network drivers.
> All forms take numeric or symbolic names for logging levels in current
> versions
>
> The third form sets the global log level only.
>
> The actual logic around what gets logged or not is a union of
> the global and per-logtype argument. So if global log level is
> set high (like NOTICE) then no info or debug messages will get printed.
>
> In other words, --log-level=8 has no effect both because it only
> sets the global level (and the default global level already is 8).
>
> To turn on all logging. Just do --log-level='.*,8' (or similar)
>
>
>
>
>

Re: [dpdk-dev] Reg DPDK EAL log-level=8 options is not throwing any DEBUG messages in /var/log/messages

[Stephen Hemminger]

On Wed, 10 Mar 2021 23:49:17 +0530
Shanmugasundaram M <shansirtt@gmail.com> wrote:

> THanks Steven for the quick response.  I want to enable  EAL module
> (complete logging at debug level ) only during bootup of fastpath. Once
> bootup is done, i want to move this module log-level to default/ original.
> Currently I'm using     --log-level='.*',8  along with
> ./sources/fp/common/fp-var.h  [ #define FP_LOG_DEFAULT FP_LOG_DEBUG ]
> instead of [ #define FP_LOG_DEFAULT FP_LOG_ERR ]. refer below snip
> <snip>
> -#define FP_LOG_DEFAULT FP_LOG_ERR
> +#define FP_LOG_DEFAULT FP_LOG_DEBUG
> The problem is its throwing all the other debug messages like this after
> fastpath started. Any way to disable below message <snip3> and only log
> bootup of EAL debug messages alone ?
> 
> REgards,
> Shanmugasundaram M
> 
> <snip3>
> ebug interface eth3 (port 3) is not operative
> 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
> packet m=0x36575b600 dropped at fp_process_input():178
> 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: info
> fp_ether_input(ifp=eth1 port=1)
> 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
> ==== mbuf at [0x36866ee80]: 1 segment(s), len=60 port=1
> 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
>                  next=(nil) shared=0 flags=0x0
> 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
>     ol flags: 0x0 hash: 0x0
> 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
>     outer offloads:
> 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
>         l2: proto ETH hlen 14 flags
> 
> 
> 
> On Wed, Mar 10, 2021 at 1:59 AM Stephen Hemminger <
> stephen@networkplumber.org> wrote:  
> 
> > On Wed, 10 Mar 2021 00:30:24 +0530
> > Shanmugasundaram M <shansirtt@gmail.com> wrote:
> >  
> > > Dear Team,
> > >
> > > we are using 18.11.1-6WIND DPDK. we are trying to get EAL Debug messages
> > > and we are not able to see any one from them in /var/log/messages. Kindly
> > > let me know --log-level=8 is enough or --log-level=*8 is correct  OR
> > > --log-level=eal,debug is correct ?
> > > Appreciate your quick response.
> > >
> > > regards,
> > > Shanmugasundaram M  
> >
> > DPDK now has log-level on a per-feature level. The old log level was
> > global.
> > The legacy option is what you are using.
> >
> > The document is missing about log-level options.
> >
> > Now log-level takes three forms:
> >     --log-level=regex,N
> >     --log-level=pattern:N
> >     --log-level=N
> >
> > The first takes a regular expressions so for example:
> >      -log-level='pmd\..*,8'
> > Would set logging at debug for Poll Mode Drivers.  Regex period
> > is match all, not match period.
> >
> > The second form takes a shell style pattern (glob):
> >     --log-level='pmd.net.*:debug'
> > Sets debug log level for all Poll Mode Network drivers.
> > All forms take numeric or symbolic names for logging levels in current
> > versions
> >
> > The third form sets the global log level only.
> >
> > The actual logic around what gets logged or not is a union of
> > the global and per-logtype argument. So if global log level is
> > set high (like NOTICE) then no info or debug messages will get printed.
> >
> > In other words, --log-level=8 has no effect both because it only
> > sets the global level (and the default global level already is 8).
> >
> > To turn on all logging. Just do --log-level='.*,8' (or similar)
> >
> >
> >
> >
> >  

If you know what you want, then use a more restrictive log-level to get
just the driver or a particular message type.

Re: [dpdk-dev] Reg DPDK EAL log-level=8 options is not throwing any DEBUG messages in /var/log/messages

[Shanmugasundaram M]

can i use

./sources/dpdk-18.11/drivers/net/vmxnet3/vmxnet3_ethdev.c:1422:
rte_log_set_level(vmxnet3_logtype_driver, RTE_LOG_WARNING);
after  EAL driver routine completes ( pci_common.c, etc) its initialization
? Please let me know where do i need  to call  rte_log_set_level() in EAL
driver init completion function?

regards,
Shanmugasundaram M

On Wed, Mar 10, 2021 at 11:55 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Wed, 10 Mar 2021 23:49:17 +0530
> Shanmugasundaram M <shansirtt@gmail.com> wrote:
>
> > THanks Steven for the quick response.  I want to enable  EAL module
> > (complete logging at debug level ) only during bootup of fastpath. Once
> > bootup is done, i want to move this module log-level to default/
> original.
> > Currently I'm using     --log-level='.*',8  along with
> > ./sources/fp/common/fp-var.h  [ #define FP_LOG_DEFAULT FP_LOG_DEBUG ]
> > instead of [ #define FP_LOG_DEFAULT FP_LOG_ERR ]. refer below snip
> > <snip>
> > -#define FP_LOG_DEFAULT FP_LOG_ERR
> > +#define FP_LOG_DEFAULT FP_LOG_DEBUG
> > The problem is its throwing all the other debug messages like this after
> > fastpath started. Any way to disable below message <snip3> and only log
> > bootup of EAL debug messages alone ?
> >
> > REgards,
> > Shanmugasundaram M
> >
> > <snip3>
> > ebug interface eth3 (port 3) is not operative
> > 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
> > packet m=0x36575b600 dropped at fp_process_input():178
> > 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: info
> > fp_ether_input(ifp=eth1 port=1)
> > 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
> > ==== mbuf at [0x36866ee80]: 1 segment(s), len=60 port=1
> > 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
> >                  next=(nil) shared=0 flags=0x0
> > 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
> >     ol flags: 0x0 hash: 0x0
> > 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
> >     outer offloads:
> > 2021-03-10T18:12:39+00:00 daemon infoblox.localdomain fp-rte[3611]: debug
> >         l2: proto ETH hlen 14 flags
> >
> >
> >
> > On Wed, Mar 10, 2021 at 1:59 AM Stephen Hemminger <
> > stephen@networkplumber.org> wrote:
> >
> > > On Wed, 10 Mar 2021 00:30:24 +0530
> > > Shanmugasundaram M <shansirtt@gmail.com> wrote:
> > >
> > > > Dear Team,
> > > >
> > > > we are using 18.11.1-6WIND DPDK. we are trying to get EAL Debug
> messages
> > > > and we are not able to see any one from them in /var/log/messages.
> Kindly
> > > > let me know --log-level=8 is enough or --log-level=*8 is correct  OR
> > > > --log-level=eal,debug is correct ?
> > > > Appreciate your quick response.
> > > >
> > > > regards,
> > > > Shanmugasundaram M
> > >
> > > DPDK now has log-level on a per-feature level. The old log level was
> > > global.
> > > The legacy option is what you are using.
> > >
> > > The document is missing about log-level options.
> > >
> > > Now log-level takes three forms:
> > >     --log-level=regex,N
> > >     --log-level=pattern:N
> > >     --log-level=N
> > >
> > > The first takes a regular expressions so for example:
> > >      -log-level='pmd\..*,8'
> > > Would set logging at debug for Poll Mode Drivers.  Regex period
> > > is match all, not match period.
> > >
> > > The second form takes a shell style pattern (glob):
> > >     --log-level='pmd.net.*:debug'
> > > Sets debug log level for all Poll Mode Network drivers.
> > > All forms take numeric or symbolic names for logging levels in current
> > > versions
> > >
> > > The third form sets the global log level only.
> > >
> > > The actual logic around what gets logged or not is a union of
> > > the global and per-logtype argument. So if global log level is
> > > set high (like NOTICE) then no info or debug messages will get printed.
> > >
> > > In other words, --log-level=8 has no effect both because it only
> > > sets the global level (and the default global level already is 8).
> > >
> > > To turn on all logging. Just do --log-level='.*,8' (or similar)
> > >
> > >
> > >
> > >
> > >
>
> If you know what you want, then use a more restrictive log-level to get
> just the driver or a particular message type.
>

[dpdk-dev] Reg 6Wind fastpath DPDK 18.11.1 virtual memory allocation ( 54.5GB of anonymous mapping)

[Shanmugasundaram M]

 Dear Team,

We are using  6Wind fastpath(fp-rte) which is using DPDK version 18.11.1.
We are seeing  65.5G(preallocated during startup) of virtual memory for
fp-rte on a 32GB machine, 10G of that was huge pages (5000 mappings of 2M.)

We found 4 strange anon (4K size) segments totaling 54.5G (16+16+16+6.5).
Note: we changed / customized memory CONFIG_RTE_MAX_MEMSEG_PER_LIST=24576
and default value of  CONFIG_RTE_MAX_MEMSEG_LISTS=64 used.
Kindly let us know
1. Why are allocating  54.5 GB anonymous mapping
2. When will we use it ?
Appreciate your quick response.

Regards,
Shanmugasundaram M

Re: [dpdk-dev] Reg 6Wind fastpath DPDK 18.11.1 virtual memory allocation ( 54.5GB of anonymous mapping)

[David Marchand]

On Wed, Jun 9, 2021 at 2:02 PM Shanmugasundaram M <shansirtt@gmail.com> wrote:
> We are using  6Wind fastpath(fp-rte) which is using DPDK version 18.11.1.
> We are seeing  65.5G(preallocated during startup) of virtual memory for
> fp-rte on a 32GB machine, 10G of that was huge pages (5000 mappings of 2M.)
>
> We found 4 strange anon (4K size) segments totaling 54.5G (16+16+16+6.5).
> Note: we changed / customized memory CONFIG_RTE_MAX_MEMSEG_PER_LIST=24576
> and default value of  CONFIG_RTE_MAX_MEMSEG_LISTS=64 used.
> Kindly let us know
> 1. Why are allocating  54.5 GB anonymous mapping
> 2. When will we use it ?

This is likely pre-reserved virtual memory from the DPDK allocator.
This virtual memory won't probably never be used.

But you'd better ask 6WIND guys as they maintain their own DPDK downstream.

Copying a few guys.

-- 
David Marchand

Re: [dpdk-dev] Reg 6Wind fastpath DPDK 18.11.1 virtual memory allocation ( 54.5GB of anonymous mapping)

[Bruce Richardson]

On Wed, Jun 09, 2021 at 02:28:04PM +0200, David Marchand wrote:
> On Wed, Jun 9, 2021 at 2:02 PM Shanmugasundaram M <shansirtt@gmail.com> wrote:
> > We are using  6Wind fastpath(fp-rte) which is using DPDK version 18.11.1.
> > We are seeing  65.5G(preallocated during startup) of virtual memory for
> > fp-rte on a 32GB machine, 10G of that was huge pages (5000 mappings of 2M.)
> >
> > We found 4 strange anon (4K size) segments totaling 54.5G (16+16+16+6.5).
> > Note: we changed / customized memory CONFIG_RTE_MAX_MEMSEG_PER_LIST=24576
> > and default value of  CONFIG_RTE_MAX_MEMSEG_LISTS=64 used.
> > Kindly let us know
> > 1. Why are allocating  54.5 GB anonymous mapping
> > 2. When will we use it ?
> 
> This is likely pre-reserved virtual memory from the DPDK allocator.
> This virtual memory won't probably never be used.
> 

There may not be actually any memory backing this, since DPDK will reserve
huge amounts of virtual address space without actually using it. So those
GB's you see in the virtual memory may be completely unused, the addresses
are just reserved to ensure that if memory does need to be mapped into the
process, we have an appropriate contiguous address block to put it in.

> But you'd better ask 6WIND guys as they maintain their own DPDK downstream.
> 
> Copying a few guys.
> 
> -- 
> David Marchand
> 

Re: [dpdk-dev] Reg 6Wind fastpath DPDK 18.11.1 virtual memory allocation ( 54.5GB of anonymous mapping)

[Olivier Matz]

Hi Shanmugasundaram,

On Wed, Jun 09, 2021 at 01:46:35PM +0100, Bruce Richardson wrote:
> On Wed, Jun 09, 2021 at 02:28:04PM +0200, David Marchand wrote:
> > On Wed, Jun 9, 2021 at 2:02 PM Shanmugasundaram M <shansirtt@gmail.com> wrote:
> > > We are using  6Wind fastpath(fp-rte) which is using DPDK version 18.11.1.
> > > We are seeing  65.5G(preallocated during startup) of virtual memory for
> > > fp-rte on a 32GB machine, 10G of that was huge pages (5000 mappings of 2M.)
> > >
> > > We found 4 strange anon (4K size) segments totaling 54.5G (16+16+16+6.5).
> > > Note: we changed / customized memory CONFIG_RTE_MAX_MEMSEG_PER_LIST=24576
> > > and default value of  CONFIG_RTE_MAX_MEMSEG_LISTS=64 used.
> > > Kindly let us know
> > > 1. Why are allocating  54.5 GB anonymous mapping
> > > 2. When will we use it ?
> > 
> > This is likely pre-reserved virtual memory from the DPDK allocator.
> > This virtual memory won't probably never be used.
> > 
> 
> There may not be actually any memory backing this, since DPDK will reserve
> huge amounts of virtual address space without actually using it. So those
> GB's you see in the virtual memory may be completely unused, the addresses
> are just reserved to ensure that if memory does need to be mapped into the
> process, we have an appropriate contiguous address block to put it in.
> 
> > But you'd better ask 6WIND guys as they maintain their own DPDK downstream.
> > 
> > Copying a few guys.

I agree with David and Bruce. This is probably related to virtual memory
areas reserved by eal_get_virtual_area() (mmap of /dev/zero). Enabling EAL
debug logs should show traces about it.

If you need support (even on old 6WINDGate versions based on
dpdk-18.11), please open a ticket at 6WIND support, we'll be happy to
help.

Regards,
Olivier