[dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Matthew Hall]

Hello,

I ran into some weird symbol conflicts between system netinet/in.h and DPDK 
rte_ip.h. They have a lot of duplicated definitions for stuff like IPPROTO_IP 
and so on. This breaks when you want to use inet_pton from arpa/inet.h, 
because it includes netinet/in.h to define struct in_addr.

Thus with all the conflicts it's impossible to use a DPDK IP struct instead of 
all the system's sockaddr stuff, to store a value from the system copy of 
inet_pton. This would be a common operation if, for example, you want to 
configure all the IP addresses on your box from a JSON file, which is what I 
was doing.

The DPDK kludged around it internally by using a file called 
cmdline_parse_ipaddr.c with private copies of these functions. But it in my 
opinion very unwisely marked all of the functions as static except for 
cmdline_parse_ipaddr, which only works on the DPDK's proprietary argument 
handling, and not with anything the user might have which is a different 
format.

So, it would be a big help for users if the macros in librte_net files would 
check if the symbols already existed, or if they had subheader files available 
to grab only non conflicting symbols, or if they would make a proper .h and 
factor all the inet_pton and inet_ntop inside the cmdline lib into a place 
where users can access them. It would also be a help if they had a less ugly 
equivalent to struct sockaddr, which let you work with IP addresses a bit more 
easily, such as something like this:

struct ip4_addr {
    uint32_t addr;
};

typedef struct ip4_addr ip4_addr;

struct ip6_addr {
    uint8_t addr[16];
};

typedef struct ip6_addr ip6_addr;

struct ip_addr {
    uint8_t family;
    uint8_t prefix;
    union {
        struct ip4_addr ipv4;
        struct ip6_addr ipv6;
    };
};

I had to create a bunch of duplicate code to handle it in my project, since 
the DPDK marked its copies of all these functions as "secret" and didn't make 
a .h for them. If any of it is useful I am happy to donate it, although I 
don't think I've got quite enough experience with this specifc part of the 
DPDK to code it up all by myself.

Thanks,
Matthew.

Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Niraj Sharma (nirajsha)]

I also noticed this problem. It is a serious one. I would like it solved.

-- Niraj Sharma

Principal Eng.
Cisco System, Inc.


On 7/24/14 12:59 AM, "Matthew Hall" <mhall@mhcomputing.net> wrote:

>Hello,
>
>I ran into some weird symbol conflicts between system netinet/in.h and
>DPDK 
>rte_ip.h. They have a lot of duplicated definitions for stuff like
>IPPROTO_IP 
>and so on. This breaks when you want to use inet_pton from arpa/inet.h,
>because it includes netinet/in.h to define struct in_addr.
>
>Thus with all the conflicts it's impossible to use a DPDK IP struct
>instead of 
>all the system's sockaddr stuff, to store a value from the system copy of
>inet_pton. This would be a common operation if, for example, you want to
>configure all the IP addresses on your box from a JSON file, which is
>what I 
>was doing.
>
>The DPDK kludged around it internally by using a file called
>cmdline_parse_ipaddr.c with private copies of these functions. But it in
>my 
>opinion very unwisely marked all of the functions as static except for
>cmdline_parse_ipaddr, which only works on the DPDK's proprietary argument
>handling, and not with anything the user might have which is a different
>format.
>
>So, it would be a big help for users if the macros in librte_net files
>would 
>check if the symbols already existed, or if they had subheader files
>available 
>to grab only non conflicting symbols, or if they would make a proper .h
>and 
>factor all the inet_pton and inet_ntop inside the cmdline lib into a
>place 
>where users can access them. It would also be a help if they had a less
>ugly 
>equivalent to struct sockaddr, which let you work with IP addresses a bit
>more 
>easily, such as something like this:
>
>struct ip4_addr {
>    uint32_t addr;
>};
>
>typedef struct ip4_addr ip4_addr;
>
>struct ip6_addr {
>    uint8_t addr[16];
>};
>
>typedef struct ip6_addr ip6_addr;
>
>struct ip_addr {
>    uint8_t family;
>    uint8_t prefix;
>    union {
>        struct ip4_addr ipv4;
>        struct ip6_addr ipv6;
>    };
>};
>
>I had to create a bunch of duplicate code to handle it in my project,
>since 
>the DPDK marked its copies of all these functions as "secret" and didn't
>make 
>a .h for them. If any of it is useful I am happy to donate it, although I
>don't think I've got quite enough experience with this specifc part of
>the 
>DPDK to code it up all by myself.
>
>Thanks,
>Matthew.

Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Antti Kantee]

On 24/07/14 07:59, Matthew Hall wrote:
> Hello,
>
> I ran into some weird symbol conflicts between system netinet/in.h and DPDK
> rte_ip.h. They have a lot of duplicated definitions for stuff like IPPROTO_IP
> and so on. This breaks when you want to use inet_pton from arpa/inet.h,
> because it includes netinet/in.h to define struct in_addr.

I would namespace the definitions in DPDK, i.e. make them 
DPDK_IPPROTO_FOO etc.

> Thus with all the conflicts it's impossible to use a DPDK IP struct instead of
> all the system's sockaddr stuff, to store a value from the system copy of
> inet_pton. This would be a common operation if, for example, you want to
> configure all the IP addresses on your box from a JSON file, which is what I
> was doing.
>
> The DPDK kludged around it internally by using a file called
> cmdline_parse_ipaddr.c with private copies of these functions. But it in my
> opinion very unwisely marked all of the functions as static except for
> cmdline_parse_ipaddr, which only works on the DPDK's proprietary argument
> handling, and not with anything the user might have which is a different
> format.

In my experience from years of fighting with more or less this exact 
same problem -- the fight is now thankfully over but the scars remain -- 
you either want to expose a complete set of types and provide support 
for everything, or you want to expose nothing.  Approaches where you use 
cute definitions and reuse some host routines is like asking for an 
audience with Tyranthraxus when armed with a kitten.  It's that doubly 
so if you don't have to and do it anyway.

> So, it would be a big help for users if the macros in librte_net files would
> check if the symbols already existed, or if they had subheader files available
> to grab only non conflicting symbols, or if they would make a proper .h and
> factor all the inet_pton and inet_ntop inside the cmdline lib into a place
> where users can access them. It would also be a help if they had a less ugly
> equivalent to struct sockaddr, which let you work with IP addresses a bit more
> easily, such as something like this:

Again, I recommend steering away from any tightrope approaches that 
"know" which types are non-conflicting, or pick out half-and-half from 
the host and IP stack.  "Do, or do not, there is no half-and-half"

Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Matthew Hall]

On Thu, Jul 24, 2014 at 10:55:59PM +0000, Antti Kantee wrote:
> In my experience from years of fighting with more or less this exact same
> problem -- the fight is now thankfully over but the scars remain -- you
> either want to expose a complete set of types and provide support for
> everything, or you want to expose nothing.

I don't have a problem with this approach. Implementing it would require the 
DPDK to create user accessible functions for creating MAC addresses, IPs, 
CIDRs, and TCP / UDP port numbers. Many of the things required are hiding 
inside of *cmdline* code where it's impossible for anybody to access them.

Matthew.

Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Wu, Jingjing]

Hello,

We also notice these conflicts, we just planned to fix it in our new feature development. The proposal is like:

#ifndef _NETINET_IN_H
#ifndef _NETINET_IN_H_

#define IPPROTO_IP     0
 ... ... 
#define IPPROTO_MAX  256

#endif
#endif

Do you think it is a good idea?

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Antti Kantee
> Sent: Friday, July 25, 2014 6:56 AM
> To: Matthew Hall; dev@dpdk.org
> Subject: Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h
> 
> On 24/07/14 07:59, Matthew Hall wrote:
> > Hello,
> >
> > I ran into some weird symbol conflicts between system netinet/in.h and DPDK
> > rte_ip.h. They have a lot of duplicated definitions for stuff like IPPROTO_IP
> > and so on. This breaks when you want to use inet_pton from arpa/inet.h,
> > because it includes netinet/in.h to define struct in_addr.
> 
> I would namespace the definitions in DPDK, i.e. make them
> DPDK_IPPROTO_FOO etc.
> 
> > Thus with all the conflicts it's impossible to use a DPDK IP struct instead of
> > all the system's sockaddr stuff, to store a value from the system copy of
> > inet_pton. This would be a common operation if, for example, you want to
> > configure all the IP addresses on your box from a JSON file, which is what I
> > was doing.
> >
> > The DPDK kludged around it internally by using a file called
> > cmdline_parse_ipaddr.c with private copies of these functions. But it in my
> > opinion very unwisely marked all of the functions as static except for
> > cmdline_parse_ipaddr, which only works on the DPDK's proprietary argument
> > handling, and not with anything the user might have which is a different
> > format.
> 
> In my experience from years of fighting with more or less this exact
> same problem -- the fight is now thankfully over but the scars remain --
> you either want to expose a complete set of types and provide support
> for everything, or you want to expose nothing.  Approaches where you use
> cute definitions and reuse some host routines is like asking for an
> audience with Tyranthraxus when armed with a kitten.  It's that doubly
> so if you don't have to and do it anyway.
> 
> > So, it would be a big help for users if the macros in librte_net files would
> > check if the symbols already existed, or if they had subheader files available
> > to grab only non conflicting symbols, or if they would make a proper .h and
> > factor all the inet_pton and inet_ntop inside the cmdline lib into a place
> > where users can access them. It would also be a help if they had a less ugly
> > equivalent to struct sockaddr, which let you work with IP addresses a bit more
> > easily, such as something like this:
> 
> Again, I recommend steering away from any tightrope approaches that
> "know" which types are non-conflicting, or pick out half-and-half from
> the host and IP stack.  "Do, or do not, there is no half-and-half"

Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Matthew Hall]

I don't know if it will work right on both Linux and BSD and/or if they also 100% agree with the libc / glibc values compiled into the system's .so files. That's the risk that you run if you don't have more complete support in the DPDK itself for these features.
-- 
Sent from my mobile device.

On July 24, 2014 6:12:18 PM PDT, "Wu, Jingjing" <jingjing.wu@intel.com> wrote:
>Hello,
>
>We also notice these conflicts, we just planned to fix it in our new
>feature development. The proposal is like:
>
>#ifndef _NETINET_IN_H
>#ifndef _NETINET_IN_H_
>
>#define IPPROTO_IP     0
> ... ... 
>#define IPPROTO_MAX  256
>
>#endif
>#endif
>
>Do you think it is a good idea?
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Antti Kantee
>> Sent: Friday, July 25, 2014 6:56 AM
>> To: Matthew Hall; dev@dpdk.org
>> Subject: Re: [dpdk-dev] symbol conflicts between netinet/in.h,
>arpa/inet.h, and rte_ip.h
>> 
>> On 24/07/14 07:59, Matthew Hall wrote:
>> > Hello,
>> >
>> > I ran into some weird symbol conflicts between system netinet/in.h
>and DPDK
>> > rte_ip.h. They have a lot of duplicated definitions for stuff like
>IPPROTO_IP
>> > and so on. This breaks when you want to use inet_pton from
>arpa/inet.h,
>> > because it includes netinet/in.h to define struct in_addr.
>> 
>> I would namespace the definitions in DPDK, i.e. make them
>> DPDK_IPPROTO_FOO etc.
>> 
>> > Thus with all the conflicts it's impossible to use a DPDK IP struct
>instead of
>> > all the system's sockaddr stuff, to store a value from the system
>copy of
>> > inet_pton. This would be a common operation if, for example, you
>want to
>> > configure all the IP addresses on your box from a JSON file, which
>is what I
>> > was doing.
>> >
>> > The DPDK kludged around it internally by using a file called
>> > cmdline_parse_ipaddr.c with private copies of these functions. But
>it in my
>> > opinion very unwisely marked all of the functions as static except
>for
>> > cmdline_parse_ipaddr, which only works on the DPDK's proprietary
>argument
>> > handling, and not with anything the user might have which is a
>different
>> > format.
>> 
>> In my experience from years of fighting with more or less this exact
>> same problem -- the fight is now thankfully over but the scars remain
>--
>> you either want to expose a complete set of types and provide support
>> for everything, or you want to expose nothing.  Approaches where you
>use
>> cute definitions and reuse some host routines is like asking for an
>> audience with Tyranthraxus when armed with a kitten.  It's that
>doubly
>> so if you don't have to and do it anyway.
>> 
>> > So, it would be a big help for users if the macros in librte_net
>files would
>> > check if the symbols already existed, or if they had subheader
>files available
>> > to grab only non conflicting symbols, or if they would make a
>proper .h and
>> > factor all the inet_pton and inet_ntop inside the cmdline lib into
>a place
>> > where users can access them. It would also be a help if they had a
>less ugly
>> > equivalent to struct sockaddr, which let you work with IP addresses
>a bit more
>> > easily, such as something like this:
>> 
>> Again, I recommend steering away from any tightrope approaches that
>> "know" which types are non-conflicting, or pick out half-and-half
>from
>> the host and IP stack.  "Do, or do not, there is no half-and-half"

Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Ananyev, Konstantin]

> 
> I don't know if it will work right on both Linux and BSD and/or if they also 100% agree with the libc / glibc values compiled into the
> system's .so files. That's the risk that you run if you don't have more complete support in the DPDK itself for these features.

Looking at linux and freebsd netinet/in.h files - I think it should work.
But I suppose we can test it on both freebsd and linux before submitting a patch.

> --
> Sent from my mobile device.
> 
> On July 24, 2014 6:12:18 PM PDT, "Wu, Jingjing" <jingjing.wu@intel.com> wrote:
> >Hello,
> >
> >We also notice these conflicts, we just planned to fix it in our new
> >feature development. The proposal is like:
> >
> >#ifndef _NETINET_IN_H
> >#ifndef _NETINET_IN_H_
> >
> >#define IPPROTO_IP     0
> > ... ...
> >#define IPPROTO_MAX  256
> >
> >#endif
> >#endif
> >
> >Do you think it is a good idea?
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Antti Kantee
> >> Sent: Friday, July 25, 2014 6:56 AM
> >> To: Matthew Hall; dev@dpdk.org
> >> Subject: Re: [dpdk-dev] symbol conflicts between netinet/in.h,
> >arpa/inet.h, and rte_ip.h
> >>
> >> On 24/07/14 07:59, Matthew Hall wrote:
> >> > Hello,
> >> >
> >> > I ran into some weird symbol conflicts between system netinet/in.h
> >and DPDK
> >> > rte_ip.h. They have a lot of duplicated definitions for stuff like
> >IPPROTO_IP
> >> > and so on. This breaks when you want to use inet_pton from
> >arpa/inet.h,
> >> > because it includes netinet/in.h to define struct in_addr.
> >>
> >> I would namespace the definitions in DPDK, i.e. make them
> >> DPDK_IPPROTO_FOO etc.
> >>
> >> > Thus with all the conflicts it's impossible to use a DPDK IP struct
> >instead of
> >> > all the system's sockaddr stuff, to store a value from the system
> >copy of
> >> > inet_pton. This would be a common operation if, for example, you
> >want to
> >> > configure all the IP addresses on your box from a JSON file, which
> >is what I
> >> > was doing.
> >> >
> >> > The DPDK kludged around it internally by using a file called
> >> > cmdline_parse_ipaddr.c with private copies of these functions. But
> >it in my
> >> > opinion very unwisely marked all of the functions as static except
> >for
> >> > cmdline_parse_ipaddr, which only works on the DPDK's proprietary
> >argument
> >> > handling, and not with anything the user might have which is a
> >different
> >> > format.
> >>
> >> In my experience from years of fighting with more or less this exact
> >> same problem -- the fight is now thankfully over but the scars remain
> >--
> >> you either want to expose a complete set of types and provide support
> >> for everything, or you want to expose nothing.  Approaches where you
> >use
> >> cute definitions and reuse some host routines is like asking for an
> >> audience with Tyranthraxus when armed with a kitten.  It's that
> >doubly
> >> so if you don't have to and do it anyway.
> >>
> >> > So, it would be a big help for users if the macros in librte_net
> >files would
> >> > check if the symbols already existed, or if they had subheader
> >files available
> >> > to grab only non conflicting symbols, or if they would make a
> >proper .h and
> >> > factor all the inet_pton and inet_ntop inside the cmdline lib into
> >a place
> >> > where users can access them. It would also be a help if they had a
> >less ugly
> >> > equivalent to struct sockaddr, which let you work with IP addresses
> >a bit more
> >> > easily, such as something like this:
> >>
> >> Again, I recommend steering away from any tightrope approaches that
> >> "know" which types are non-conflicting, or pick out half-and-half
> >from
> >> the host and IP stack.  "Do, or do not, there is no half-and-half"

Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Thomas Monjalon]

Hi all,

2014-07-24 22:55, Antti Kantee:
> On 24/07/14 07:59, Matthew Hall wrote:
> > I ran into some weird symbol conflicts between system netinet/in.h and DPDK
> > rte_ip.h. They have a lot of duplicated definitions for stuff like IPPROTO_IP
> > and so on. This breaks when you want to use inet_pton from arpa/inet.h,
> > because it includes netinet/in.h to define struct in_addr.
[...]
> Again, I recommend steering away from any tightrope approaches that 
> "know" which types are non-conflicting, or pick out half-and-half from 
> the host and IP stack.  "Do, or do not, there is no half-and-half"

The general problem here is that DPDK is conflicting with libc.
So the obvious question would be: "why DPDK needs to redefine libc stuff"?
I don't see any obvious answer since bare metal is planned to be removed.
(see http://dpdk.org/ml/archives/dev/2014-June/003868.html)

Objections?

-- 
Thomas

Re: [dpdk-dev] symbol conflicts between netinet/in.h, arpa/inet.h, and rte_ip.h

[Antti Kantee]

On 25/07/14 10:43, Thomas Monjalon wrote:
>> On 24/07/14 07:59, Matthew Hall wrote:
>>> I ran into some weird symbol conflicts between system netinet/in.h and DPDK
>>> rte_ip.h. They have a lot of duplicated definitions for stuff like IPPROTO_IP
>>> and so on. This breaks when you want to use inet_pton from arpa/inet.h,
>>> because it includes netinet/in.h to define struct in_addr.
> [...]
>> Again, I recommend steering away from any tightrope approaches that
>> "know" which types are non-conflicting, or pick out half-and-half from
>> the host and IP stack.  "Do, or do not, there is no half-and-half"
>
> The general problem here is that DPDK is conflicting with libc.
> So the obvious question would be: "why DPDK needs to redefine libc stuff"?
> I don't see any obvious answer since bare metal is planned to be removed.
> (see http://dpdk.org/ml/archives/dev/2014-June/003868.html)

One reason is if you want DPDK to be a portable network programming 
environment.  Especially in that case you do not want definitions based 
on hackish assumptions of some particular version of some particular 
host implementation.  However, I'm not trying to argue if DPDK should or 
shouldn't be that, just that you should either dramatically improve the 
current implementation or nuke it.