From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <cian.ferriter@intel.com>
Received: from mga03.intel.com (mga03.intel.com [134.134.136.65])
by dpdk.org (Postfix) with ESMTP id 109645A6F
for <dev@dpdk.org>; Mon, 19 Jan 2015 19:39:04 +0100 (CET)
Received: from orsmga001.jf.intel.com ([10.7.209.18])
by orsmga103.jf.intel.com with ESMTP; 19 Jan 2015 10:34:39 -0800
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.09,428,1418112000"; d="scan'208";a="639451032"
Received: from irsmsx153.ger.corp.intel.com ([163.33.192.75])
by orsmga001.jf.intel.com with ESMTP; 19 Jan 2015 10:38:33 -0800
Received: from irsmsx108.ger.corp.intel.com ([169.254.11.64]) by
IRSMSX153.ger.corp.intel.com ([169.254.9.2]) with mapi id 14.03.0195.001;
Mon, 19 Jan 2015 18:38:32 +0000
From: "Ferriter, Cian" <cian.ferriter@intel.com>
To: "Richardson, Bruce" <bruce.richardson@intel.com>
Thread-Topic: [dpdk-dev] [PATCH] lib/librte_ether: change socket_id passed
to rte_memzone_reserve
Thread-Index: AQHQLxJxCtZNxstVx0ajqB3SBqs+d5y9xuPQgABMPYCAADmXQIABGZmAgAhk2eA=
Date: Mon, 19 Jan 2015 18:38:32 +0000
Message-ID: <D9B9B7EDAA4D17468E64B453458633E4044F605A@IRSMSX108.ger.corp.intel.com>
References: <1421140920-9964-1-git-send-email-cian.ferriter@intel.com>
<D9B9B7EDAA4D17468E64B453458633E4044E9D21@IRSMSX108.ger.corp.intel.com>
<20150113135549.GA3460@bricha3-MOBL3>
<D9B9B7EDAA4D17468E64B453458633E4044EA5E5@IRSMSX108.ger.corp.intel.com>
<20150114100946.GA10476@bricha3-MOBL3>
In-Reply-To: <20150114100946.GA10476@bricha3-MOBL3>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-originating-ip: [163.33.239.181]
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
Cc: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH] lib/librte_ether: change socket_id passed to
rte_memzone_reserve
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Mon, 19 Jan 2015 18:39:05 -0000
I would be happy with the original suggestion. If the ethdev data for a por=
t in use is in cache it removes the performance concern associated the curr=
ent setup and my fix. The original suggestion also fixes the crash that I w=
as seeing because of memory being reserved from a numa node with no "--sock=
et-mem" allocated.
Cian
-----Original Message-----
From: Richardson, Bruce=20
Sent: Wednesday, January 14, 2015 10:10 AM
To: Ferriter, Cian
Cc: dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH] lib/librte_ether: change socket_id passed t=
o rte_memzone_reserve
On Tue, Jan 13, 2015 at 06:05:25PM +0000, Ferriter, Cian wrote:
> Comments on alternative solutions:
> 1) how would this solution work when there is no NIC present, and "rte_et=
h_from_rings" is called? Here, could you have an else where the socket id o=
f the master core is passed to the "memzone_reserve"?
> 2) how would you advise making this change? I have looked at where "rte_e=
th_dev_allocate" is being called and in all but one case, there is a "numa_=
id" that could be passed in. This isn't the case for " rte_eth_dev_init" ho=
wever, is there an easy solution for this? Would there now need to be an "r=
te_eth_dev_data" struct for each socket that there is a NIC attached to, re=
serving memory from that socket?
>=20
> Cian
While I think the issues you highlight can probably be overcome, I'm not so=
sure any more how much it matters what numa node this is allocated on. The=
ethdev data for any port in use by a port should be in the cache. In that =
case, if it doesn't matter, your original suggestion would work fine.
/Bruce
>=20
> -----Original Message-----
> From: Richardson, Bruce
> Sent: Tuesday, January 13, 2015 1:56 PM
> To: Ferriter, Cian
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] lib/librte_ether: change socket_id=20
> passed to rte_memzone_reserve
>=20
> On Tue, Jan 13, 2015 at 09:23:16AM +0000, Ferriter, Cian wrote:
> > Passing a socket id of "rte_socket_id()" can cause problems in non DPDK=
applications as there is a dependency on the current logical core we are r=
unning on.
> > Passing " rte_lcore_to_socket_id(rte_get_master_lcore())" as the socket=
id to rte_memzone_reserve resolves these issues as the master lcore doesn'=
t change.
> >=20
>=20
> The only trouble is that when affinitizing the memory for the NICs to the=
socket of the master lcore, it gives us no way to correctly configure an a=
pp to use NICs connected to two different sockets on the one system. All me=
mory for all NICs will end up on the same socket. Two possible alternative =
solutions:
> 1) affinitize memory to the socket the NIC is connected to
> 2) add a socket parameter to the API calls to allow the user complete=20
> control over their memory allocations
>=20
> Obviously the second one breaks backward compatibility (assume we modify =
existing API call), but is more powerful.
>=20
> Thoughts?
>=20
> /Bruce
>=20
> > -----Original Message-----
> > From: Ferriter, Cian
> > Sent: Tuesday, January 13, 2015 9:22 AM
> > To: dev@dpdk.org
> > Cc: Ferriter, Cian
> > Subject: [PATCH] lib/librte_ether: change socket_id passed to=20
> > rte_memzone_reserve
> >=20
> > Change the socket id that is passed to rte_memzone_reserve from the soc=
ket id of current logical core to the socket id of the master_lcore.
> > ---
> > lib/librte_ether/rte_ethdev.c | 2 +-
> > 1 files changed, 1 insertions(+), 1 deletions(-) mode change=20
> > 100644 =3D> 100755 lib/librte_ether/rte_ethdev.c
> >=20
> > diff --git a/lib/librte_ether/rte_ethdev.c=20
> > b/lib/librte_ether/rte_ethdev.c old mode 100644 new mode 100755=20
> > index 95f2ceb..835540d
> > --- a/lib/librte_ether/rte_ethdev.c
> > +++ b/lib/librte_ether/rte_ethdev.c
> > @@ -184,7 +184,7 @@ rte_eth_dev_data_alloc(void)
> > if (rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY){
> > mz =3D rte_memzone_reserve(MZ_RTE_ETH_DEV_DATA,
> > RTE_MAX_ETHPORTS * sizeof(*rte_eth_dev_data),
> > - rte_socket_id(), flags);
> > + rte_lcore_to_socket_id(rte_get_master_lcore()), flags);
> > } else
> > mz =3D rte_memzone_lookup(MZ_RTE_ETH_DEV_DATA);
> > if (mz =3D=3D NULL)
> > --
> > 1.7.4.1
> >=20