[dpdk-users] Running DPDK application in Linux init level 3

[dpdk-users] Running DPDK application in Linux init level 3

[Anand Prasad]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 4961 bytes --]

Hi DPDK Team,
  I am trying to run DPDK application in Linux with init level 3 (no gui) and I am observing continuous packet drops on Rx side. But, If I boot Linux in init level 5 (GUI), then I dont see drops.
  I have provided linux command line arguments (for isolating the cpu cores and hugepates etc in "GRUB_CMDLINE_LINUX_DEFAULT" variable, and with those it's working fine in init level 5. But, not sureif these configuration are applicable when I boot in Init level 3?
 To boot in init level 3, I am setting GRUB_CMDLINE_LINUX="3" , is it correct? If not please suggest what is theother method.
Thanks in advance.Anand
From iain.barker@oracle.com  Tue Apr 10 21:11:51 2018
Return-Path: <iain.barker@oracle.com>
Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86])
 by dpdk.org (Postfix) with ESMTP id 86F541BA93
 for <users@dpdk.org>; Tue, 10 Apr 2018 21:11:51 +0200 (CEST)
Received: from pps.filterd (userp2130.oracle.com [127.0.0.1])
 by userp2130.oracle.com (8.16.0.22/8.16.0.22) with SMTP id w3AJAeMm081650
 for <users@dpdk.org>; Tue, 10 Apr 2018 19:11:50 GMT
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com;
 h=mime-version :
 message-id : date : from : sender : to : subject : content-type :
 content-transfer-encoding; s=corp-2017-10-26;
 bh=o2N+ivnkL+n0HRmVycrSrN5od3XW6BmM6zkFBRSRe7E=;
 b=WASuOj+JI6Sn40LDytKI+74B/qlpADsPvviBAd9oogILf1cpvGz/qBEEi4rSQ1C1US5d
 gdDwIZNOXfeNYibGI9xyCLY7T3ajhj3jyE70u81T+QldBSb2VVVyNvHyTPj502rDwL6g
 y50JZyVScCJ+b9UTEXQcNCMyWhonyTX2funr9suZgCx4su5EaZFb8TjZBGlydDhm4aBU
 C91M+QW3mLv5YLLIQfqMGOc7/QDhwe1byuDwhnzKzB7ot1YOjD9lE9VpWpb5f3iiZoO6
 n/GK67SsY2uqmR2SoAqDHIWHiUrk+RIvzzi3tGiBANBUtbdrw5DAyM8FO6iBDr1S5Ahz hA=Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71])
 by userp2130.oracle.com with ESMTP id 2h6ne7bwfm-1
 (version=TLSv1.2 cipherìDHE-RSA-AES256-GCM-SHA384 bits%6 verify=OK)
 for <users@dpdk.org>; Tue, 10 Apr 2018 19:11:50 +0000
Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72])
 by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id w3AJBoi4017581
 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits%6 verify=OK)
 for <users@dpdk.org>; Tue, 10 Apr 2018 19:11:50 GMT
Received: from abhmp0009.oracle.com (abhmp0009.oracle.com [141.146.116.15])
 by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id w3AJBnvU016325
 for <users@dpdk.org>; Tue, 10 Apr 2018 19:11:50 GMT
MIME-Version: 1.0
Message-ID: <64897879-3872-445c-8179-4ea5852e4ecf@default>
Date: Tue, 10 Apr 2018 12:11:49 -0700 (PDT)
From: Iain Barker <iain.barker@oracle.com>
Sender: Iain Barker <iain.barker@oracle.com>
To: users@dpdk.org
X-Priority: 3
X-Mailer: Oracle Beehive Extensions for Outlook 2.0.1.9.1  (1003210) [OL
 15.0.5023.0 (x86)]
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
X-Proofpoint-Virus-Version: vendor=nai engineY00 definitionsˆ59
 signaturesf8698
X-Proofpoint-Spam-Details: rule=notspam policyÞfault score=0 suspectscore=1
 malwarescore=0
 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore™9
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.0.1-1711220000 definitions=main-1804100182
Subject: [dpdk-users] Wrong OFED version for DPDK 17.11.1 ?
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://dpdk.org/ml/options/users>,
 <mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://dpdk.org/ml/listinfo/users>,
 <mailto:users-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Tue, 10 Apr 2018 19:11:51 -0000

Hi all,

I'm trying to build 17.11.1 against libibverbs from OFED 4.2-1.0.0.0 but having some API compatibility issues.

According to the DPDK docs, that's the correct version for DPDK 17.11  
Ref: "Mellanox OFED version: 4.2" in https://dpdk.org/doc/guides-17.11/nics/mlx5.html

But I get multiple errors which suggest the API versions are incompatible.  For example:

           priv->hw_csum =
                  !!(device_attr_ex.device_cap_flags_ex &
                     IBV_DEVICE_RAW_IP_CSUM);

drivers/net/mlx5/mlx5.c:794:21: error: 'struct ibv_device_attr_ex' has no member named 'device_cap_flags_ex'

MLNX_OFED_SRC-4.2-1.0.0.0 contains libibverbs-41mlnx1, and that has include/infiniband/verbs.h with the following typedef:
struct ibv_device_attr_ex {
      struct ibv_device_attr  orig_attr;
      uint32_t          comp_mask;
      struct ibv_odp_caps     odp_caps;
};

So the docs appear to be wrong. no way the OFED 4.2 version of mlx5.c is going to compile against that header.

What version of libibverbs should I actually be using for DPDK 17.11.1 ?

thanks,
Iain

Re: [dpdk-users] Running DPDK application in Linux init level 3

[Rosen, Rami]

Hi Anan,
You better use the same Grub kernel command line parameters both in 
run level 5 and in run level 3.
In order to be able to do so, keep the GRUB_CMDLINE_LINUX as what you had in init level 5, and assuming you are using one of the popular distros (like Fedora or recent Ubuntus), which runs systemd, the recommended way to what you call "disable the gui" is by a systemd command called systemctl.

So, when the gui mode is enabled, you will get:
systemctl get-default
graphical.target

And to disable the gui you can run:

sudo systemctl enable multi-user.target --force
sudo systemctl set-default multi-user.target

After boot, both in run level 3 and in run level 5, you will be able to verify that the kernel command line is as expected with:

cat /proc/cmdline

Regards,
Rami Rosen


-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Anand Prasad
Sent: Tuesday, April 10, 2018 15:12
To: users@dpdk.org
Subject: [dpdk-users] Running DPDK application in Linux init level 3

Hi DPDK Team,
  I am trying to run DPDK application in Linux with init level 3 (no gui) and I am observing continuous packet drops on Rx side. But, If I boot Linux in init level 5 (GUI), then I dont see drops.
  I have provided linux command line arguments (for isolating the cpu cores and hugepates etc in "GRUB_CMDLINE_LINUX_DEFAULT" variable, and with those it's working fine in init level 5. But, not sureif these configuration are applicable when I boot in Init level 3?
 To boot in init level 3, I am setting GRUB_CMDLINE_LINUX="3" , is it correct? If not please suggest what is theother method.
Thanks in advance.Anand

Re: [dpdk-users] Running DPDK application in Linux init level 3

[Anand Prasad]

 Thanks Rami Rosen
I tried the steps you have suggested, and with that, I am able to boot in command line.But still I face the same packet drop issue.
Any guess, what causing problem?
Regards,Anand Prasad
    On Wednesday, 11 April, 2018, 1:02:47 AM IST, Rosen, Rami <rami.rosen@intel.com> wrote:  
 
 Hi Anan,
You better use the same Grub kernel command line parameters both in 
run level 5 and in run level 3.
In order to be able to do so, keep the GRUB_CMDLINE_LINUX as what you had in init level 5, and assuming you are using one of the popular distros (like Fedora or recent Ubuntus), which runs systemd, the recommended way to what you call "disable the gui" is by a systemd command called systemctl.

So, when the gui mode is enabled, you will get:
systemctl get-default
graphical.target

And to disable the gui you can run:

sudo systemctl enable multi-user.target --force
sudo systemctl set-default multi-user.target

After boot, both in run level 3 and in run level 5, you will be able to verify that the kernel command line is as expected with:

cat /proc/cmdline

Regards,
Rami Rosen


-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Anand Prasad
Sent: Tuesday, April 10, 2018 15:12
To: users@dpdk.org
Subject: [dpdk-users] Running DPDK application in Linux init level 3

Hi DPDK Team,
  I am trying to run DPDK application in Linux with init level 3 (no gui) and I am observing continuous packet drops on Rx side. But, If I boot Linux in init level 5 (GUI), then I dont see drops.
  I have provided linux command line arguments (for isolating the cpu cores and hugepates etc in "GRUB_CMDLINE_LINUX_DEFAULT" variable, and with those it's working fine in init level 5. But, not sureif these configuration are applicable when I boot in Init level 3?
 To boot in init level 3, I am setting GRUB_CMDLINE_LINUX="3" , is it correct? If not please suggest what is theother method.
Thanks in advance.Anand
  
From shahafs@mellanox.com  Wed Apr 11 10:50:17 2018
Return-Path: <shahafs@mellanox.com>
Received: from EUR02-AM5-obe.outbound.protection.outlook.com
 (mail-eopbgr00087.outbound.protection.outlook.com [40.107.0.87])
 by dpdk.org (Postfix) with ESMTP id 7009C1BAAC
 for <users@dpdk.org>; Wed, 11 Apr 2018 10:50:17 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com;
 s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version;
 bh=xiKLqbg4jt6LHMr7X6+hwQ0jG/WC019+TM87qutFg7c=;
 b=csxDXwOUE0P7o2spuMHI6Fjie/fPDrhZpzj2VKaXgPcSJaSzpoABwkFIKvt8WxTZlLGh7k+WXmD8fZetIVvIOdHuT/m4Jsmw9UN3G3/4sPM8XjfIzQhGNvJ0NNqzm/V4Nq1WEGMXivyD54ww1D8Vo7Up1OEaDeVi9b8CCdI3+H0Received: from DB7PR05MB4426.eurprd05.prod.outlook.com (52.134.109.15) by
 DB7PR05MB4396.eurprd05.prod.outlook.com (52.134.108.161) with Microsoft SMTP
 Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id
 15.20.675.9; Wed, 11 Apr 2018 08:50:16 +0000
Received: from DB7PR05MB4426.eurprd05.prod.outlook.com
 ([fe80::808d:386e:26f3:859f]) by DB7PR05MB4426.eurprd05.prod.outlook.com
 ([fe80::808d:386e:26f3:859f%13]) with mapi id 15.20.0653.014; Wed, 11 Apr
 2018 08:50:16 +0000
From: Shahaf Shuler <shahafs@mellanox.com>
To: Iain Barker <iain.barker@oracle.com>, "users@dpdk.org" <users@dpdk.org>
Thread-Topic: [dpdk-users] Wrong OFED version for DPDK 17.11.1 ?
Thread-Index: AQHT0P/P6IlBsvjB/UyWc5Rsy2xzxqP7P+eQ
Date: Wed, 11 Apr 2018 08:50:16 +0000
Message-ID: <DB7PR05MB442663C54C01DAE234726A1AC3BD0@DB7PR05MB4426.eurprd05.prod.outlook.com>
References: <64897879-3872-445c-8179-4ea5852e4ecf@default>
In-Reply-To: <64897879-3872-445c-8179-4ea5852e4ecf@default>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
authentication-results: spf=none (sender IP is )
 smtp.mailfrom=shahafs@mellanox.com;
x-originating-ip: [193.47.165.251]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; DB7PR05MB4396;
 7:dJ0dH7d9cItf3MXB/uc+fino3nj3nIqpZZi6NSu1lOs4i8e/9wNPU3G63cZXp7YcE2w0RoTkbQ68oN/4tOJJcadRA/bnUGQTi8P/sDZB2Me75HPiw7mUWsBIpWJnYA/3Nm99to20jEguhSC4ijZkhJfduiMJwnAeO0uZZuXSLntRxq5vhvcXky3A4GQ5ruvUnS6f7b0TmBZhl3W5vP7kSzsW72cMTV6W35zbuOohFMhyzrSsZyqSRucHO9v4LMUy
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: UriScan:; BCL:0; PCL:0;
 RULEID:(7020095)(4652020)(5600026)(3008032)(4534165)(4627221)(201703031133081)(201702281549075)(48565401081)(2017052603328)(7153060)(7193020);
 SRVR:DB7PR05MB4396;
x-ms-traffictypediagnostic: DB7PR05MB4396:
x-microsoft-antispam-prvs: <DB7PR05MB4396FC4CF2F7F14A0D9EC08CC3BD0@DB7PR05MB4396.eurprd05.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(189930954265078)(15185016700835)(45079756050767);
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0;
 RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(10201501046)(3231221)(944501327)(52105095)(93006095)(93001095)(6055026)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(20161123558120)(20161123562045)(6072148)(201708071742011);
 SRVR:DB7PR05MB4396; BCL:0; PCL:0; RULEID:; SRVR:DB7PR05MB4396;
x-forefront-prvs: 0639027A9E
x-forefront-antispam-report: SFV:NSPM;
 SFS:(10009020)(39380400002)(346002)(366004)(39860400002)(376002)(396003)(189003)(199004)(53754006)(105586002)(110136005)(3846002)(97736004)(6116002)(99286004)(3660700001)(966005)(2900100001)(3280700002)(5660300001)(86362001)(76176011)(106356001)(6506007)(14454004)(316002)(45080400002)(7696005)(478600001)(81156014)(229853002)(5250100002)(305945005)(7736002)(6436002)(74316002)(66066001)(9686003)(8936002)(6306002)(2906002)(2501003)(55016002)(26005)(81166006)(68736007)(102836004)(53936002)(11346002)(33656002)(446003)(6246003)(8676002)(186003)(25786009)(486006)(476003);
 DIR:OUT; SFP:1101; SCL:1; SRVR:DB7PR05MB4396;
 H:DB7PR05MB4426.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en;
 PTR:InfoNoRecords; MX:1; A:1;
received-spf: None (protection.outlook.com: mellanox.com does not designate
 permitted sender hosts)
x-microsoft-antispam-message-info: grvNc2p/JfczS4278NYZXV4mKH66YtrGBySzlN7nCXu+73oiixaEXMFbQZUnBNoOmjz65mG+TgpJEvMExng/5Ig4bcASM0gksgPWezxoB9na3IstoEr8d4IPicR7yy5UHrUljlTh7x3FBNeEFY5pnKU3E8BO3ETY7tVhS4a3KqzTzYWAQEtKqpmfy65RviayyYDkfjajPlYXKWf2bs4mjiVKTXEvP8soyjOmVUrF1I2SQB5WmIe5yf0HNMcSOq+x0gfijvtkTiJ1jyPdk7PRy0c/ivbFoUDlzXSjODZBAHNuNGL+mV+DepgoxFx5U4JJvQKqOvld7PCj6tOGQ1PkS/xVlUkuYZNo6VvTPr8QtcHGgcXEvukt9XUh0WtN5KZUyok6oNZRP9iM/+ysXbvxRMyHa/54f2DHXh9EtRXedXMspamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Office365-Filtering-Correlation-Id: 5025ba9c-5cef-49ce-1c38-08d59f893f16
X-OriginatorOrg: Mellanox.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 5025ba9c-5cef-49ce-1c38-08d59f893f16
X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Apr 2018 08:50:16.0230 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB7PR05MB4396
Subject: Re: [dpdk-users] Wrong OFED version for DPDK 17.11.1 ?
X-BeenThere: users@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK usage discussions <users.dpdk.org>
List-Unsubscribe: <https://dpdk.org/ml/options/users>,
 <mailto:users-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/users/>
List-Post: <mailto:users@dpdk.org>
List-Help: <mailto:users-request@dpdk.org?subject=help>
List-Subscribe: <https://dpdk.org/ml/listinfo/users>,
 <mailto:users-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 11 Apr 2018 08:50:17 -0000

Hi Iain, 

Tuesday, April 10, 2018 10:12 PM, Iain Barker:
> Subject: [dpdk-users] Wrong OFED version for DPDK 17.11.1 ?
> 
> Hi all,
> 
> I'm trying to build 17.11.1 against libibverbs from OFED 4.2-1.0.0.0 but having
> some API compatibility issues.
> 
> According to the DPDK docs, that's the correct version for DPDK 17.11
> Ref: "Mellanox OFED version: 4.2" in
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdp
> dk.org%2Fdoc%2Fguides-
> 17.11%2Fnics%2Fmlx5.html&data=02%7C01%7Cshahafs%40mellanox.com%7
> C00e205bcc93b4ab687ce08d59f16ec56%7Ca652971c7d2e4d9ba6a4d149256f4
> 61b%7C0%7C0%7C636589843233074081&sdata=tRgQGtKP3WlalZ8XYVp6Qvq
> pkStiVgdup6EmZSKEC3c%3D&reserved=0
> 
> But I get multiple errors which suggest the API versions are incompatible.  For
> example:
> 
>            priv->hw_csum =
>                   !!(device_attr_ex.device_cap_flags_ex &
>                      IBV_DEVICE_RAW_IP_CSUM);
> 
> drivers/net/mlx5/mlx5.c:794:21: error: 'struct ibv_device_attr_ex' has no
> member named 'device_cap_flags_ex'
> 
> MLNX_OFED_SRC-4.2-1.0.0.0 contains libibverbs-41mlnx1, and that has
> include/infiniband/verbs.h with the following typedef:
> struct ibv_device_attr_ex {
>       struct ibv_device_attr  orig_attr;
>       uint32_t          comp_mask;
>       struct ibv_odp_caps     odp_caps;
> };
> 
> So the docs appear to be wrong. no way the OFED 4.2 version of mlx5.c is
> going to compile against that header.
> 
> What version of libibverbs should I actually be using for DPDK 17.11.1 ?

Starting from DPDK 17.11 MLX5 PMD uses rdma-core as the user space instead of libibverbs/libmlx5. As you can see from the doc it will be installed when calling mlnx ofed installation script using --usptream-libs. 

You were installing the legacy ibverbs, hence the compilation errors. Try to install rdma-core instead. 

> 
> thanks,
> Iain

Re: [dpdk-users] Running DPDK application in Linux init level 3

[Rosen, Rami]

Hi Anad,

Happy to hear that it worked for you.

Regarding the performance: this is strange to hear.
Are you sure that you are running DPDK the same way in
both cases (same parameters, same /proc/cmdline, etc)?


Regards,
Rami

From: Anand Prasad [mailto:anandnprasad@yahoo.com]
Sent: Wednesday, April 11, 2018 09:55
To: users@dpdk.org; Rosen, Rami <rami.rosen@intel.com>
Subject: Re: RE: [dpdk-users] Running DPDK application in Linux init level 3

Thanks Rami Rosen

I tried the steps you have suggested, and with that, I am able to boot in command line.
But still I face the same packet drop issue.

Any guess, what causing problem?

Regards,
Anand Prasad

On Wednesday, 11 April, 2018, 1:02:47 AM IST, Rosen, Rami <rami.rosen@intel.com<mailto:rami.rosen@intel.com>> wrote:


Hi Anan,
You better use the same Grub kernel command line parameters both in
run level 5 and in run level 3.
In order to be able to do so, keep the GRUB_CMDLINE_LINUX as what you had in init level 5, and assuming you are using one of the popular distros (like Fedora or recent Ubuntus), which runs systemd, the recommended way to what you call "disable the gui" is by a systemd command called systemctl.

So, when the gui mode is enabled, you will get:
systemctl get-default
graphical.target

And to disable the gui you can run:

sudo systemctl enable multi-user.target --force
sudo systemctl set-default multi-user.target

After boot, both in run level 3 and in run level 5, you will be able to verify that the kernel command line is as expected with:

cat /proc/cmdline

Regards,
Rami Rosen

-----Original Message-----
From: users [mailto:users-bounces@dpdk.org<mailto:users-bounces@dpdk.org>] On Behalf Of Anand Prasad
Sent: Tuesday, April 10, 2018 15:12
To: users@dpdk.org<mailto:users@dpdk.org>
Subject: [dpdk-users] Running DPDK application in Linux init level 3

Hi DPDK Team,
  I am trying to run DPDK application in Linux with init level 3 (no gui) and I am observing continuous packet drops on Rx side. But, If I boot Linux in init level 5 (GUI), then I dont see drops.
  I have provided linux command line arguments (for isolating the cpu cores and hugepates etc in "GRUB_CMDLINE_LINUX_DEFAULT" variable, and with those it's working fine in init level 5. But, not sureif these configuration are applicable when I boot in Init level 3?
 To boot in init level 3, I am setting GRUB_CMDLINE_LINUX="3" , is it correct? If not please suggest what is theother method.
Thanks in advance.Anand