[dpdk-dev] running dpdk 2.1 on openstak causes CPU soft lockup

[dpdk-dev] running dpdk 2.1 on openstak causes CPU soft lockup

[Nissim Nisimov]

Hi,

I am running DPDK 2.1.0 based app on OpenStack KVM guest.  OS of guest is Ubuntu LTS 14.04 3.13 kernel.
virtio.

After upgrade to dpdk 2.1 (previous version was dpdk 1.8 and it worked fine) we are seeing the following issue:

[960.004535] BUG: soft lockup - CPU#3 stuck for 23s! [ip:8419]

The message will be printed in an endless loop and the system won't recover.

Is it a known issue in dpdk 2.1? any patch I can apply in order to overcome this?


Thx
Nissim

Re: [dpdk-dev] running dpdk 2.1 on openstak causes CPU soft lockup

[Franck BAUDIN]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Nissim Nisimov
> Sent: lundi 23 novembre 2015 20:44
> To: dev@dpdk.org
> Subject: [dpdk-dev] running dpdk 2.1 on openstak causes CPU soft lockup
>
> Hi,
>
> I am running DPDK 2.1.0 based app on OpenStack KVM guest.  OS of guest is
> Ubuntu LTS 14.04 3.13 kernel.
> virtio.
>
> After upgrade to dpdk 2.1 (previous version was dpdk 1.8 and it worked fine)
> we are seeing the following issue:
>
> [960.004535] BUG: soft lockup - CPU#3 stuck for 23s! [ip:8419]
>
> The message will be printed in an endless loop and the system won't
> recover.
>
> Is it a known issue in dpdk 2.1? any patch I can apply in order to overcome
> this?
>
>
> Thx
> Nissim

Do you have virtIO interfaces in your guest, with at least one (usually the first one, used for cloud-init) used by the kernel? If so,  even if they are in use by the kernel, DPDK virtIO driver will attach to them (without detaching the kernel), leading to either kernel crash or infinite loops. Until DPDK 1.8.0 included, you had to explicitly bind the virtIO interfaces, via igb_uio. But since DPDK 2.x, all non-blacklisted interfaces are bound.

Franck
This message and any attachments (the "message") are confidential, intended solely for the addressees. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this message from your system. In this case, you are not authorized to use, copy this message and/or disclose the content to any other person. E-mails are susceptible to alteration. Neither Qosmos nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.

Ce message et toutes ses pièces jointes (ci-après le "message")sont confidentiels et établis à l'intention exclusive de ses destinataires. Si vous avez reçu ce message par erreur, merci d’en informer immédiatement son émetteur par courrier électronique et d’effacer ce message de votre système. Dans cette hypothèse, vous n’êtes pas autorisé à utiliser, copier ce message et/ou en divulguer le contenu à un tiers. Tout message électronique est susceptible d'altération. Qosmos et ses filiales déclinent toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié.

Re: [dpdk-dev] running dpdk 2.1 on openstak causes CPU soft lockup

[Nissim Nisimov]

Yup this is exactly our issue.

We blacked list the specific interface and its working again.

Thx
Nissim

-----Original Message-----
From: Franck BAUDIN [mailto:Franck.BAUDIN@qosmos.com] 
Sent: Tuesday, December 01, 2015 2:20 PM
To: Nissim Nisimov; dev@dpdk.org
Subject: RE: running dpdk 2.1 on openstak causes CPU soft lockup

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Nissim Nisimov
> Sent: lundi 23 novembre 2015 20:44
> To: dev@dpdk.org
> Subject: [dpdk-dev] running dpdk 2.1 on openstak causes CPU soft 
> lockup
>
> Hi,
>
> I am running DPDK 2.1.0 based app on OpenStack KVM guest.  OS of guest 
> is Ubuntu LTS 14.04 3.13 kernel.
> virtio.
>
> After upgrade to dpdk 2.1 (previous version was dpdk 1.8 and it worked 
> fine) we are seeing the following issue:
>
> [960.004535] BUG: soft lockup - CPU#3 stuck for 23s! [ip:8419]
>
> The message will be printed in an endless loop and the system won't 
> recover.
>
> Is it a known issue in dpdk 2.1? any patch I can apply in order to 
> overcome this?
>
>
> Thx
> Nissim

Do you have virtIO interfaces in your guest, with at least one (usually the first one, used for cloud-init) used by the kernel? If so,  even if they are in use by the kernel, DPDK virtIO driver will attach to them (without detaching the kernel), leading to either kernel crash or infinite loops. Until DPDK 1.8.0 included, you had to explicitly bind the virtIO interfaces, via igb_uio. But since DPDK 2.x, all non-blacklisted interfaces are bound.

Franck
This message and any attachments (the "message") are confidential, intended solely for the addressees. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this message from your system. In this case, you are not authorized to use, copy this message and/or disclose the content to any other person. E-mails are susceptible to alteration. Neither Qosmos nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.

Ce message et toutes ses pièces jointes (ci-après le "message")sont confidentiels et établis à l'intention exclusive de ses destinataires. Si vous avez reçu ce message par erreur, merci d’en informer immédiatement son émetteur par courrier électronique et d’effacer ce message de votre système. Dans cette hypothèse, vous n’êtes pas autorisé à utiliser, copier ce message et/ou en divulguer le contenu à un tiers. Tout message électronique est susceptible d'altération. Qosmos et ses filiales déclinent toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié.