From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 148BD42BE5; Tue, 30 May 2023 15:53:28 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0638C40A82; Tue, 30 May 2023 15:53:28 +0200 (CEST) Received: from mail-lf1-f54.google.com (mail-lf1-f54.google.com [209.85.167.54]) by mails.dpdk.org (Postfix) with ESMTP id B0754406BC for ; Tue, 30 May 2023 15:53:26 +0200 (CEST) Received: by mail-lf1-f54.google.com with SMTP id 2adb3069b0e04-4f4d6aee530so4751816e87.2 for ; Tue, 30 May 2023 06:53:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weka.io; s=google; t=1685454806; x=1688046806; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=g+0Mdm827g/MjSFM6f4uQYmWLoIoYdWZ5XSgbfqkBCs=; b=KXANI6osPdtT3m5Lesi8aDYgwOTY6kpGvs6XGL6E7LczhM6gsV+X636DCBABlXhJ5q hkObkBp9DtnxTXEgTQDXZkYXmjBTflmmzN9VKiooeHEYUdUeYdkza+A80FNZSBJd1xT1 tstofG8J14P8+BcCCvzcpy72aOOoy4hqvDFgs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685454806; x=1688046806; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=g+0Mdm827g/MjSFM6f4uQYmWLoIoYdWZ5XSgbfqkBCs=; b=dN9Yo4pftk3VFYfGaidVo9AfKENWbDVm0hmbucxCXhcQ0/PgvjlGUEyXf4vCufP12o 5uer/N8WEMqMvJLmVUp89F+jKMie6AXcepXzKLSpTSAmskWnQXGFcdLlVduqI4uQXRt8 xpXtUPzoNUFR0nqzS1taT+3cTfWHE6vmcULAFQoXrO9214naa1s0wLiMyewN3XRoBDjC uoSy5WGejaM5ShdaMOQ5m2KAr+zMdHXvBwpbUedqrcl8FV/hJ44VadkbvEnrOGRKlCoE sayHAQOvHdks2s0EOUKorTmACN/xeGTD1kNweRDGkfoA0yiYrPBWN+jAgvzxBDSLly+J PGbw== X-Gm-Message-State: AC+VfDy3NfLJJpQUe6k3sPRxyb8bCevcBDrpQrSsUXxYUgAiDSBweOri bmUQf22ZuQYWPgyx92OOELhW2jPFg+i4t452qtQT1dcCvjh1tETsz1n+degEnEM1yI4kpB+TPwV NQAUaRYV6eaxxKp0JTJl5k33/9hhXcYmNR02ABB8dX0a83c/aK+OhskASGEfC4/wwBeLZdc1h5C RA+A== X-Google-Smtp-Source: ACHHUZ6hao6X1uRdnX/Qbj5LMRJlyLiDlTuMUkf4CwRdoF2XjB/Jap4a4b3vwVGZEWrkfLcv+rKaPSFDYCMxxDzsJyk= X-Received: by 2002:ac2:48a8:0:b0:4f4:cbee:bfa7 with SMTP id u8-20020ac248a8000000b004f4cbeebfa7mr797069lfg.12.1685454806051; Tue, 30 May 2023 06:53:26 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Baruch Even Date: Tue, 30 May 2023 16:53:14 +0300 Message-ID: Subject: Re: Hugepage migration To: Bruce Richardson Cc: dpdk-dev Content-Type: multipart/alternative; boundary="00000000000000562d05fce987ad" X-CLOUD-SEC-AV-Sent: true X-CLOUD-SEC-AV-Info: weka,google_mail,monitor X-Gm-Spam: 0 X-Gm-Phishy: 0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --00000000000000562d05fce987ad Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, May 30, 2023 at 11:04=E2=80=AFAM Bruce Richardson < bruce.richardson@intel.com> wrote: > On Sun, May 28, 2023 at 11:07:40PM +0300, Baruch Even wrote: > > Hi, > > We found an issue with newer kernels (5.13+) that are found on newer > > OSes (Ubuntu22, Rocky9, Ubuntu20 with kernel 5.15) where a 2M page > that > > was allocated for DPDK was migrated (moved into another physical pag= e) > > when a 1G page was allocated. > > From our reading of the kernel commits this started with commit > > ae37c7ff79f1f030e28ec76c46ee032f8fd07607 > > mm: make alloc_contig_range handle in-use hugetlb pages > > This caused what looked like memory corruptions to us and cases wher= e > > the rings were moved from their physical location and communication > was > > no longer possible. > > I wanted to ask if anyone else hit this issue and what mitigations a= re > > available? > > We are currently looking at using a kernel driver to pin the pages b= ut > > I expect that this issue will affect others and that a more general > > approach is needed. > > Thanks, > > Baruch > > -- > > Hi, > > what kernel driver was being used for the device I/O part? Was it a UIO > based driver or "vfio-pci"? When using vfio-pci and configuring IOMMU > mappings, the pages mapped should be pinned by the kernel, I would have > thought, since the kernel knows they are being used by devices. > > /Bruce > This was using igb_uio on an AWS instance with their ena driver. Baruch --=20 Baruch Even Platform Technical Lead, WEKA E baruch@weka.io* =C2=AD*W www.weka.io * =C2=AD* * =C2=AD* --00000000000000562d05fce987ad Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

On Tue, May 30, 2023 at 11:04=E2=80=AFAM = Bruce Richardson <bruce.ri= chardson@intel.com> wrote:
On Sun, May 28, 2023 at 11:07:40PM +0300, Baruch Even wro= te:
>=C2=A0 =C2=A0 Hi,
>=C2=A0 =C2=A0 We found an issue with newer kernels (5.13+) that are fou= nd on newer
>=C2=A0 =C2=A0 OSes (Ubuntu22, Rocky9, Ubuntu20 with kernel 5.15) where = a 2M page that
>=C2=A0 =C2=A0 was allocated for DPDK was migrated (moved into another p= hysical page)
>=C2=A0 =C2=A0 when a 1G page was allocated.
>=C2=A0 =C2=A0 From our reading of the kernel commits this started with = commit
>=C2=A0 =C2=A0 ae37c7ff79f1f030e28ec76c46ee032f8fd07607
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 mm: make alloc_contig_range handle in-use h= ugetlb pages
>=C2=A0 =C2=A0 This caused what looked like memory corruptions to us and= cases where
>=C2=A0 =C2=A0 the rings were moved from their physical location and com= munication was
>=C2=A0 =C2=A0 no longer possible.
>=C2=A0 =C2=A0 I wanted to ask if anyone else hit this issue and what mi= tigations are
>=C2=A0 =C2=A0 available?
>=C2=A0 =C2=A0 We are currently looking at using a kernel driver to pin = the pages but
>=C2=A0 =C2=A0 I expect that this issue will affect others and that a mo= re general
>=C2=A0 =C2=A0 approach is needed.
>=C2=A0 =C2=A0 Thanks,
>=C2=A0 =C2=A0 Baruch
>=C2=A0 =C2=A0 --

Hi,

what kernel driver was being used for the device I/O part? Was it a UIO
based driver or "vfio-pci"? When using vfio-pci and configuring I= OMMU
mappings, the pages mapped should be pinned by the kernel, I would have
thought, since the kernel knows they are being used by devices.

/Bruce

This was using igb_uio on an AWS ins= tance with their ena driver.

Baruch
=
--
<= /tr>
=
=
Baruch Eve= n
Platform Technical Lead,=C2= =A0 WEKA
E=C2= =A0baruch@weka.io<= i style=3D"color:rgb(255,255,255)">=E2=80=85=C2=ADW=C2=A0www= .weka.io=E2=80=85=C2=AD=C2=A0=E2=80=85=C2=AD
=
<= /table>
<= a href=3D"https://www.weka.io/lp/weka-named-a-2023-customers-choice-by-gart= ner-peer-insights/?utm_source=3Dsignature&utm_medium=3Demail" target=3D= "_blank">
3D""
--00000000000000562d05fce987ad--