From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A0B31A053C for ; Thu, 30 Jul 2020 11:41:49 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0C9BB1BFFA; Thu, 30 Jul 2020 11:41:49 +0200 (CEST) Received: from mail-oi1-f194.google.com (mail-oi1-f194.google.com [209.85.167.194]) by dpdk.org (Postfix) with ESMTP id A96CE1BFE7 for ; Thu, 30 Jul 2020 11:41:47 +0200 (CEST) Received: by mail-oi1-f194.google.com with SMTP id k4so23254640oik.2 for ; Thu, 30 Jul 2020 02:41:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=VK18hnfTPVA8KNfEJ/yV5mlXjDKzSuK4eHZW2NI6r9Q=; b=j73Cedewrlal9ehL6uVh9Vj2YVbDH26uFKKHOFzyXtXlHmmchQBp6p5SoeLVJotSoK Arhyqw5Z1cyrOhcZHaJsw2OYgzQTTuS8M3FuKwKFdLRNPj9QlsnARFg7eLb8GC8q5pjp HNGkfMVq7LOglpeRAbkl7U9GYG7LBMIj/Qqea2j/qsWR5wR7hllFPVb95lD8z9iKMeE/ SG0Naf1CAkub4rzJYcTXGRiTEZNPH0gMvB2JxOYBngs5XNufQ6IL+Fph8qDwj1BfoZKL 9dpaXjjWe7jm2W9JA7y7mzZiTExdxqa8PUTSnO5c/teZB59Q4OBHK0CGU8/Wl7b/P88I EbwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=VK18hnfTPVA8KNfEJ/yV5mlXjDKzSuK4eHZW2NI6r9Q=; b=r9mu1ujERbsOBDzn5oaD6DsUlKMGc2crYNRuaYhcn2oSofQVmzt3FAPvmQTzI7tonP dlGcScfOrBwreS+ECkY7t3KvsRT7phY/dL4ejmDVTGHPXmi79sndGYP8h0d5jeNl3QcH bBOFv4ipX4pTX0Evbyvk2oJt4NiG6lE1yTcNKg32BfJFKJ5LMktdD5EZwCKynhnWbotv sD9eNr9KYAhgFshQ4dszxRZKLaS+LK0tGFI/F97ZOinW9YP4ePELDp32QQNznr+M7efB w3IdoVdsC7qadpeqiviPyDQKdUAVhthJNa2uc1HdBPVKyCnu36zAETKC06ztn8WA5Pi7 UV8g== X-Gm-Message-State: AOAM533oqSB+fcEzmZFGGF+3lfZW4NlrR+N2k+5UC7rqP1Wts+JxBKXA uw8ohLTDoegGztdbcwXSP1hkBDoRDEvBW0urut4OmzTdPIrUqS/6 X-Google-Smtp-Source: ABdhPJwFOr9Q2Pbig+xQgUPH+LmJWSqAB9c3BB7tQnaKs6985SQ++RWr9bzCZPEGG6tNOkFN5sdVZR4LNKRzWGQPXFE= X-Received: by 2002:aca:dfc1:: with SMTP id w184mr10431087oig.79.1596102106684; Thu, 30 Jul 2020 02:41:46 -0700 (PDT) MIME-Version: 1.0 From: Michael K Date: Thu, 30 Jul 2020 12:41:35 +0300 Message-ID: To: users@dpdk.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] No free hugepages after process restart X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi, I have a hugepages-related problem with a DPDK process not being able to run after it was terminated and then restarted again. When the process is restarted, the rte_eal_init() call fails and the terror message is: EAL: No free hugepages reported in hugepages-2048kB EAL: FATAL: Cannot get hugepage information. More details below. DPDK version is 17.11.3. OS is Ubuntu 14/3.13.0-32-generic. Hugepages are 2MB large. hugetlbfs is mounted in the process's startup script, the amount of hugepages is set by writing to /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages. The amount of memory for DPDK to allocate is passed during the initialization with the "--socket-mem" parameter. After the initial start, DPDK works without any problems. When I stop the process (with kill/TERM) and start it again, DPDK initialization fails as I wrote above. After the failure /proc/meminfo shows zero free hugepages. So, since DPDK is complaining on "No free hugepages reported", I have tried to find what other processes could be using hugepages: - transparent hugepages are disabled via the kernel's commandline. - ipcs -a shows nothing - there are no other hugetlbfs mounts on the system - running lsof and trying to find files open in the hugetlbfs mount dir gives nothing - grepping 'KernelPageSize: 2048 kB' in /proc/*/smaps gives nothing - grep huge /proc/*/numa_maps gives nothing Basically I've tried every "find the hugepage user" trick I could find on the net. If after the failure I add more hugepages via /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages, the process starts successfully and everything is fine again, until I kill it and then it's not able to run again with the same error message. And /proc/meminfo shows no free hugepages again. One suspicious detail: the problem happens only on machines with Docker running. But I'm not sure whether this is related or not. Would appreciate any ideas. Thanks, Michael