From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id AEDD2A046B for ; Wed, 24 Jul 2019 18:07:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9F87B1C217; Wed, 24 Jul 2019 18:07:47 +0200 (CEST) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id 8EE9A2BAC; Wed, 24 Jul 2019 18:07:44 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 24 Jul 2019 09:07:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,303,1559545200"; d="scan'208";a="181149023" Received: from silpixa00399498.ir.intel.com (HELO silpixa00399498.ger.corp.intel.com) ([10.237.223.125]) by orsmga002.jf.intel.com with ESMTP; 24 Jul 2019 09:07:42 -0700 From: Anatoly Burakov To: dev@dpdk.org Cc: Bruce Richardson , stephen@networkplumber.org, stable@dpdk.org Date: Wed, 24 Jul 2019 17:07:41 +0100 Message-Id: <67a795e77bc9f5ac79ab78a878ae19abbead9f50.1563984454.git.anatoly.burakov@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: References: Subject: [dpdk-dev] [PATCH v4] eal: fix proc type auto detection X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Currently, primary process holds an exclusive lock on the config file, thereby preventing other primaries from spinning up. However, when the primary dies, the lock is no longer being held, even though there might be other secondary processes still running. The fix is two-fold. First of all, downgrade the primary process's exclusive lock to a shared lock once we have it. Second of all, also take out shared locks on the config from the secondaries. We are using fcntl() locks, which get dropped when the file handle is closed, so also remove the closure of config file handle. Fixes: af75078fece3 ("first public release") Cc: stable@dpdk.org Signed-off-by: Anatoly Burakov --- Notes: v4: - Fixed FreeBSD log message to match Linux version v3: - Added similar changes to FreeBSD version v2: - Adjusted indentation lib/librte_eal/freebsd/eal/eal.c | 36 +++++++++++++++++++++++++++---- lib/librte_eal/linux/eal/eal.c | 37 +++++++++++++++++++++++++++----- 2 files changed, 64 insertions(+), 9 deletions(-) diff --git a/lib/librte_eal/freebsd/eal/eal.c b/lib/librte_eal/freebsd/eal/eal.c index d53f0fe69..69995bf8f 100644 --- a/lib/librte_eal/freebsd/eal/eal.c +++ b/lib/librte_eal/freebsd/eal/eal.c @@ -72,6 +72,13 @@ static struct flock wr_lock = { .l_len = sizeof(early_mem_config.memsegs), }; +static struct flock rd_lock = { + .l_type = F_RDLCK, + .l_whence = SEEK_SET, + .l_start = offsetof(struct rte_mem_config, memsegs), + .l_len = sizeof(early_mem_config.memsegs), +}; + /* Address of global and public configuration */ static struct rte_config rte_config = { .mem_config = &early_mem_config, @@ -249,8 +256,21 @@ rte_eal_config_create(void) if (retval < 0){ close(mem_cfg_fd); mem_cfg_fd = -1; - RTE_LOG(ERR, EAL, "Cannot create lock on '%s'. Is another primary " - "process running?\n", pathname); + RTE_LOG(ERR, EAL, "Cannot create exclusive lock on '%s'. " + "Is another process running?\n", pathname); + return -1; + } + + /* we hold an exclusive lock - now downgrade it to a read lock to allow + * other processes to also hold onto this file while preventing other + * primaries from spinning up. + */ + retval = fcntl(mem_cfg_fd, F_SETLK, &rd_lock); + if (retval < 0) { + close(mem_cfg_fd); + mem_cfg_fd = -1; + RTE_LOG(ERR, EAL, "Cannot downgrade to shared lock on '%s': %s\n", + pathname, strerror(errno)); return -1; } @@ -292,6 +312,16 @@ rte_eal_config_attach(void) return -1; } } + /* lock the file to prevent primary from initializing while this + * process is still running. + */ + if (fcntl(mem_cfg_fd, F_SETLK, &rd_lock) < 0) { + close(mem_cfg_fd); + mem_cfg_fd = -1; + RTE_LOG(ERR, EAL, "Cannot create shared lock on '%s': %s\n", + pathname, strerror(errno)); + return -1; + } rte_mem_cfg_addr = mmap(NULL, sizeof(*rte_config.mem_config), PROT_READ, MAP_SHARED, mem_cfg_fd, 0); @@ -330,8 +360,6 @@ rte_eal_config_reattach(void) mem_config = (struct rte_mem_config *) mmap(rte_mem_cfg_addr, sizeof(*mem_config), PROT_READ | PROT_WRITE, MAP_SHARED, mem_cfg_fd, 0); - close(mem_cfg_fd); - mem_cfg_fd = -1; if (mem_config == MAP_FAILED) { RTE_LOG(ERR, EAL, "Cannot mmap memory for rte_config! error %i (%s)\n", diff --git a/lib/librte_eal/linux/eal/eal.c b/lib/librte_eal/linux/eal/eal.c index 34db78753..0f0726703 100644 --- a/lib/librte_eal/linux/eal/eal.c +++ b/lib/librte_eal/linux/eal/eal.c @@ -83,6 +83,13 @@ static struct flock wr_lock = { .l_len = sizeof(early_mem_config.memsegs), }; +static struct flock rd_lock = { + .l_type = F_RDLCK, + .l_whence = SEEK_SET, + .l_start = offsetof(struct rte_mem_config, memsegs), + .l_len = sizeof(early_mem_config.memsegs), +}; + /* Address of global and public configuration */ static struct rte_config rte_config = { .mem_config = &early_mem_config, @@ -343,8 +350,21 @@ rte_eal_config_create(void) if (retval < 0){ close(mem_cfg_fd); mem_cfg_fd = -1; - RTE_LOG(ERR, EAL, "Cannot create lock on '%s'. Is another primary " - "process running?\n", pathname); + RTE_LOG(ERR, EAL, "Cannot create exclusive lock on '%s'. " + "Is another process running?\n", pathname); + return -1; + } + + /* we hold an exclusive lock - now downgrade it to a read lock to allow + * other processes to also hold onto this file while preventing other + * primaries from spinning up. + */ + retval = fcntl(mem_cfg_fd, F_SETLK, &rd_lock); + if (retval < 0) { + close(mem_cfg_fd); + mem_cfg_fd = -1; + RTE_LOG(ERR, EAL, "Cannot downgrade to shared lock on '%s': %s\n", + pathname, strerror(errno)); return -1; } @@ -389,6 +409,16 @@ rte_eal_config_attach(void) return -1; } } + /* lock the file to prevent primary from initializing while this + * process is still running. + */ + if (fcntl(mem_cfg_fd, F_SETLK, &rd_lock) < 0) { + close(mem_cfg_fd); + mem_cfg_fd = -1; + RTE_LOG(ERR, EAL, "Cannot create shared lock on '%s': %s\n", + pathname, strerror(errno)); + return -1; + } /* map it as read-only first */ mem_config = (struct rte_mem_config *) mmap(NULL, sizeof(*mem_config), @@ -427,9 +457,6 @@ rte_eal_config_reattach(void) sizeof(*mem_config), PROT_READ | PROT_WRITE, MAP_SHARED, mem_cfg_fd, 0); - close(mem_cfg_fd); - mem_cfg_fd = -1; - if (mem_config == MAP_FAILED || mem_config != rte_mem_cfg_addr) { if (mem_config != MAP_FAILED) { /* errno is stale, don't use */ -- 2.17.1