From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id B62A4A32A4 for ; Fri, 25 Oct 2019 18:13:16 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 502521D50F; Fri, 25 Oct 2019 18:13:16 +0200 (CEST) Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by dpdk.org (Postfix) with ESMTP id 80E351D443 for ; Fri, 25 Oct 2019 18:13:15 +0200 (CEST) Received: by mail-pg1-f196.google.com with SMTP id k20so1856843pgi.1 for ; Fri, 25 Oct 2019 09:13:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=networkplumber-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DbUUYSktrWx+Mr7+/XFosbA8omgMeae0KrcIpIaZY+4=; b=g0+LbZNvC0/oQnA0E65voBPgI5E6LKImNIIY7vUgQIElbTP/N/BrP2eccHFrkwmFmY s/iRUc4IV33YfRn/jX8Dy5Rq/NFKpFf4+d+q8dsh1+mEWOqN1FDBuMLWAS/i7aiQDsTm rjZgYLDqNmvlES6m6SYlAsPse7N2DqlyyNu11QSUp8oEHIxqBkd52aZQWm93ynBV3QBI VCthI6AJ3X/avwK3KIFyPQAxihrQ2sHp2eIyAT+Q3Xy0iCAz40Kk5xkOkrJWkVcFD2wn MvWrFjUXMSAKazqGJ9Szf/+biteH0kYub08b0Xot8NmyY7SnoyQ4Z6XqA8WCb+zcBNyS Pmhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DbUUYSktrWx+Mr7+/XFosbA8omgMeae0KrcIpIaZY+4=; b=Jizc9GIREHp4LzrAa7SgfQdJOeWo/csWuoS63gSsCyp8eMNd/mC8QAIse/yIOVCwSu DaNhRN473vjxzZnB/khbBFbNsAgqSL0sgPE6Oxq0T1TQt/QCcYn/amMJj+6rYXGYTJKG RSfswzLJwahtdZMjmlxFZ74XC5Q4uJNcB93R3+kTfdUMZv7J+v+q2/IOCVXY4L0MGP3V rYQQJvS0nlxtpn/0v9pAHLh75dSUm0b3JfQF1k2hT6bIYTHHJp4I6v5WHxQcwKk+s3fQ 3F0JPz2H0Kr+J7NaUmS6D5EIVGCtBXFsayUbAscNsm4xXYlC0xnG0UCmyNkiC2QLJtax O6ig== X-Gm-Message-State: APjAAAXzVQauODrSuIAZxK7hRQJXz6Shc8POF7R4eKY2vMWEgMWOOnAg hsp5go+HQwk+ULzHxTG7xXe3XA== X-Google-Smtp-Source: APXvYqz718L4D//TNkiHsd9UJJI/rWp8OgsETUNxs43YHSJTame92Lk+FqeWsvk+iMUqY5sAjjX7wQ== X-Received: by 2002:a63:4c1c:: with SMTP id z28mr5276599pga.167.1572019994202; Fri, 25 Oct 2019 09:13:14 -0700 (PDT) Received: from hermes.lan (204-195-22-127.wavecable.com. [204.195.22.127]) by smtp.gmail.com with ESMTPSA id h14sm3024927pfo.15.2019.10.25.09.13.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Oct 2019 09:13:13 -0700 (PDT) Date: Fri, 25 Oct 2019 09:13:10 -0700 From: Stephen Hemminger To: Andy Lutomirski Cc: dev@dpdk.org, Thomas Gleixner , Peter Zijlstra , LKML Message-ID: <20191025091310.05770edc@hermes.lan> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] Please stop using iopl() in DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Thu, 24 Oct 2019 21:45:56 -0700 Andy Lutomirski wrote: > Hi all- > > Supporting iopl() in the Linux kernel is becoming a maintainability > problem. As far as I know, DPDK is the only major modern user of > iopl(). > > After doing some research, DPDK uses direct io port access for only a > single purpose: accessing legacy virtio configuration structures. > These structures are mapped in IO space in BAR 0 on legacy virtio > devices. Yes. Legacy virtio seems to have been designed without consideration of how to use it in userspace. Xen, Vmware and Hyper-V all use memory as a doorbell mechanism which is easier to use from userspace. > There are at least three ways you could avoid using iopl(). Here they > are in rough order of quality in my opinion: > > 1. Change pci_uio_ioport_read() and pci_uio_ioport_write() to use > read() and write() on resource0 in sysfs. The cost of entering the kernel for a doorbell mechanism is too expensive and would kill performance. > 2. Use the alternative access mechanism in the virtio legacy spec: > there is a way to access all of these structures via configuration > space. There is no way to use memory doorbell on older versions of virtio. Users want to run DPDK on old stuff like RHEL6 and even older kernel forks. There are even use cases where virtio is used for a non-Linux host; such as GCP. > 3. Use ioperm() instead of iopl(). Ioperm has the wrong thread semantics. All DPDK applications have multiple threads and the initialization logic needs to work even if the thread is started later; threads can also be started by the user application. Iopl applies to whole process so this is not an issue. > > > We are considering changes to the kernel that will potentially harm > the performance of any program that uses iopl(3) -- in particular, > context switches will become more expensive, and the scheduler might > need to explicitly penalize such programs to ensure fairness. Using > ioperm() already hurts performance, and the proposed changes to iopl() > will make it even worse. Alternatively, the kernel could drop iopl() > support entirely. I will certainly make a change to allow > distributions to remove iopl() support entirely from their kernels, > and I expect that distributions will do this. > > Please fix DPDK. Please fix virtio.