From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f173.google.com (mail-wi0-f173.google.com [209.85.212.173]) by dpdk.org (Postfix) with ESMTP id 479177FFA for ; Mon, 27 Oct 2014 09:37:51 +0100 (CET) Received: by mail-wi0-f173.google.com with SMTP id ex7so5644095wid.6 for ; Mon, 27 Oct 2014 01:46:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:organization :user-agent:in-reply-to:references:mime-version :content-transfer-encoding:content-type; bh=J3G6nbDHuTxRNMEYHZTWwno2pPNrS6gw7xUMB/2Kdqo=; b=M/fhlEVn0Dsn8nXAcVasUrEJlNh/DFNmuYmHQjCbDJ13H2LqPkbNYiR/+KZ1OYzvrW eTnDsXtgWj7G35+MUKCBrTXCqd2Tf68UZ6U3Kmf3wrreyzGy5IUDd8h7/fxk0o4HPyX6 hXpaBuY29e67gz94roguLY3Bl4b1bW/Zk5WwR5JebIUxSSX+vXRH9VGg08eSmRbIDoIV nuRgp3t4dbqY3VTqDsgOxlK8UOsn6/ueb6rC/JbeaO12ehTQvfRUqHJqPPOSi2TuNCiB Lvl8qX/QUDuMm6nSAEANTeo1DwwWOx2aVleaf3+8lZjbXFGwcQpDY+V3ct4+sizSFNfU fzJA== X-Gm-Message-State: ALoCoQme5prfGJZpiiAuqQI1e1qS2C5P+QYs7MUv/pZQUH32TWIrL30O3bP2idt7/UOx389AhGpe X-Received: by 10.194.219.230 with SMTP id pr6mr7860877wjc.40.1414399592826; Mon, 27 Oct 2014 01:46:32 -0700 (PDT) Received: from xps13.localnet (136-92-190-109.dsl.ovh.fr. [109.190.92.136]) by mx.google.com with ESMTPSA id pn4sm14784238wjc.38.2014.10.27.01.46.31 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 27 Oct 2014 01:46:32 -0700 (PDT) From: Thomas Monjalon To: "Ouyang, Changchun" Date: Mon, 27 Oct 2014 09:46:15 +0100 Message-ID: <1433643.g0MTQjq1Jf@xps13> Organization: 6WIND User-Agent: KMail/4.14.2 (Linux/3.16.4-1-ARCH; KDE/4.14.2; x86_64; ; ) In-Reply-To: References: <1414139898-26562-1-git-send-email-changchun.ouyang@intel.com> <1684003.hLxV2SOth0@xps13> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Cc: dev@dpdk.org Subject: Re: [dpdk-dev] [PATCH] vhost: Check descriptor number for vector Rx X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Oct 2014 08:37:51 -0000 2014-10-25 00:48, Ouyang, Changchun: > From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com] > > 2014-10-24 16:38, Ouyang Changchun: > > > For zero copy, it need check whether RX descriptor num meets the least > > > requirement when using vector PMD Rx function, and give user more > > > hints if it fails to meet the least requirement. > > [...] > > > --- a/examples/vhost/main.c > > > +++ b/examples/vhost/main.c > > > @@ -131,6 +131,10 @@ > > > #define RTE_TEST_RX_DESC_DEFAULT_ZCP 32 /* legacy: 32, DPDK virt FE: 128. */ > > > #define RTE_TEST_TX_DESC_DEFAULT_ZCP 64 /* legacy: 64, DPDK virt FE: 64. */ > > > > > > +#ifdef RTE_IXGBE_INC_VECTOR > > > +#define VPMD_RX_BURST 32 > > > +#endif > > > + > > > /* Get first 4 bytes in mbuf headroom. */ #define > > > MBUF_HEADROOM_UINT32(mbuf) (*(uint32_t *)((uint8_t *)(mbuf) \ > > > + sizeof(struct rte_mbuf))) > > > @@ -792,6 +796,19 @@ us_vhost_parse_args(int argc, char **argv) > > > return -1; > > > } > > > > > > +#ifdef RTE_IXGBE_INC_VECTOR > > > + if ((zero_copy == 1) && (num_rx_descriptor <= VPMD_RX_BURST)) { > > > + RTE_LOG(INFO, VHOST_PORT, > > > + "The RX desc num: %d is too small for PMD to work\n" > > > + "properly, please enlarge it to bigger than %d if\n" > > > + "possible by the option: '--rx-desc-num '\n" > > > + "One alternative is disabling RTE_IXGBE_INC_VECTOR\n" > > > + "in config file and rebuild the libraries.\n", > > > + num_rx_descriptor, VPMD_RX_BURST); > > > + return -1; > > > + } > > > +#endif > > > + > > > return 0; > > > } > > > > I feel there is a design problem here. > > An application shouldn't have to care about the underlying driver. > > For most of other applications, as their descriptor numbers are set as big enough(1024 or so) , > So there is no need to check the descriptor number at the early stage of running. > > But for vhost zero copy(note vhost one copy also has 1024 descriptor number) has the default > descriptor number of 32. > Why use 32? > because vhost zero copy implementation (working as backend) need support dpdk based app which use pmd virtio-net driver, > And also need support linux legacy virtio-net based application. > When it is the linux legacy virtio-net case, on one side the qemu has hard code to confine the total virtio descriptor size to 256, > On other side, legacy virtio use half of them as virtio header, and then only another half i.e. 128 descriptors are available to use as real buffer. > > In PMD mode, all HW descriptors need to be filled DMA address in the rx initial stage, otherwise there is probably exceptional in rx process. > Based on that, we need use really limited virtio buffer to fully fill all hw descriptor DMA address, > Or in other word, the available virtio descriptor size will determine the total mbuf size and hw descriptor size in the case of zero copy, > > Tune and find that 32 is the suitable value for vhost zero copy to work properly when it legacy linux virtio case. > Another factor to reduce the value to 32, is that mempool use ring to accommodate the mbuf, it cost one to flag the ring head/tail, > And there are some other overheads like temporary mbufs(size as RX_BURST) when rx. > Note that number descriptor should need power 2. > > Why the change occur at this moment? > Recently the default rx function is modified into vector RX function, while it use non-vector mode (scalar mode) Rx previously, > Vector RX function need more than 32 descriptor to work properly, but scalar mode RX hasn't this limitation. > > As the RX function is changeable(you can use vector mode or non-vector), and descriptor number can also be changed. > So here in the vhost app, check if they match to make sure all things could work normally, and give some hints if they don't match. > > Hope the above could make it a bit clearer. :-) Thank you for your explanation. Your fix shows that driver and application are tightly linked. It's a design issue. As I said: "An application shouldn't have to care about the underlying driver." I didn't dig enough in vhost to suggest a good fix but I'm sure someone could have an idea. -- Thomas