From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vk0-f44.google.com (mail-vk0-f44.google.com [209.85.213.44]) by dpdk.org (Postfix) with ESMTP id F24295A52 for ; Thu, 24 Dec 2015 06:37:29 +0100 (CET) Received: by mail-vk0-f44.google.com with SMTP id j66so143450773vkg.1 for ; Wed, 23 Dec 2015 21:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bigswitch-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=KtePdJ0VLUTdmo9CjyI2lZesjyts6Ib3fr+ugI3LUQE=; b=yYznaxH46yQJ8aFXDhmN4Q/G3RDVhPnzX3F1bWnMnWhpcXttW7Rar5Rn6V8ybXgi6X 18W8xW2SsfLL1OfgkXmKEGD3tPkW5TznITApbODYw0RZnVp04tsHW5rsd4fgcw8nTMcJ jmhJiu0Dcl0VG82xtPOfUBS9+A94f0N0DJR8KhBwURxPuts2n/fCU7fKbTGK19IeXstZ gYll5X3mrfMdQDdEXzIRdsTle+FTZHkaS04DblIoCTakduYLvekQVkxcmLg6irAh9PpW HrZfo+zxCkvw5hKvv/P/iUFAmI2z/zbOv9adhimYwUKUUzEdKmQIXR3HkmDocJbsiJNk XMPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=KtePdJ0VLUTdmo9CjyI2lZesjyts6Ib3fr+ugI3LUQE=; b=UZGZHwPXTglCSzo1kSdog8WKw1pIDpje/evkScIHUzhBnqmK4R7V+Ib1VZ6CSxZYff Dd2kQ48M7nHpOJVgKmvqFIoIp/Qz+6G3WC1eiMSxCzdlf6NjtS3YWTVqWmrmGIiP+Z8D jfMFONAYv8VXxEOIVEE00QdxCgSOjL3l2mYndTLw4wzjwWuHWaJFNwXSydte+QZSA2pr DGuqfc1/5VqaMfEcVI7ULAKKbakisr9SIyMoJdIPA++T0tIqviRehnSfRwv04zbt/DTd Md695BiPVVUc/nMPoYLpO/gOQKUKDytbXYQ8BLOJTPL4SZpzNJsFz7WawQ3AZhoHvfxG Z7Sw== X-Gm-Message-State: ALoCoQmpSKdqmku+R/KJHFrDokRnKWzZdcfVnK6pnEIYyUpw5yG9owq/x6huTykB/MpBE4yjE/1lFmb24BMGUn+pFOViLXMpWcBTaZKMe2BPezZtw10y7RQ= MIME-Version: 1.0 X-Received: by 10.31.162.22 with SMTP id l22mr23074721vke.76.1450935449376; Wed, 23 Dec 2015 21:37:29 -0800 (PST) Received: by 10.31.156.16 with HTTP; Wed, 23 Dec 2015 21:37:29 -0800 (PST) In-Reply-To: <567B61D6.1090806@igel.co.jp> References: <1447392031-24970-3-git-send-email-mukawa@igel.co.jp> <1448355603-21275-1-git-send-email-mukawa@igel.co.jp> <1448355603-21275-2-git-send-email-mukawa@igel.co.jp> <20151217114223.GC29571@yliu-dev.sh.intel.com> <56737A5E.1030803@igel.co.jp> <20151218041536.GI29571@yliu-dev.sh.intel.com> <56738B5C.1030206@igel.co.jp> <20151222034158.GH18863@yliu-dev.sh.intel.com> <567B61D6.1090806@igel.co.jp> Date: Wed, 23 Dec 2015 21:37:29 -0800 Message-ID: From: Rich Lane To: Tetsuya Mukawa Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Cc: dev@dpdk.org, ann.zhuangyanying@huawei.com Subject: Re: [dpdk-dev] [PATCH v5 1/3] vhost: Add callback and private data for vhost PMD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2015 05:37:30 -0000 On Wed, Dec 23, 2015 at 7:09 PM, Tetsuya Mukawa wrote: > On 2015/12/22 13:47, Rich Lane wrote: > > On Mon, Dec 21, 2015 at 7:41 PM, Yuanhan Liu < > yuanhan.liu@linux.intel.com> > > wrote: > > > >> On Fri, Dec 18, 2015 at 10:01:25AM -0800, Rich Lane wrote: > >>> I'm using the vhost callbacks and struct virtio_net with the vhost PMD > >> in a few > >>> ways: > >> Rich, thanks for the info! > >> > >>> 1. new_device/destroy_device: Link state change (will be covered by the > >> link > >>> status interrupt). > >>> 2. new_device: Add first queue to datapath. > >> I'm wondering why vring_state_changed() is not used, as it will also be > >> triggered at the beginning, when the default queue (the first queue) is > >> enabled. > >> > > Turns out I'd misread the code and it's already using the > > vring_state_changed callback for the > > first queue. Not sure if this is intentional but vring_state_changed is > > called for the first queue > > before new_device. > > > > > >>> 3. vring_state_changed: Add/remove queue to datapath. > >>> 4. destroy_device: Remove all queues (vring_state_changed is not called > >> when > >>> qemu is killed). > >> I had a plan to invoke vring_state_changed() to disable all vrings > >> when destroy_device() is called. > >> > > That would be good. > > > > > >>> 5. new_device and struct virtio_net: Determine NUMA node of the VM. > >> You can get the 'struct virtio_net' dev from all above callbacks. > > > > > >> 1. Link status interrupt. > >> > >> To vhost pmd, new_device()/destroy_device() equals to the link status > >> interrupt, where new_device() is a link up, and destroy_device() is link > >> down(). > >> > >> > >>> 2. New queue_state_changed callback. Unlike vring_state_changed this > >> should > >>> cover the first queue at new_device and removal of all queues at > >>> destroy_device. > >> As stated above, vring_state_changed() should be able to do that, except > >> the one on destroy_device(), which is not done yet. > >> > >>> 3. Per-queue or per-device NUMA node info. > >> You can query the NUMA node info implicitly by get_mempolicy(); check > >> numa_realloc() at lib/librte_vhost/virtio-net.c for reference. > >> > > Your suggestions are exactly how my application is already working. I was > > commenting on the > > proposed changes to the vhost PMD API. I would prefer to > > use RTE_ETH_EVENT_INTR_LSC > > and rte_eth_dev_socket_id for consistency with other NIC drivers, instead > > of these vhost-specific > > hacks. The queue state change callback is the one new API that needs to > be > > added because > > normal NICs don't have this behavior. > > > > You could add another rte_eth_event_type for the queue state change > > callback, and pass the > > queue ID, RX/TX direction, and enable bit through cb_arg. > > Hi Rich, > > So far, EAL provides rte_eth_dev_callback_register() for event handling. > DPDK app can register callback handler and "callback argument". > And EAL will call callback handler with the argument. > Anyway, vhost library and PMD cannot change the argument. > You're right, I'd mistakenly thought that the PMD controlled the void * passed to the callback. Here's a thought: struct rte_eth_vhost_queue_event { uint16_t queue_id; bool rx; bool enable; }; int rte_eth_vhost_get_queue_event(uint8_t port_id, struct rte_eth_vhost_queue_event *event); On receiving the ethdev event the application could repeatedly call rte_eth_vhost_get_queue_event to find out what happened. An issue with having the application dig into struct virtio_net is that it can only be safely accessed from a callback on the vhost thread. A typical application running its control plane on lcore 0 would need to copy all the relevant info from struct virtio_net before sending it over. As you mentioned, queues for a single vhost port could be located on different NUMA nodes. I think this is an uncommon scenario but if needed you could add an API to retrieve the NUMA node for a given port and queue.