From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <jianfeng.tan@intel.com>
Received: from mga02.intel.com (mga02.intel.com [134.134.136.20])
by dpdk.org (Postfix) with ESMTP id CB8592C0C
for <dev@dpdk.org>; Wed, 6 Apr 2016 08:05:54 +0200 (CEST)
Received: from fmsmga002.fm.intel.com ([10.253.24.26])
by orsmga101.jf.intel.com with ESMTP; 05 Apr 2016 23:05:54 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.24,447,1455004800"; d="scan'208";a="952659829"
Received: from shwdeisgchi083.ccr.corp.intel.com (HELO [10.239.67.193])
([10.239.67.193])
by fmsmga002.fm.intel.com with ESMTP; 05 Apr 2016 23:05:53 -0700
To: Yuanhan Liu <yuanhan.liu@linux.intel.com>
References: <1459872587-11655-1-git-send-email-ciara.loftus@intel.com>
<57049F5C.5080604@intel.com> <20160406054406.GV3080@yliu-dev.sh.intel.com>
Cc: Ciara Loftus <ciara.loftus@intel.com>, dev@dpdk.org, mukawa@igel.co.jp
From: "Tan, Jianfeng" <jianfeng.tan@intel.com>
Message-ID: <5704A73F.1040302@intel.com>
Date: Wed, 6 Apr 2016 14:05:51 +0800
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101
Thunderbird/38.7.1
MIME-Version: 1.0
In-Reply-To: <20160406054406.GV3080@yliu-dev.sh.intel.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
Subject: Re: [dpdk-dev] [PATCH] vhost: Fix retrieval of numa information in
PMD
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
X-List-Received-Date: Wed, 06 Apr 2016 06:05:55 -0000
On 4/6/2016 1:44 PM, Yuanhan Liu wrote:
> On Wed, Apr 06, 2016 at 01:32:12PM +0800, Tan, Jianfeng wrote:
>> Hi,
>>
>> Just out of interest, seems that the message handling thread which runs
>> new_device() is pthread_create() from the thread which calls the
>> dev_start(), usually master thread, right? But it's not necessary to be the
>> master thread to poll pkts from this vhost port, right? So what's the
>> significance to record the numa_node information of message handling thread
>> here? Shall we make the decision of numa_realloc based on the final PMD
>> thread who is responsible for polling this vhost port?
> It doesn't matter on which core we made the decision: the result
> would be same since we are querying the numa node info of the
> virtio_net dev struct.
OK, according to your hint, read numa_realloc() again, it's to keep *dev
(struct virtio_net), *dev->virtqueue[], on the same numa socket of
shared memory with virtio?
And numa_realloc() is called in vhost_set_vring_addr(), which is after
new_device()?
>
> --yliu
>> It's not related to this patch itself. And it seems good to me.
>>
>>
>> Thanks,
>> Jianfeng
>>
>>
>>
>> On 4/6/2016 12:09 AM, Ciara Loftus wrote:
>>> After some testing, it was found that retrieving numa information
>>> about a vhost device via a call to get_mempolicy is more
>>> accurate when performed during the new_device callback versus
>>> the vring_state_changed callback, in particular upon initial boot
>>> of the VM. Performing this check during new_device is also
>>> potentially more efficient as this callback is only triggered once
>>> during device initialisation, compared with vring_state_changed
>>> which may be called multiple times depending on the number of
>>> queues assigned to the device.
>>>
>>> Reorganise the code to perform this check and assign the correct
>>> socket_id to the device during the new_device callback.
>>>
>>> Signed-off-by: Ciara Loftus <ciara.loftus@intel.com>
>>> ---
>>> drivers/net/vhost/rte_eth_vhost.c | 28 ++++++++++++++--------------
>>> 1 file changed, 14 insertions(+), 14 deletions(-)
>>>
>>> diff --git a/drivers/net/vhost/rte_eth_vhost.c b/drivers/net/vhost/rte_eth_vhost.c
>>> index 4cc6bec..b1eb082 100644
>>> --- a/drivers/net/vhost/rte_eth_vhost.c
>>> +++ b/drivers/net/vhost/rte_eth_vhost.c
>>> @@ -229,6 +229,9 @@ new_device(struct virtio_net *dev)
>>> struct pmd_internal *internal;
>>> struct vhost_queue *vq;
>>> unsigned i;
>>> +#ifdef RTE_LIBRTE_VHOST_NUMA
>>> + int newnode, ret;
>>> +#endif
>>> if (dev == NULL) {
>>> RTE_LOG(INFO, PMD, "Invalid argument\n");
>>> @@ -244,6 +247,17 @@ new_device(struct virtio_net *dev)
>>> eth_dev = list->eth_dev;
>>> internal = eth_dev->data->dev_private;
>>> +#ifdef RTE_LIBRTE_VHOST_NUMA
>>> + ret = get_mempolicy(&newnode, NULL, 0, dev,
>>> + MPOL_F_NODE | MPOL_F_ADDR);
>>> + if (ret < 0) {
>>> + RTE_LOG(ERR, PMD, "Unknown numa node\n");
>>> + return -1;
>>> + }
>>> +
>>> + eth_dev->data->numa_node = newnode;
If it's correct of above analysis, dev, here, could be numa_realloc() later?
Thanks,
Jianfeng
>>> +#endif
>>> +
>>> for (i = 0; i < eth_dev->data->nb_rx_queues; i++) {
>>> vq = eth_dev->data->rx_queues[i];
>>> if (vq == NULL)
>>> @@ -352,9 +366,6 @@ vring_state_changed(struct virtio_net *dev, uint16_t vring, int enable)
>>> struct rte_vhost_vring_state *state;
>>> struct rte_eth_dev *eth_dev;
>>> struct internal_list *list;
>>> -#ifdef RTE_LIBRTE_VHOST_NUMA
>>> - int newnode, ret;
>>> -#endif
>>> if (dev == NULL) {
>>> RTE_LOG(ERR, PMD, "Invalid argument\n");
>>> @@ -370,17 +381,6 @@ vring_state_changed(struct virtio_net *dev, uint16_t vring, int enable)
>>> eth_dev = list->eth_dev;
>>> /* won't be NULL */
>>> state = vring_states[eth_dev->data->port_id];
>>> -
>>> -#ifdef RTE_LIBRTE_VHOST_NUMA
>>> - ret = get_mempolicy(&newnode, NULL, 0, dev,
>>> - MPOL_F_NODE | MPOL_F_ADDR);
>>> - if (ret < 0) {
>>> - RTE_LOG(ERR, PMD, "Unknown numa node\n");
>>> - return -1;
>>> - }
>>> -
>>> - eth_dev->data->numa_node = newnode;
>>> -#endif
>>> rte_spinlock_lock(&state->lock);
>>> state->cur[vring] = enable;
>>> state->max_vring = RTE_MAX(vring, state->max_vring);