From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 55812A034C; Tue, 18 Aug 2020 16:21:57 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9728014581; Tue, 18 Aug 2020 16:21:55 +0200 (CEST) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.120]) by dpdk.org (Postfix) with ESMTP id 83FBF2BAA for ; Tue, 18 Aug 2020 16:21:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1597760514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Rz1n4xPGnrJhYe1Uu1gjTrzT0QKWzhtrNm4U9HHcwqE=; b=M2LfJzf25L9JvT7S1gCG/CsI/5MBYybEkwNw3I1l+H1s5H5CP3AxX8mSyGPtCN66pN5GIC JuGFl0g2JImn1NIEqu8njFLShTQxMovwpK+RarAM5p8t4rVRhzwA6ItSCwljfmLwy280L4 B9sZ9sj3LiSraXT5HT4JO78dKMs6O+Y= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-526-W2hN29akNSubWIdGHvmcXA-1; Tue, 18 Aug 2020 10:21:52 -0400 X-MC-Unique: W2hN29akNSubWIdGHvmcXA-1 Received: by mail-lf1-f70.google.com with SMTP id c204so4490159lfg.16 for ; Tue, 18 Aug 2020 07:21:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Rz1n4xPGnrJhYe1Uu1gjTrzT0QKWzhtrNm4U9HHcwqE=; b=LTBlYJXVZCY0DTSC1sTJtRP4289BvvrdH+Wrbl5NffGWpj5R93DI/OHcEHVG4YM7kE +fp3DKzufmgeq41wQxSfIzTyOv6q1AGZU3lr06gXPXJBLfsKg2TOWZKqlDulQ/+sWfK/ pEvERJij2F9BIdcgkdjcgmahfiPb/2c0DnJNtw8tFienPoD/UVMdrtVNOKQn6drE32JU GshFtUaEBDmfBAJoyq34mgS6TOOKZYJnNF5UuinpLBBhJBaIiRzWCScIpBSUZ6JL6BH8 NguTP/cGvqiItD+dql5OEacvz1G56QdSl26pyZ11Je5lCbE3AorYAQadKc2gUfbjVq+7 Ls5A== X-Gm-Message-State: AOAM53269yg68baZMtr9jLqaYUZhSrWuh8ylnaDnRMYh4N/bJtTUE2xS DZdthAwnDlbVrooyPBIMtVaJi9oPfs4efCYtq4urgWuArsubnKcveEQL2CyjwCyxfW+ETEXZkMw 2DllFE7ywT1RrsCJuWOw= X-Received: by 2002:ac2:5bc2:: with SMTP id u2mr9895963lfn.46.1597760509428; Tue, 18 Aug 2020 07:21:49 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx5NbaeRpnZlNbRMIdj4rp0l5juLifO2YeKq9P4BmUuu5z7KQ1DlniE3oU1883wfqwLvkB9H13B29tNP5fWt8Q= X-Received: by 2002:ac2:5bc2:: with SMTP id u2mr9895950lfn.46.1597760509046; Tue, 18 Aug 2020 07:21:49 -0700 (PDT) MIME-Version: 1.0 References: <20200810141103.8015-1-eperezma@redhat.com> In-Reply-To: <20200810141103.8015-1-eperezma@redhat.com> From: Eugenio Perez Martin Date: Tue, 18 Aug 2020 16:21:12 +0200 Message-ID: To: dev@dpdk.org Cc: Adrian Moreno Zapata , Maxime Coquelin , stable@dpdk.org, Zhihong Wang , Chenbo Xia Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=eperezma@redhat.com X-Mimecast-Spam-Score: 0.002 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [PATCH 0/1] vhost: fix iotlb mempool single-consumer flag X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On Mon, Aug 10, 2020 at 4:11 PM Eugenio P=C3=A9rez wr= ote: > > Bugzilla bug: 523 > > Using testpmd as a vhost-user with iommu: > > /home/dpdk/build/app/dpdk-testpmd -l 1,3 \ > --vdev net_vhost0,iface=3D/tmp/vhost-user1,queues=3D1,iommu-suppo= rt=3D1 \ > -- --auto-start --stats-period 5 --forward-mode=3Dtxonly > > And qemu with packed virtqueue: > > > > > > >
> > ... > > > > > > > -- > > Is it possible to consume the iotlb's entries of the mempoo from differen= t > threads. Thread sanitizer example output (after change rwlocks to POSIX o= nes): > > WARNING: ThreadSanitizer: data race (pid=3D76927) > Write of size 8 at 0x00017ffd5628 by thread T5: > #0 vhost_user_iotlb_cache_insert ../lib/librte_vhost/iotlb.c:181 (dpd= k-testpmd+0x769343) > #1 vhost_user_iotlb_msg ../lib/librte_vhost/vhost_user.c:2380 (dpdk-t= estpmd+0x78e4bf) > #2 vhost_user_msg_handler ../lib/librte_vhost/vhost_user.c:2848 (dpdk= -testpmd+0x78fcf8) > #3 vhost_user_read_cb ../lib/librte_vhost/socket.c:311 (dpdk-testpmd+= 0x770162) > #4 fdset_event_dispatch ../lib/librte_vhost/fd_man.c:286 (dpdk-testpm= d+0x7591c2) > #5 ctrl_thread_init ../lib/librte_eal/common/eal_common_thread.c:193 = (dpdk-testpmd+0xa2890b) > #6 (libtsan.so.0+0x2a68d) > > Previous read of size 8 at 0x00017ffd5628 by thread T3: > #0 vhost_user_iotlb_cache_find ../lib/librte_vhost/iotlb.c:252 (dpdk-= testpmd+0x76ee96) > #1 __vhost_iova_to_vva ../lib/librte_vhost/vhost.c:42 (dpdk-testpmd+0= x77488c) > #2 vhost_iova_to_vva ../lib/librte_vhost/vhost.h:753 (dpdk-testpmd+0x= 7abeb3) > #3 map_one_desc ../lib/librte_vhost/virtio_net.c:497 (dpdk-testpmd+0x= 7abeb3) > #4 fill_vec_buf_packed ../lib/librte_vhost/virtio_net.c:751 (dpdk-tes= tpmd+0x7abeb3) > #5 vhost_enqueue_single_packed ../lib/librte_vhost/virtio_net.c:1170 = (dpdk-testpmd+0x7abeb3) > #6 virtio_dev_rx_single_packed ../lib/librte_vhost/virtio_net.c:1346 = (dpdk-testpmd+0x7abeb3) > #7 virtio_dev_rx_packed ../lib/librte_vhost/virtio_net.c:1384 (dpdk-t= estpmd+0x7abeb3) > #8 virtio_dev_rx ../lib/librte_vhost/virtio_net.c:1435 (dpdk-testpmd+= 0x7b0654) > #9 rte_vhost_enqueue_burst ../lib/librte_vhost/virtio_net.c:1465 (dpd= k-testpmd+0x7b0654) > #10 eth_vhost_tx ../drivers/net/vhost/rte_eth_vhost.c:470 (dpdk-testp= md+0x1ddfbd8) > #11 rte_eth_tx_burst ../lib/librte_ethdev/rte_ethdev.h:4800 (dpdk-tes= tpmd+0x505fdb) > #12 pkt_burst_transmit ../app/test-pmd/txonly.c:365 (dpdk-testpmd+0x5= 106ad) > #13 run_pkt_fwd_on_lcore ../app/test-pmd/testpmd.c:2080 (dpdk-testpmd= +0x4f8951) > #14 start_pkt_forward_on_core ../app/test-pmd/testpmd.c:2106 (dpdk-te= stpmd+0x4f89d7) > #15 eal_thread_loop ../lib/librte_eal/linux/eal_thread.c:127 (dpdk-te= stpmd+0xa5b20a) > #16 (libtsan.so.0+0x2a68d) > > Location is global '' at 0x000000000000 (rtemap_0+0x00003ffd5628) > > Thread T5 'vhost-events' (tid=3D76933, running) created by main thread = at: > #0 pthread_create (libtsan.so.0+0x2cd42) > #1 rte_ctrl_thread_create ../lib/librte_eal/common/eal_common_thread.= c:216 (dpdk-testpmd+0xa289e7) > #2 rte_vhost_driver_start ../lib/librte_vhost/socket.c:1190 (dpdk-tes= tpmd+0x7728ef) > #3 vhost_driver_setup ../drivers/net/vhost/rte_eth_vhost.c:1028 (dpdk= -testpmd+0x1de233d) > #4 eth_dev_configure ../drivers/net/vhost/rte_eth_vhost.c:1126 (dpdk-= testpmd+0x1de29cc) > #5 rte_eth_dev_configure ../lib/librte_ethdev/rte_ethdev.c:1439 (dpdk= -testpmd+0x991ce2) > #6 start_port ../app/test-pmd/testpmd.c:2450 (dpdk-testpmd+0x4f9b45) > #7 main ../app/test-pmd/testpmd.c:3777 (dpdk-testpmd+0x4fe1ac) > > Thread T3 'lcore-slave-3' (tid=3D76931, running) created by main thread= at: > #0 pthread_create (libtsan.so.0+0x2cd42) > #1 rte_eal_init ../lib/librte_eal/linux/eal.c:1244 (dpdk-testpmd+0xa4= 6e2b) > #2 main ../app/test-pmd/testpmd.c:3673 (dpdk-testpmd+0x4fdd75) > > -- > > Or: > WARNING: ThreadSanitizer: data race (pid=3D76927) > Write of size 1 at 0x00017ffd00f8 by thread T5: > #0 vhost_user_iotlb_cache_insert ../lib/librte_vhost/iotlb.c:182 (dpd= k-testpmd+0x769370) > #1 vhost_user_iotlb_msg ../lib/librte_vhost/vhost_user.c:2380 (dpdk-t= estpmd+0x78e4bf) > #2 vhost_user_msg_handler ../lib/librte_vhost/vhost_user.c:2848 (dpdk= -testpmd+0x78fcf8) > #3 vhost_user_read_cb ../lib/librte_vhost/socket.c:311 (dpdk-testpmd+= 0x770162) > #4 fdset_event_dispatch ../lib/librte_vhost/fd_man.c:286 (dpdk-testpm= d+0x7591c2) > #5 ctrl_thread_init ../lib/librte_eal/common/eal_common_thread.c:193 = (dpdk-testpmd+0xa2890b) > #6 (libtsan.so.0+0x2a68d) > > Previous write of size 1 at 0x00017ffd00f8 by thread T3: > #0 vhost_user_iotlb_pending_insert ../lib/librte_vhost/iotlb.c:86 (dp= dk-testpmd+0x75eb0c) > #1 __vhost_iova_to_vva ../lib/librte_vhost/vhost.c:58 (dpdk-testpmd+0= x774926) > #2 vhost_iova_to_vva ../lib/librte_vhost/vhost.h:753 (dpdk-testpmd+0x= 7a79d1) > #3 virtio_dev_rx_batch_packed ../lib/librte_vhost/virtio_net.c:1295 (= dpdk-testpmd+0x7a79d1) > #4 virtio_dev_rx_packed ../lib/librte_vhost/virtio_net.c:1376 (dpdk-t= estpmd+0x7a79d1) > #5 virtio_dev_rx ../lib/librte_vhost/virtio_net.c:1435 (dpdk-testpmd+= 0x7b0654) > #6 rte_vhost_enqueue_burst ../lib/librte_vhost/virtio_net.c:1465 (dpd= k-testpmd+0x7b0654) > #7 eth_vhost_tx ../drivers/net/vhost/rte_eth_vhost.c:470 (dpdk-testpm= d+0x1ddfbd8) > #8 rte_eth_tx_burst ../lib/librte_ethdev/rte_ethdev.h:4800 (dpdk-test= pmd+0x505fdb) > #9 pkt_burst_transmit ../app/test-pmd/txonly.c:365 (dpdk-testpmd+0x51= 06ad) > #10 run_pkt_fwd_on_lcore ../app/test-pmd/testpmd.c:2080 (dpdk-testpmd= +0x4f8951) > #11 start_pkt_forward_on_core ../app/test-pmd/testpmd.c:2106 (dpdk-te= stpmd+0x4f89d7) > #12 eal_thread_loop ../lib/librte_eal/linux/eal_thread.c:127 (dpdk-te= stpmd+0xa5b20a) > #13 (libtsan.so.0+0x2a68d) > > -- > > As a consequence, the two threads can modify the same entry of the mempoo= l. > Usually, this cause a loop in iotlb_pending_entries list. > > This behavior is only observed on packed vq + virtio-net kernel driver in= the > guest, so we could make the single-consumer flag optional. However, I hav= e not > found why I cannot see the issue in split, so the safer option is to neve= r set > it. > > Any comments? > > Thanks! > > Eugenio P=C3=A9rez (1): > vhost: fix iotlb mempool single-consumer flag > > lib/librte_vhost/iotlb.c | 3 +-- > 1 file changed, 1 insertion(+), 2 deletions(-) > > -- > 2.18.1 > Hi! Just a friendly ping on this. Should I CC somebody else? Thanks!