From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mailout1.w1.samsung.com (mailout1.w1.samsung.com [210.118.77.11]) by dpdk.org (Postfix) with ESMTP id 50E1A5F3B for ; Mon, 3 Sep 2018 17:48:44 +0200 (CEST) Received: from eucas1p1.samsung.com (unknown [182.198.249.206]) by mailout1.w1.samsung.com (KnoxPortal) with ESMTP id 20180903154844euoutp013bd880b4a23768cc8f510ee1ab10f76b~Q7oQjBmk61709417094euoutp01s for ; Mon, 3 Sep 2018 15:48:44 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 mailout1.w1.samsung.com 20180903154844euoutp013bd880b4a23768cc8f510ee1ab10f76b~Q7oQjBmk61709417094euoutp01s DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=samsung.com; s=mail20170921; t=1535989724; bh=xYOzXoscztVulN/6wDYq6ONe8FWteL1hbVwUW7xrvtE=; h=Subject:To:Cc:From:Date:In-Reply-To:References:From; b=bGH2cqB5LdNW7qGOn8NgOs7yZOQOV5HoUodRQGCxCE5ZixahV8kHQ6MU1YN57RkPb L6+ajK1T6aRTmI1QefVss6SVIgDgXrC+z/jPB6sFYRNv5j/JTKU4SWolzeG0Ei4BO6 dVNy58KmXFroGIvY0llrDNbNmEm5kTNXMZI+dfeA= Received: from eusmges2new.samsung.com (unknown [203.254.199.244]) by eucas1p1.samsung.com (KnoxPortal) with ESMTP id 20180903154842eucas1p1d6ddb119fdab71bcc78b328be8e17ffd~Q7oPZSaBw2380923809eucas1p1Q; Mon, 3 Sep 2018 15:48:42 +0000 (GMT) Received: from eucas1p1.samsung.com ( [182.198.249.206]) by eusmges2new.samsung.com (EUCPMTA) with SMTP id 44.F7.04294.AD75D8B5; Mon, 3 Sep 2018 16:48:42 +0100 (BST) Received: from eusmtrp2.samsung.com (unknown [182.198.249.139]) by eucas1p1.samsung.com (KnoxPortal) with ESMTPA id 20180903154841eucas1p1016af55e5cd460a7e1029e20ad5c5e4a~Q7oN_vC0l2380923809eucas1p1N; Mon, 3 Sep 2018 15:48:41 +0000 (GMT) Received: from eusmgms2.samsung.com (unknown [182.198.249.180]) by eusmtrp2.samsung.com (KnoxPortal) with ESMTP id 20180903154841eusmtrp2d616f5c4ea9120b3a8fb30a27854baba~Q7oNsRnM_2438024380eusmtrp2d; Mon, 3 Sep 2018 15:48:41 +0000 (GMT) X-AuditID: cbfec7f4-84fff700000010c6-f7-5b8d57da1028 Received: from eusmtip1.samsung.com ( [203.254.199.221]) by eusmgms2.samsung.com (EUCPMTA) with SMTP id 7B.9E.04128.9D75D8B5; Mon, 3 Sep 2018 16:48:41 +0100 (BST) Received: from [106.109.129.180] (unknown [106.109.129.180]) by eusmtip1.samsung.com (KnoxPortal) with ESMTPA id 20180903154840eusmtip15fe7a649c5e5a1e9cb8c2cff07994b5e~Q7oNG-HSQ2574225742eusmtip1B; Mon, 3 Sep 2018 15:48:40 +0000 (GMT) To: "Wiles, Keith" Cc: "dev@dpdk.org" , "Wu, Jingjing" , "Ananyev, Konstantin" , "Lu, Wenzhuo" , "Xing, Beilei" , "Zhang, Qi Z" , "Wang, Xiao W" , "Richardson, Bruce" From: Ilya Maximets Date: Mon, 3 Sep 2018 18:50:12 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <832A4FA5-2615-4CD1-8B48-0176557BE6E6@intel.com> Content-Language: en-GB Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrAKsWRmVeSWpSXmKPExsWy7djPc7q3wnujDe591rDo/tDCZHFjlb3F u0/bmSyutP9kt5i54DOjxdXj35kt3v9ZxGIxfUM/m8XWM38ZLfY/P8zuwOXxa8FSVo/Fe14y efRtWcUYwBzFZZOSmpNZllqkb5fAlTHh9zz2gmnmFQ+mnmVrYDyu18XIySEhYCIxd3U/Uxcj F4eQwApGiRk/HzFDOF8YJfYdm8sO4XxmlJh/eiYrTMuB/X8YQWwhgeWMEv1fyiGKPjJKrNy/ jw0kISwQJfF0+2xmEFtEQEti4b83YJOYBa4ySbQc/MwOkmAT0JE4tfoI2CQWARWJB/8ug9mi AhESRx4sBLN5BQQlTs58wgJicwrYSkz83A5mMwuISzR9WckKYctLbH87B+xuCYFt7EDNl1gg msskvsxvYoc420Wi4eoSFghbWOLV8S1QcRmJ/zvnM0HY9RL3W14yQgzqYJSYfugfVMJeYsvr c0ANHEDbNCXW79KHCDtKLH17hgkkLCHAJ3HjrSDEPXwSk7ZNZ4YI80p0tAlBVKtI/D64nBnC lpK4+e4z+wRGpVlIvpyF5LNZSD6bhbB3ASPLKkbx1NLi3PTUYqO81HK94sTc4tK8dL3k/NxN jMB0dPrf8S87GHf9STrEKMDBqMTDG2DQGy3EmlhWXJl7iFGCg1lJhFcjDCjEm5JYWZValB9f VJqTWnyIUZqDRUmcl08rLVpIID2xJDU7NbUgtQgmy8TBKdXA2Ll1dsGxhX3M1nWX3m9gXXZ+ 6iubQ15XdCLcTdSbMqdvK52/edkiVk2VXf4lIu++x+vqcbSvYtVYf6ni4BM7thORE8SEgnxa nevWnPpQxin5af4s9ud6ZxP5U2+mHBZ4kr6FIdSed/K1uRvv7N4eMPPop6On3q7i9zp54NsW 4UpHK6OAjHLhDiWW4oxEQy3mouJEAGLR5qdDAwAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrMIsWRmVeSWpSXmKPExsVy+t/xu7o3w3ujDS5u4bDo/tDCZHFjlb3F u0/bmSyutP9kt5i54DOjxdXj35kt3v9ZxGIxfUM/m8XWM38ZLfY/P8zuwOXxa8FSVo/Fe14y efRtWcUYwBylZ1OUX1qSqpCRX1xiqxRtaGGkZ2hpoWdkYqlnaGwea2VkqqRvZ5OSmpNZllqk b5eglzHh9zz2gmnmFQ+mnmVrYDyu18XIySEhYCJxYP8fxi5GLg4hgaWMEgsuH2SFSEhJ/Ph1 AcoWlvhzrYsNoug9o8Tbj3+ZQBLCAlESa2ZMBSsSEdCSWPjvDTuIzSxwnUli6f8IiIbzzBLN z1aDNbAJ6EicWn2EEcTmFbCTmL1uKZjNIqAi8eDfZTBbVCBCYvXyF6wQNYISJ2c+YQGxOQVs JSZ+bmeBWKAu8WfeJWYIW1yi6ctKVghbXmL72znMExiFZiFpn4WkZRaSlllIWhYwsqxiFEkt Lc5Nzy020itOzC0uzUvXS87P3cQIjMBtx35u2cHY9S74EKMAB6MSD+8P3d5oIdbEsuLK3EOM EhzMSiK8GmFAId6UxMqq1KL8+KLSnNTiQ4ymQM9NZJYSTc4HJoe8knhDU0NzC0tDc2NzYzML JXHe8waVUUIC6YklqdmpqQWpRTB9TBycUg2MSv5sp+Ka+Ppv5L6YZdDDIhv6VSeON24Bm0T4 7XbZtAX5TdNZJ1mel1mwUkfyhNaPzsD2gO+HJ6l92W1wQ0D/hVqndXPIy2P7t4usiTHk/56o 8vCsy8rZuabbSq1iLJjq3BdGP+0qDmy/ff2zXj/PZ0dr/VV3V3Pyu517IxV4irO8+leXtKoS S3FGoqEWc1FxIgAHAJzT1gIAAA== Message-Id: <20180903154841eucas1p1016af55e5cd460a7e1029e20ad5c5e4a~Q7oN_vC0l2380923809eucas1p1N@eucas1p1.samsung.com> X-CMS-MailID: 20180903154841eucas1p1016af55e5cd460a7e1029e20ad5c5e4a X-Msg-Generator: CA Content-Type: text/plain; charset="utf-8" X-RootMTR: 20180831124404eucas1p20daff43600dfe450c9106616f886eab4 X-EPHeader: CA CMS-TYPE: 201P X-CMS-RootMailID: 20180831124404eucas1p20daff43600dfe450c9106616f886eab4 References: <20180831124517.27619-1-i.maximets@samsung.com> <20180831124404eucas1p20daff43600dfe450c9106616f886eab4~P_LLRV5aM1664316643eucas1p2w@eucas1p2.samsung.com> <20180903153131eucas1p122e24d751d0f87d0cff88f3360c50e37~Q7ZOaSu4X0341203412eucas1p1A@eucas1p1.samsung.com> <832A4FA5-2615-4CD1-8B48-0176557BE6E6@intel.com> Subject: Re: [dpdk-dev] [RFC 2/2] drivers/net: use sleep delay by default for intel NICs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Sep 2018 15:48:45 -0000 On 03.09.2018 18:39, Wiles, Keith wrote: > > >> On Sep 3, 2018, at 4:33 PM, Ilya Maximets wrote: >> >> On 03.09.2018 18:14, Wiles, Keith wrote: >>> >>> >>>> On Aug 31, 2018, at 1:45 PM, Ilya Maximets wrote: >>>> >>>> NICs uses different delays up to a second during their >>>> configuration. It makes no sense to busy-wait so long wasting >>>> CPU cycles and preventing any other threads to execute on the >>>> same CPU core. These busy polling are the rudiments that came >>>> from the kernel drivers where you can not sleep in interrupt >>>> context, but as we're in userspace, we're able and should >>>> sleep to allow other threads to run. >>>> Delays never called on rx/tx path, so this should not affect >>>> performance. >>> >>> I have a question, the only thread being effected by the busy wait is the thread assigned to the core or the master lcore in the case of rte_eal_init(). It should not effect other threads running on other cores, right? If you do have other threads running in the same code then I see the need, please help me understand the use and how it is effecting DPDK. >> >> Originally, this patch comes from this issue: >> http://mails.dpdk.org/archives/dev/2018-August/110640.html >> >> non-EAL threads is one of the answers. >> For example, main thread in OVS periodically checks the link statuses >> and hangs there busy waiting on different NIC configuration operations. > > We have link state get no wait why is that not working instead of wait polling the line state? Yes, and OVS uses it, but ixgbe driver contains the workaround for a fiber links configuration issue that calls ixgbe_setup_link, that busy waits trying to configure multispeed fiber. You may found details in the patch I mentioned. I moved this code to the separate alarm thread, but it will eat CPU cycles anyway preventing others from working. > >> Also, main OVS tread is responsible for port configuration and >> re-configuration. There are few other in-dpdk threads like interrupts >> handling thread (alarm handling thread). vhost_thread is working on >> the same lcores and wants some time to work too. >> >> In case of hyperthreading busy-waiting will significantly affect >> threads on the siblings. >> >> Best regards, Ilya Maximets. >> >>>> >>>> Signed-off-by: Ilya Maximets >>>> --- >>>> drivers/net/avf/Makefile | 1 + >>>> drivers/net/avf/base/avf_osdep.h | 4 ++-- >>>> drivers/net/e1000/Makefile | 1 + >>>> drivers/net/e1000/base/e1000_osdep.h | 2 +- >>>> drivers/net/i40e/base/i40e_osdep.h | 6 +++--- >>>> drivers/net/ifc/base/ifcvf_osdep.h | 2 +- >>>> drivers/net/ixgbe/base/ixgbe_osdep.h | 2 +- >>>> 7 files changed, 10 insertions(+), 8 deletions(-) >>>> >>>> diff --git a/drivers/net/avf/Makefile b/drivers/net/avf/Makefile >>>> index 3f815bbc4..8ee707529 100644 >>>> --- a/drivers/net/avf/Makefile >>>> +++ b/drivers/net/avf/Makefile >>>> @@ -9,6 +9,7 @@ include $(RTE_SDK)/mk/rte.vars.mk >>>> LIB = librte_pmd_avf.a >>>> >>>> CFLAGS += -O3 >>>> +CFLAGS += -DALLOW_EXPERIMENTAL_API >>>> LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring >>>> LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs -lrte_hash >>>> LDLIBS += -lrte_bus_pci >>>> diff --git a/drivers/net/avf/base/avf_osdep.h b/drivers/net/avf/base/avf_osdep.h >>>> index 9ef45968e..442a5acd0 100644 >>>> --- a/drivers/net/avf/base/avf_osdep.h >>>> +++ b/drivers/net/avf/base/avf_osdep.h >>>> @@ -93,8 +93,8 @@ typedef uint64_t u64; >>>> #define avf_memset(a, b, c, d) memset((a), (b), (c)) >>>> #define avf_memcpy(a, b, c, d) rte_memcpy((a), (b), (c)) >>>> >>>> -#define avf_usec_delay(x) rte_delay_us(x) >>>> -#define avf_msec_delay(x) rte_delay_us(1000*(x)) >>>> +#define avf_usec_delay(x) rte_delay_us_sleep(x) >>>> +#define avf_msec_delay(x) avf_usec_delay(1000 * (x)) >>>> >>>> #define AVF_PCI_REG(reg) rte_read32(reg) >>>> #define AVF_PCI_REG_ADDR(a, reg) \ >>>> diff --git a/drivers/net/e1000/Makefile b/drivers/net/e1000/Makefile >>>> index 9c87e883b..0ed627656 100644 >>>> --- a/drivers/net/e1000/Makefile >>>> +++ b/drivers/net/e1000/Makefile >>>> @@ -10,6 +10,7 @@ LIB = librte_pmd_e1000.a >>>> >>>> CFLAGS += -O3 >>>> CFLAGS += $(WERROR_FLAGS) >>>> +CFLAGS += -DALLOW_EXPERIMENTAL_API >>>> LDLIBS += -lrte_eal -lrte_mbuf -lrte_mempool -lrte_ring >>>> LDLIBS += -lrte_ethdev -lrte_net -lrte_kvargs >>>> LDLIBS += -lrte_bus_pci >>>> diff --git a/drivers/net/e1000/base/e1000_osdep.h b/drivers/net/e1000/base/e1000_osdep.h >>>> index b8868049f..5958ea157 100644 >>>> --- a/drivers/net/e1000/base/e1000_osdep.h >>>> +++ b/drivers/net/e1000/base/e1000_osdep.h >>>> @@ -48,7 +48,7 @@ >>>> >>>> #include "../e1000_logs.h" >>>> >>>> -#define DELAY(x) rte_delay_us(x) >>>> +#define DELAY(x) rte_delay_us_sleep(x) >>>> #define usec_delay(x) DELAY(x) >>>> #define usec_delay_irq(x) DELAY(x) >>>> #define msec_delay(x) DELAY(1000*(x)) >>>> diff --git a/drivers/net/i40e/base/i40e_osdep.h b/drivers/net/i40e/base/i40e_osdep.h >>>> index 8e5c593c9..a6072e153 100644 >>>> --- a/drivers/net/i40e/base/i40e_osdep.h >>>> +++ b/drivers/net/i40e/base/i40e_osdep.h >>>> @@ -233,9 +233,9 @@ struct i40e_spinlock { >>>> #define i40e_memcpy(a, b, c, d) rte_memcpy((a), (b), (c)) >>>> >>>> #define DIV_ROUND_UP(n,d) (((n) + (d) - 1) / (d)) >>>> -#define DELAY(x) rte_delay_us(x) >>>> -#define i40e_usec_delay(x) rte_delay_us(x) >>>> -#define i40e_msec_delay(x) rte_delay_us(1000*(x)) >>>> +#define DELAY(x) rte_delay_us_sleep(x) >>>> +#define i40e_usec_delay(x) DELAY(x) >>>> +#define i40e_msec_delay(x) DELAY(1000 * (x)) >>>> #define udelay(x) DELAY(x) >>>> #define msleep(x) DELAY(1000*(x)) >>>> #define usleep_range(min, max) msleep(DIV_ROUND_UP(min, 1000)) >>>> diff --git a/drivers/net/ifc/base/ifcvf_osdep.h b/drivers/net/ifc/base/ifcvf_osdep.h >>>> index cf151ef52..6aef25ea4 100644 >>>> --- a/drivers/net/ifc/base/ifcvf_osdep.h >>>> +++ b/drivers/net/ifc/base/ifcvf_osdep.h >>>> @@ -17,7 +17,7 @@ >>>> #define DEBUGOUT(S, args...) RTE_LOG(DEBUG, PMD, S, ##args) >>>> #define STATIC static >>>> >>>> -#define msec_delay rte_delay_ms >>>> +#define msec_delay(x) rte_delay_us_sleep(1000 * (x)) >>>> >>>> #define IFCVF_READ_REG8(reg) rte_read8(reg) >>>> #define IFCVF_WRITE_REG8(val, reg) rte_write8((val), (reg)) >>>> diff --git a/drivers/net/ixgbe/base/ixgbe_osdep.h b/drivers/net/ixgbe/base/ixgbe_osdep.h >>>> index bb5dfd2af..94ede9bc2 100644 >>>> --- a/drivers/net/ixgbe/base/ixgbe_osdep.h >>>> +++ b/drivers/net/ixgbe/base/ixgbe_osdep.h >>>> @@ -51,7 +51,7 @@ >>>> >>>> #define ASSERT(x) if(!(x)) rte_panic("IXGBE: x") >>>> >>>> -#define DELAY(x) rte_delay_us(x) >>>> +#define DELAY(x) rte_delay_us_sleep(x) >>>> #define usec_delay(x) DELAY(x) >>>> #define msec_delay(x) DELAY(1000*(x)) >>>> >>>> -- >>>> 2.17.1 >>>> >>> >>> Regards, >>> Keith > > Regards, > Keith > > >