From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yb0-f172.google.com (mail-yb0-f172.google.com [209.85.213.172]) by dpdk.org (Postfix) with ESMTP id 830E32BF6 for ; Thu, 15 Dec 2016 04:42:17 +0100 (CET) Received: by mail-yb0-f172.google.com with SMTP id d59so26179693ybi.1 for ; Wed, 14 Dec 2016 19:42:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telaverge-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Ngov5NIws/IF1JX3lEdK3gh6GBH5Tfa5tzRDaJ7eSOA=; b=YhHp9pn0Hl6tb1/AM8Hzl+pk9MfS80JzmJwCMe/XnW6FmMRvjGDvr3exf1ft7J00t7 gnzeZ+bwWC8M1kT3zlKI4vcbuJQ4WZoPmKz8H5Az3p1VYw9tVzttOyFzO5sgesVT4i/j dxM1EC48MpfrI95D8YYDJ9HBGjKGE0nU2rwiHBqOdIezZrkDW3GGOCYUtRv1IYDp1n3y hPKXJso8CJLPuLLQ5C41/i5O1hroSSTYibAqL68yCrSFLblZxwn7kSVmowZ/TuMjOf7S E5FsbOJYCeilI+3GvRfRlkjj5tFqCYz81pNWoRIV32m5q1iqtNT1XCS8/aNZJ7z0C3jp BZ7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Ngov5NIws/IF1JX3lEdK3gh6GBH5Tfa5tzRDaJ7eSOA=; b=fB37A0W7kcVop6jXO3SnMl2Qn6xDN7M3Ce4DaTWiWBfB76SMnSh/xeirLV258l36Hv B8vHezQDE18zG5xKkdrAlYHansEoYRCIByx9YjzPR1UJb/gcWLvHMUr4FnPGj8KNLC6V EclUWR3tMp3h5Q4CJTYbUMJ0td1YSrg1/LfGvzVIXwWoQ4WXaLQiCoLjYTDDLXZvMMSJ CmNhkTWmt9k1GQF35GVgfa/IbIDFDgiz1YjTCIdAL6WUAEgiusScUN2cBmYYyvnWzMOT MH4Ex7B+8edT8uZ80J73FniTGxCeFB/dqVKd2/q3Zc4tnNFqR2CrbabLl4sDBLlDmkZg cQiQ== X-Gm-Message-State: AIkVDXKjAZXeCoC/yYy5txlxwY0vY24oEmO4Owqu+PpG6FiL4mDv1dN37aGKIsyN9EIPV1IDaF2KoeI7vaIhDA== X-Received: by 10.37.216.137 with SMTP id p131mr149093ybg.42.1481773336830; Wed, 14 Dec 2016 19:42:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.111.4 with HTTP; Wed, 14 Dec 2016 19:42:16 -0800 (PST) X-Originating-IP: [182.72.141.194] In-Reply-To: <20161213142241.0eef9572@xeon-e3> References: <20161213142241.0eef9572@xeon-e3> From: Prem Chaitanya Date: Thu, 15 Dec 2016 09:12:16 +0530 Message-ID: To: Stephen Hemminger Cc: users@dpdk.org, Ramana Babu , azad N , Sanjay Raju , Piyush Vijay Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-users] Facing issues with Interrupt mode (IGB,VFIO) X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Dec 2016 03:42:17 -0000 Hi Stephen, Thanks a lot for your response. We wrote our application(by modifying l3fwd application) and by looking at the l3fwd-power sample application. But didnt know about this race you are talking about. Question: If we enable interrupt and go into epoll_wait, wont the interrupts be pending in eventfd ? Or am i missing something. I applied the queue_count check and tried to read again in case there are packets. But still i see there is a lot of inconsistency in receiving interrupts. After i did this change, first time i ran the application i saw interrupts coming in, but from subsequent time i don't see them anymore. Here is the our code flow. 1. Register for interrupts 2. Poll Rx-Queue. 3. If idle_poll crosses configured limit, turn-on interrupt for that queue and goto epoll_wait 4. One interrupt is received then disable interrupt for that queue and poll. Let me know if you need any specific information. On Wed, Dec 14, 2016 at 3:52 AM, Stephen Hemminger < stephen@networkplumber.org> wrote: > On Tue, 13 Dec 2016 14:43:41 +0530 > Prem Chaitanya wrote: > > > Hi All, > > > > We modified the l3fwd application to run in interrupt+poll mode. The > driver > > we used was IGB with Intel I350 NIC. > > When we ran with 1 lcore and 1 queue, there were no issues. When we tried > > to run with 2 lcores and 2 Rx-queues we are not getting interrupts. > > In one of the documentation(DPDK-201.pdf), it was mentioned that IGB > driver > > doesn't support interrupt mode with multiple lcores since it supports > only > > 1 epoll-fd. > > Can you please confirm if this is the case? > > igb_uio device driver can not support an interrupt per queue (MSI). > > > > Then we went ahead and loaded the VFIO driver and tried to run the same > > application. > > Now we get an interrupt once and after that when we go back to waiting on > > epoll, we dont receive any further interrupts. > > But when we print the stats for rx-packets we see that there are packets > > waiting in the NIC Rx queue. > > Same behavior is observed even when we run application with 1 lcore and 1 > > Rx queue. > > You need to do some careful coding to handle interrupt+poll mode. > Look at l3fwd-power example, and even that isn't enough. > > Basically, you need to dynamically go in and out of interrupt mode > and use the rx_queue_count function to check for races. > > The application must have poll loop like: > > while() { > rte_epoll_wait() > > if (rx_event) { > readmore: > rte_eth_dev_rx_intr_disable() > > while ( (n = rte_eth_rx_queue_burst()) > 0) { > process n packets; > } > > rte_eth_dev_rx_intr_enable(); > if (rte_eth_dev_rx_queue_count() > 0) { > // lost race > goto readmore; > } > } > } > -- Regards Prem