From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by dpdk.org (Postfix) with ESMTP id 5403E2C8 for ; Mon, 19 Dec 2016 10:41:58 +0100 (CET) Received: by mail-wm0-f65.google.com with SMTP id a20so17565825wme.2 for ; Mon, 19 Dec 2016 01:41:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NN4bySHVWnpX2g7TDAr/RhDFotOElrcpQwCQ7buWqxk=; b=KVJ4M0fOL19pACr2SGVdB06u9ySIrKmyV2/MerP9tb7bHnyUlvyq7TaSWXROn63kpc nYVvgwzNRFgXsRGGpqzTYd+RMEJODyUuvcFXsZDKkUQfK05CDtsYjVpPs689F5b+a+8s kipdSxZ2zrLgaLmIVpkDu0lj+NAspv/vXJEAEk9H37VwpGx2aA/1KPeHgD4pBhq6uIZB 6TgRVweMTafK3ueRGDNEm7lSohaakkWEJs0vDTnk57iKa2fgeXf9VkOH0Oq+3tZU52Dd SWhgILSH3AeUx5EZ/kit/ySBXfd1NVL5dUyc2XwEPxnHlmmWETwXDO1sMt+1Hl9aejtF b0qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NN4bySHVWnpX2g7TDAr/RhDFotOElrcpQwCQ7buWqxk=; b=QRBS4Ofht1Fk8VIIixFc3VrcJ9igPN6qpKZPWnk55nSEK9EJzZhxyUYDBT/QQDOVN/ k28PCp/PDDi0K1RqXrKRI6sy4e21jUviwwtpIZYSZr2SGWyHlfQomxTrob/WdzXDq8Mq K2iSPGa1aPS/+yzhcm3FhD1wUFHVE1L9K/DXbrzkrW+GsqyztPRcdBTyKakCpSmTpyTl COHYRubEoyq5GAD/uahnSb3luXUBB8mjCljnTRiFvQQmF+9wTvUry6z7IlplnERSBSTU ZiudhaFOGUWOv/AGQHXonjRaBw221jDqu9e5ATMnBaRTnkIrPZMw435InOHf5QZhtCGu FVKg== X-Gm-Message-State: AIkVDXIUlas5/QCG9vZtj426yIOlBQHHoQZq/1BOE6s1nJgxRgYxlzElryeAN+GPkkzRWA== X-Received: by 10.28.52.76 with SMTP id b73mr13830148wma.8.1482140517745; Mon, 19 Dec 2016 01:41:57 -0800 (PST) Received: from spuiu-vm2.anuesystems.local ([109.100.41.154]) by smtp.gmail.com with ESMTPSA id f10sm19740365wjl.28.2016.12.19.01.41.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 19 Dec 2016 01:41:57 -0800 (PST) From: Stefan Puiu To: dev@dpdk.org Cc: yongwang@vmware.com, mac_leehk@yahoo.com.hk, Stefan Puiu Date: Mon, 19 Dec 2016 11:40:53 +0200 Message-Id: <1482140453-49649-1-git-send-email-stefan.puiu@gmail.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1481902617-16050-1-git-send-email-stefan.puiu@gmail.com> References: <1481902617-16050-1-git-send-email-stefan.puiu@gmail.com> Subject: [dpdk-dev] [PATCH v4] vmxnet3: fix Rx deadlock X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Dec 2016 09:41:58 -0000 Our use case is that we have an app that needs to keep mbufs around for a while. We've seen cases when calling vmxnet3_post_rx_bufs() from vmxet3_recv_pkts(), it might not succeed to add any mbufs to any RX descriptors (where it returns -err). Since there are no mbufs that the virtual hardware can use, no packets will be received after this; the driver won't refill the mbuf after this so it gets stuck in this state. I call this a deadlock for lack of a better term - the virtual HW waits for free mbufs, while the app waits for the hardware to notify it for data (by flipping the generation bit on the used Rx descriptors). Note that after this, the app can't recover. This fix is a rework of this patch by Marco Lee: http://dpdk.org/dev/patchwork/patch/6575/. I had to forward port it, address review comments and also reverted the allocation failure handling to the first version of the patch (http://dpdk.org/ml/archives/dev/2015-July/022079.html), since that's the only approach that seems to work, and seems to be what other drivers are doing (I checked ixgbe and em). Reusing the mbuf that's getting passed to the application doesn't seem to make sense, and it was causing weird issues in our app. Also, reusing rxm without checking if it's NULL could cause the code to crash. Signed-off-by: Stefan Puiu --- v4: * no change, just added sign-off v3: * rework description after review, explain how HW signals receipt of packets v2: * address review comments, reworded description a bit drivers/net/vmxnet3/vmxnet3_rxtx.c | 39 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/drivers/net/vmxnet3/vmxnet3_rxtx.c b/drivers/net/vmxnet3/vmxnet3_rxtx.c index b109168..93db10f 100644 --- a/drivers/net/vmxnet3/vmxnet3_rxtx.c +++ b/drivers/net/vmxnet3/vmxnet3_rxtx.c @@ -518,6 +518,32 @@ return nb_tx; } +static inline void +vmxnet3_renew_desc(vmxnet3_rx_queue_t *rxq, uint8_t ring_id, + struct rte_mbuf *mbuf) +{ + uint32_t val = 0; + struct vmxnet3_cmd_ring *ring = &rxq->cmd_ring[ring_id]; + struct Vmxnet3_RxDesc *rxd = + (struct Vmxnet3_RxDesc *)(ring->base + ring->next2fill); + vmxnet3_buf_info_t *buf_info = &ring->buf_info[ring->next2fill]; + + if (ring_id == 0) + val = VMXNET3_RXD_BTYPE_HEAD; + else + val = VMXNET3_RXD_BTYPE_BODY; + + buf_info->m = mbuf; + buf_info->len = (uint16_t)(mbuf->buf_len - RTE_PKTMBUF_HEADROOM); + buf_info->bufPA = rte_mbuf_data_dma_addr_default(mbuf); + + rxd->addr = buf_info->bufPA; + rxd->btype = val; + rxd->len = buf_info->len; + rxd->gen = ring->gen; + + vmxnet3_cmd_ring_adv_next2fill(ring); +} /* * Allocates mbufs and clusters. Post rx descriptors with buffer details * so that device can receive packets in those buffers. @@ -657,9 +683,18 @@ } while (rcd->gen == rxq->comp_ring.gen) { + struct rte_mbuf *newm; + if (nb_rx >= nb_pkts) break; + newm = rte_mbuf_raw_alloc(rxq->mp); + if (unlikely(newm == NULL)) { + PMD_RX_LOG(ERR, "Error allocating mbuf"); + rxq->stats.rx_buf_alloc_failure++; + break; + } + idx = rcd->rxdIdx; ring_idx = (uint8_t)((rcd->rqID == rxq->qid1) ? 0 : 1); rxd = (Vmxnet3_RxDesc *)rxq->cmd_ring[ring_idx].base + idx; @@ -759,8 +794,8 @@ VMXNET3_INC_RING_IDX_ONLY(rxq->cmd_ring[ring_idx].next2comp, rxq->cmd_ring[ring_idx].size); - /* It's time to allocate some new buf and renew descriptors */ - vmxnet3_post_rx_bufs(rxq, ring_idx); + /* It's time to renew descriptors */ + vmxnet3_renew_desc(rxq, ring_idx, newm); if (unlikely(rxq->shared->ctrl.updateRxProd)) { VMXNET3_WRITE_BAR0_REG(hw, rxprod_reg[ring_idx] + (rxq->queue_id * VMXNET3_REG_ALIGN), rxq->cmd_ring[ring_idx].next2fill); -- 1.9.1