From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0433EA04C3; Fri, 22 Nov 2019 16:44:38 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C0ED62BAE; Fri, 22 Nov 2019 16:44:38 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by dpdk.org (Postfix) with ESMTP id 4E71823D for ; Fri, 22 Nov 2019 16:44:36 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id xAMFeDL8023925 for ; Fri, 22 Nov 2019 07:44:35 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding : content-type; s=pfpt0818; bh=xFUeNpl82yEJAlZMNWSvfYNbB2htPacbvp4g2lK2vGM=; b=uw3e9yEpSx8WWzZOPM7VEGyZpoSngauir93rzQkhbQUnpd/o6b36uwCzv44hDbr9jTiu R+ofwzZPCKWrXE3NyfcY7/5S91lrZMGlO2ncbz/iUVOoycFbKeGFZ9LkYvdNWwqGM5Uq Z1lgCqhE/2TWgZ4PQ1FacxHLG/YL0n2Ajiqa7M0gpL7Z6SMh4AlDqA33TvqwVfdYPyoy 752ECbsFw11kH9k8yT40X2HosMzkDd1+8+hclTU82RINopcEoMc3vN8yfDLLujaXL8gu eLC9iu7LDR0fudRn5W2F9XAL/tm7m1p9mBv2J1FRjbZwqABYJVI8j+W6sAe7TwiufMa5 Sw== Received: from sc-exch03.marvell.com ([199.233.58.183]) by mx0a-0016f401.pphosted.com with ESMTP id 2weafba245-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Fri, 22 Nov 2019 07:44:35 -0800 Received: from SC-EXCH03.marvell.com (10.93.176.83) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Fri, 22 Nov 2019 07:44:34 -0800 Received: from maili.marvell.com (10.93.176.43) by SC-EXCH03.marvell.com (10.93.176.83) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Fri, 22 Nov 2019 07:44:34 -0800 Received: from BG-LT7430.marvell.com (unknown [10.28.17.72]) by maili.marvell.com (Postfix) with ESMTP id B31D83F703F; Fri, 22 Nov 2019 07:44:32 -0800 (PST) From: To: , Pavan Nikhilesh CC: Date: Fri, 22 Nov 2019 21:14:26 +0530 Message-ID: <20191122154431.17416-1-pbhagavatula@marvell.com> X-Mailer: git-send-email 2.17.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.95,18.0.572 definitions=2019-11-22_03:2019-11-21,2019-11-22 signatures=0 Subject: [dpdk-dev] [PATCH v3 1/5] event/octeontx2: fix TIM HW race condition X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Pavan Nikhilesh Fix HW race condition observed when timeout resolution is low (<5us). When HW traverses a given TIM bucket it will clear chunk_remainder, but since SW always decreases the chunk_remainder at the start of the arm routine it might cause a race where SW updates chunk_remainder after HW has cleared it that lead to nasty side effects. Fixes: 95e4e4ec7469 ("event/octeontx2: add timer arm timeout burst") Signed-off-by: Pavan Nikhilesh --- drivers/event/octeontx2/otx2_tim_worker.h | 141 +++++++++++++++++++--- 1 file changed, 124 insertions(+), 17 deletions(-) diff --git a/drivers/event/octeontx2/otx2_tim_worker.h b/drivers/event/octeontx2/otx2_tim_worker.h index 50db6543c..c896b5433 100644 --- a/drivers/event/octeontx2/otx2_tim_worker.h +++ b/drivers/event/octeontx2/otx2_tim_worker.h @@ -7,6 +7,13 @@ #include "otx2_tim_evdev.h" +static inline uint8_t +tim_bkt_fetch_lock(uint64_t w1) +{ + return (w1 >> TIM_BUCKET_W1_S_LOCK) & + TIM_BUCKET_W1_M_LOCK; +} + static inline int16_t tim_bkt_fetch_rem(uint64_t w1) { @@ -188,7 +195,6 @@ tim_insert_chunk(struct otx2_tim_bkt * const bkt, } else { bkt->first_chunk = (uintptr_t)chunk; } - return chunk; } @@ -208,11 +214,38 @@ tim_add_entry_sp(struct otx2_tim_ring * const tim_ring, __retry: /* Get Bucket sema*/ - lock_sema = tim_bkt_fetch_sema(bkt); + lock_sema = tim_bkt_fetch_sema_lock(bkt); /* Bucket related checks. */ - if (unlikely(tim_bkt_get_hbt(lock_sema))) - goto __retry; + if (unlikely(tim_bkt_get_hbt(lock_sema))) { + if (tim_bkt_get_nent(lock_sema) != 0) { + uint64_t hbt_state; +#ifdef RTE_ARCH_ARM64 + asm volatile( + " ldaxr %[hbt], [%[w1]] \n" + " tbz %[hbt], 33, dne%= \n" + " sevl \n" + "rty%=: wfe \n" + " ldaxr %[hbt], [%[w1]] \n" + " tbnz %[hbt], 33, rty%= \n" + "dne%=: \n" + : [hbt] "=&r" (hbt_state) + : [w1] "r" ((&bkt->w1)) + : "memory" + ); +#else + do { + hbt_state = __atomic_load_n(&bkt->w1, + __ATOMIC_ACQUIRE); + } while (hbt_state & BIT_ULL(33)); +#endif + + if (!(hbt_state & BIT_ULL(34))) { + tim_bkt_dec_lock(bkt); + goto __retry; + } + } + } /* Insert the work. */ rem = tim_bkt_fetch_rem(lock_sema); @@ -224,14 +257,15 @@ tim_add_entry_sp(struct otx2_tim_ring * const tim_ring, chunk = tim_insert_chunk(bkt, tim_ring); if (unlikely(chunk == NULL)) { - tim_bkt_set_rem(bkt, 0); + bkt->chunk_remainder = 0; + tim_bkt_dec_lock(bkt); tim->impl_opaque[0] = 0; tim->impl_opaque[1] = 0; tim->state = RTE_EVENT_TIMER_ERROR; return -ENOMEM; } bkt->current_chunk = (uintptr_t)chunk; - tim_bkt_set_rem(bkt, tim_ring->nb_chunk_slots - 1); + bkt->chunk_remainder = tim_ring->nb_chunk_slots - 1; } else { chunk = (struct otx2_tim_ent *)(uintptr_t)bkt->current_chunk; chunk += tim_ring->nb_chunk_slots - rem; @@ -241,6 +275,7 @@ tim_add_entry_sp(struct otx2_tim_ring * const tim_ring, *chunk = *pent; tim_bkt_inc_nent(bkt); + tim_bkt_dec_lock(bkt); tim->impl_opaque[0] = (uintptr_t)chunk; tim->impl_opaque[1] = (uintptr_t)bkt; @@ -263,19 +298,60 @@ tim_add_entry_mp(struct otx2_tim_ring * const tim_ring, __retry: bkt = tim_get_target_bucket(tim_ring, rel_bkt, flags); - /* Get Bucket sema*/ lock_sema = tim_bkt_fetch_sema_lock(bkt); /* Bucket related checks. */ if (unlikely(tim_bkt_get_hbt(lock_sema))) { - tim_bkt_dec_lock(bkt); - goto __retry; + if (tim_bkt_get_nent(lock_sema) != 0) { + uint64_t hbt_state; +#ifdef RTE_ARCH_ARM64 + asm volatile( + " ldaxr %[hbt], [%[w1]] \n" + " tbz %[hbt], 33, dne%= \n" + " sevl \n" + "rty%=: wfe \n" + " ldaxr %[hbt], [%[w1]] \n" + " tbnz %[hbt], 33, rty%= \n" + "dne%=: \n" + : [hbt] "=&r" (hbt_state) + : [w1] "r" ((&bkt->w1)) + : "memory" + ); +#else + do { + hbt_state = __atomic_load_n(&bkt->w1, + __ATOMIC_ACQUIRE); + } while (hbt_state & BIT_ULL(33)); +#endif + + if (!(hbt_state & BIT_ULL(34))) { + tim_bkt_dec_lock(bkt); + goto __retry; + } + } } rem = tim_bkt_fetch_rem(lock_sema); - if (rem < 0) { +#ifdef RTE_ARCH_ARM64 + asm volatile( + " ldaxrh %w[rem], [%[crem]] \n" + " tbz %w[rem], 15, dne%= \n" + " sevl \n" + "rty%=: wfe \n" + " ldaxrh %w[rem], [%[crem]] \n" + " tbnz %w[rem], 15, rty%= \n" + "dne%=: \n" + : [rem] "=&r" (rem) + : [crem] "r" (&bkt->chunk_remainder) + : "memory" + ); +#else + while (__atomic_load_n(&bkt->chunk_remainder, + __ATOMIC_ACQUIRE) < 0) + ; +#endif /* Goto diff bucket. */ tim_bkt_dec_lock(bkt); goto __retry; @@ -294,17 +370,23 @@ tim_add_entry_mp(struct otx2_tim_ring * const tim_ring, tim->state = RTE_EVENT_TIMER_ERROR; return -ENOMEM; } - bkt->current_chunk = (uintptr_t)chunk; - tim_bkt_set_rem(bkt, tim_ring->nb_chunk_slots - 1); + *chunk = *pent; + while (tim_bkt_fetch_lock(lock_sema) != + (-tim_bkt_fetch_rem(lock_sema))) + lock_sema = __atomic_load_n(&bkt->w1, __ATOMIC_ACQUIRE); + + bkt->current_chunk = (uintptr_t)chunk; + __atomic_store_n(&bkt->chunk_remainder, + tim_ring->nb_chunk_slots - 1, __ATOMIC_RELEASE); } else { - chunk = (struct otx2_tim_ent *)(uintptr_t)bkt->current_chunk; + chunk = (struct otx2_tim_ent *)bkt->current_chunk; chunk += tim_ring->nb_chunk_slots - rem; + *chunk = *pent; } /* Copy work entry. */ - *chunk = *pent; - tim_bkt_dec_lock(bkt); tim_bkt_inc_nent(bkt); + tim_bkt_dec_lock(bkt); tim->impl_opaque[0] = (uintptr_t)chunk; tim->impl_opaque[1] = (uintptr_t)bkt; tim->state = RTE_EVENT_TIMER_ARMED; @@ -360,8 +442,33 @@ tim_add_entry_brst(struct otx2_tim_ring * const tim_ring, /* Bucket related checks. */ if (unlikely(tim_bkt_get_hbt(lock_sema))) { - tim_bkt_dec_lock(bkt); - goto __retry; + if (tim_bkt_get_nent(lock_sema) != 0) { + uint64_t hbt_state; +#ifdef RTE_ARCH_ARM64 + asm volatile( + " ldaxr %[hbt], [%[w1]] \n" + " tbz %[hbt], 33, dne%= \n" + " sevl \n" + "rty%=: wfe \n" + " ldaxr %[hbt], [%[w1]] \n" + " tbnz %[hbt], 33, rty%= \n" + "dne%=: \n" + : [hbt] "=&r" (hbt_state) + : [w1] "r" ((&bkt->w1)) + : "memory" + ); +#else + do { + hbt_state = __atomic_load_n(&bkt->w1, + __ATOMIC_ACQUIRE); + } while (hbt_state & BIT_ULL(33)); +#endif + + if (!(hbt_state & BIT_ULL(34))) { + tim_bkt_dec_lock(bkt); + goto __retry; + } + } } chunk_remainder = tim_bkt_fetch_rem(lock_sema); -- 2.17.1