From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id BECFAA04B7; Mon, 7 Sep 2020 12:09:18 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 647E11BEE1; Mon, 7 Sep 2020 12:09:18 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by dpdk.org (Postfix) with ESMTP id 4BBC61BE85 for ; Mon, 7 Sep 2020 12:09:17 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1599473356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=jq4hGhyM5NDiVj0Zv09Ghq0yu6IldvrN4wFpgIw3z7I=; b=HdkZPHkdcC9uZyzGt31FEzClqQNYpmwZLYsI8Vg3gqayXunQkq38VoJp9+y34r1g9Dk3R5 xRkGsUYS0/zJ3TUl9u0DbLtDpqT7TIdTof+unA7f1QERSfwEAwGI1KqRjvoG/fH7yBDHLV eMkbRdHFOiOzl7Pwd0kZo1VoVoVbW1c= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-488--SCvXsnHNheAPFTmXrs0RA-1; Mon, 07 Sep 2020 06:09:12 -0400 X-MC-Unique: -SCvXsnHNheAPFTmXrs0RA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6F5C61007469; Mon, 7 Sep 2020 10:09:11 +0000 (UTC) Received: from [10.33.36.212] (unknown [10.33.36.212]) by smtp.corp.redhat.com (Postfix) with ESMTP id 501155D9D2; Mon, 7 Sep 2020 10:09:06 +0000 (UTC) To: "Kinsella, Ray" , alangordondewar@gmail.com, cristian.dumitrescu@intel.com Cc: dev@dpdk.org, Alan Dewar , David Marchand , Luca Boccassi , Jasvinder Singh References: <20200416084821.12591-1-alan.dewar@att.com> <20200625095930.18429-1-alan.dewar@att.com> <6404e021-336b-eef3-c34a-99aa17c6c3b1@redhat.com> <406b64c8-0c88-5d71-2179-f48d8384b8e7@ashroe.eu> From: Kevin Traynor Autocrypt: addr=ktraynor@redhat.com; keydata= mQINBF2J2awBEADUEPNhgNI+nJNgiTAUcw4YIgVXEoHlsNPyyzG1BEXkWXALy0Y3fNTiw6+r ltWDkF9jzL9kfkecgQ67itGfk1OaBXgSGKuw1PUpxAwX2Bi76LAR6M5OsyGM9TSVVQwARalz hMwRBIZPzPc7or6Pw7jAOJ8SQGJ1Zlp1YJCjrvpe87V1tH/LY8Wnxn/EuoseFmWILAQZAtYS tGjcrAgYn3SPMLR1B0BP5bTBY06vWQjiufH8drenfDnMJAzuBdG1mqjnTqCjULZ3Hunv4xqZ aMnkvL/K5Tj1c12Oe4930EE53LrXIBUltRg5mBudSWHnC7twjH0082HH9f963Z/2UI63SFIT iUvRvAzJYytgy7XnWLQ0+goZBADKYfolOuC0H8VgCaux8u8KFF28Dy+N6TV2KI58jTlyg1Zu l7QwykZpnOkJFiy37Gfbu3YEOzO72cP/S7/A+zvuqkxi63jyEkd+FY99vLt/HN2MUZwRmKDw UPbLkmrs8WU01/POVsqDcfvz7vu2St8hqqTiSIdQGS2zyTKB2/DvPSM3jws3udkIYSuhn+X4 QBiV6lkVZ7DSE6a065gnAauAql+b32Eymy+xnG5jCt1tR+0Cp2VZYCR9OU2gmomUKBDoX/He pSgED01CqYPNjN+TddirwmQX7ep4DtXc8FWvv2g/pq9WZFQk2QARAQABtCNLZXZpbiBUcmF5 bm9yIDxrdHJheW5vckByZWRoYXQuY29tPokCTgQTAQgAOBYhBAoiOaH51tHF7VYtEI9CINER a+yJBQJdidmsAhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEI9CINERa+yJoxIP/3VF 2TIgW4ckxhRFCvFu/606bnvCPie88ake4uWVWMAWwcMc4fKEltRWRCpkSVOwgqoMHnyHxK5r kOKzx2CLJMX5TgTMfKzPuaBDHngHLUzl2DStpBzrod0cVg5TShdmmfjY61uxRJKz+DlSkwgJ riADdVF5PPosQXTkKSGf2ombpTGpx/pue9ocjnr3x4SDpRLlnooM6Jf/3Y3Ib4jX6HPEyWuY b+owIIk9y2nRRGPQ6jbqAhsrXd9V+77UL0QuGWloMuKMZFbNg8hbu7X5aFijAbfxj4YUgojS ba7gfGZQan8h32A9KGQWrmsCBc3j2GqEPsX0r05X7cn7WL6IOPgQJ5EiQ7PlazQYVLrvZg9B n0GKK0k6895mLG0ZZ5v/qajOPF52etSmvFD1WUPb4OqaHqGA9ZtMpaKFRt7Y6rpXqKNU1xzW F5KjbTPtTb9WF3An8dciVv+AYUI7totkZYkWvQtgss8lfaX3NKUvXLVxqK0z3dQyr7rF/tYz PneTKypSksjCgaEBLSrsRmM5zKfe7tSNF/fDntfIq/029Jtcw29TcWEP57peNu6TtejewQD9 sTI+oqiXvW2D5l7LNUDYG8eMJp2oT7I0ZSBRvwcbmjH0DtN/bXCCFfCvk8Yic68F3tV1ctix wQARVKDBhT30uCxycRWojCYqTgNJJS71uQINBF2J2awBEADP57PR2IpSYBeNSrsAjeIcsahE N4SQP2C4s50S8QEWAUhqMRI7WNv5cfeef0nDvcl1IUA6oz5SokbcsbMa+mRgaNF4N5KikWTO LPYxq2YVJoXwJ+tKmNzyOLFUIfFJ4NBJZple5dTfWzD00Dbb19Mri1hy1mWMqNTPGBee1+hw Qcp6n3mmGECvajs8G5A7NyXbwL8ihN7HX9D01ucD62b4G03yKe2g/hvKgcdUVmhCldJlF27I 2fSR9tDxH9pZqRODY4rjbFZEey/vWKXqjE+DQ8AtMSEaDfFe5D+i4Aw6erWQ3Wr+DwZt1/7G dIAElGA/q90T1ENVwJX9y7fsQssawKYYdDqURHCl5JuDXI+VXUypExipUUT5SPycMmbLsx0D iKEqPPDQWKxkIDVKqj2+EhamSuJznZUwBLJKn0h4zrIWiXWUy07lRwtVuhaDXhF3GfW+5W/x wAg7Qg3w00ASsb/XTHBIhMnenKDfS7ihtQA8SacwX8ySdxb+15XPyiplM979qBQ0mhnilulm MIJzEf/JxoYR5huuj4f1PFqqrsP06Dl+YGB7dQZp3IKggS5c3/TAynARRg9N89UsDXNtp7X0 tgIPFF5k6fnHE0J5O64GYHeTqN/1aE6dAEOV9WrGzQAJxU9ipikb8jKAWXzLewRIKGmoPcRZ WdB0NmIjmQARAQABiQI2BBgBCAAgFiEECiI5ofnW0cXtVi0Qj0Ig0RFr7IkFAl2J2awCGwwA CgkQj0Ig0RFr7IkkORAAl/NbX93WK5MEoRw7/DaPTo/Lo6Pj1XMeSqGyACigHK/452UDvlEH NjNJMzYYrNIjMtEmN9VVCfjT38CSca7mpGQVwchc0mC7QSPAETLCS+UacVf/Kwxz5FfkEUUw UT7A+uyVOIgW3d9ldlRzkHA2czonSSgTQU+i2g6DM4ha+BuQb4byAXH6HQHt/Zh1J64z0ohH v6iGsCzCY/sMWF8+LEGSnzMGRCLiiwSF0vJBHbzWK68fANaF4gBV0Z/+6tQRFN7YMhj/INmk qgvHj1ZzHFNtirjMGPRxoZs51YoLQM/aBPxKrnmXThx1ufH+0L6sGmFTugiDt0XSEkC5reH7 a+VhQ1VTFFQrClA8NmDSPzFeuhru4ryaaDHO+uEB16cNHxHrQtlP/2hts2JM5lwkZRWJ5A57 h8eDEIK5be47T85NVHfuTaboNRmgg1HygVejhGUtt69u/0MVRg/roUTa0FyEbNsvz4qAecyW yWzMcVrcGJDQLC9JLKEpoyUF6gdTKaiDL2Vao4+XRIA3Y57b6MO35a3HuzAv7+i5Z0mnDEJO XxXqTOmKYpMIGexzM/PtuA0712sT1abG9tAJ17ao/B7cqMW5IkKkalemFbWfI2unns4Papvo tk9igVqyp6EJDU98z5TJioCVojwK2laDaoIjTJk9YYv3iwCsqPd5feU= Message-ID: <73ab57ba-c69b-157e-9f3b-dc14fa40c929@redhat.com> Date: Mon, 7 Sep 2020 11:09:05 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <406b64c8-0c88-5d71-2179-f48d8384b8e7@ashroe.eu> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ktraynor@redhat.com X-Mimecast-Spam-Score: 0.002 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US Subject: Re: [dpdk-dev] [PATCH v2] sched: fix port time rounding error X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 21/08/2020 16:28, Kinsella, Ray wrote: > > > On 20/08/2020 15:32, Kevin Traynor wrote: >> Hi, >> >> On 25/06/2020 10:59, alangordondewar@gmail.com wrote: >>> From: Alan Dewar >>> >>> The QoS scheduler works off port time that is computed from the number >>> of CPU cycles that have elapsed since the last time the port was >>> polled. It divides the number of elapsed cycles to calculate how >>> many bytes can be sent, however this division can generate rounding >>> errors, where some fraction of a byte sent may be lost. >>> >>> Lose enough of these fractional bytes and the QoS scheduler >>> underperforms. The problem is worse with low bandwidths. >>> >>> To compensate for this rounding error this fix doesn't advance the >>> port's time_cpu_cycles by the number of cycles that have elapsed, >>> but by multiplying the computed number of bytes that can be sent >>> (which has been rounded down) by number of cycles per byte. >>> This will mean that port's time_cpu_cycles will lag behind the CPU >>> cycles momentarily. At the next poll, the lag will be taken into >>> account. >>> >>> v2: >>> If the cycles value wraps (100 year+) reset the port's cpu cycle back >>> to zero. >>> >>> Fixes: de3cfa2c98 ("sched: initial import") >>> >>> Signed-off-by: Alan Dewar >>> --- >>> lib/librte_sched/rte_sched.c | 11 +++++++++-- >>> 1 file changed, 9 insertions(+), 2 deletions(-) >>> >>> diff --git a/lib/librte_sched/rte_sched.c b/lib/librte_sched/rte_sched.c >>> index c0983ddda..7c022cd61 100644 >>> --- a/lib/librte_sched/rte_sched.c >>> +++ b/lib/librte_sched/rte_sched.c >>> @@ -222,6 +222,7 @@ struct rte_sched_port { >>> uint64_t time_cpu_bytes; /* Current CPU time measured in bytes */ >>> uint64_t time; /* Current NIC TX time measured in bytes */ >>> struct rte_reciprocal inv_cycles_per_byte; /* CPU cycles per byte */ >>> + uint64_t cycles_per_byte; >>> >> >> I was backporting this patch to 18.11. The older ABI checker complains >> about this structure change. >> >> "cycles_per_byte has been added at the middle position of this >> structural type." >> >> Isn't this an ABI break? Dropping from 18.11 for time being. > > So it looks like rte_sched_port * is an opaque pointers as it's contents are only > known to rte_sched.c and not outside. To everyone else it is an opaque data structure, > so structural changes to it would not be an ABI breakage. > Thanks Ray, makes sense. I've included the fix in 18.11.10-rc1. Kevin. >> >>> /* Grinders */ >>> struct rte_mbuf **pkts_out; >>> @@ -852,6 +853,7 @@ rte_sched_port_config(struct rte_sched_port_params *params) >>> cycles_per_byte = (rte_get_tsc_hz() << RTE_SCHED_TIME_SHIFT) >>> / params->rate; >>> port->inv_cycles_per_byte = rte_reciprocal_value(cycles_per_byte); >>> + port->cycles_per_byte = cycles_per_byte; >>> >>> /* Grinders */ >>> port->pkts_out = NULL; >>> @@ -2673,16 +2675,21 @@ static inline void >>> rte_sched_port_time_resync(struct rte_sched_port *port) >>> { >>> uint64_t cycles = rte_get_tsc_cycles(); >>> - uint64_t cycles_diff = cycles - port->time_cpu_cycles; >>> + uint64_t cycles_diff; >>> uint64_t bytes_diff; >>> uint32_t i; >>> >>> + if (cycles < port->time_cpu_cycles) >>> + port->time_cpu_cycles = 0; >>> + >>> + cycles_diff = cycles - port->time_cpu_cycles; >>> /* Compute elapsed time in bytes */ >>> bytes_diff = rte_reciprocal_divide(cycles_diff << RTE_SCHED_TIME_SHIFT, >>> port->inv_cycles_per_byte); >>> >>> /* Advance port time */ >>> - port->time_cpu_cycles = cycles; >>> + port->time_cpu_cycles += >>> + (bytes_diff * port->cycles_per_byte) >> RTE_SCHED_TIME_SHIFT; >>> port->time_cpu_bytes += bytes_diff; >>> if (port->time < port->time_cpu_bytes) >>> port->time = port->time_cpu_bytes; >>> >> >