From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 47EC741E1D for ; Wed, 8 Mar 2023 10:55:03 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A41941140; Wed, 8 Mar 2023 10:55:03 +0100 (CET) Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2087.outbound.protection.outlook.com [40.107.100.87]) by mails.dpdk.org (Postfix) with ESMTP id B19A540ED6; Wed, 8 Mar 2023 10:55:00 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZKS3NA+dBvAUPgfXBBNsSdu1MWbRpfzlGXBBhU8aoCws1XwcOlCpd25K00v+wLKo+rdMadLXIXRcK/kPRkEpwrMti6iuz0funKKA2JtlsMckr5lsJrqz9jNKaBM2Gf1V+gzhxixpCXeuFo0uJJInsCg6AI6NYZBS6fJmJmLrIfNA3wHN3S8GWakj2UrnHAKYteo7g1rvMDwWOp4gymg3s5q27SRCH6UVWkFbdQyA78L03mUUqkWtSko5THiEXpNBBiRCS8dR8ZK3SHCXd/nZdHiAppRyazNVi7NtqVy/1vGGQavXfHYFLKRN5rH4hUBRl5Y3gRZDS/erXH7McH2dgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j4V3nrnCwSBhgXJjb+gWO55t00n+t5RJI61I78Qmy6w=; b=RMTDl/RBPxNcqoc7kdiuSr7ZtWI1Lz+ifMIQY0I9QXdxONy1EMzuZKvEBLxijvjF3QI5wN0Sc9cFxfYVGk0ATiC456sMzYyzDyU8qUnox3B3kbkkl0LuTlDApx3nMQ+aQ71irdXC3XzQCxA78d9ruscuPoH4x+G1sXq2iqXi5IWTtGICgmUBQFjF6p3YMbPxyATFrzxgo8+U60Rjrn1NxJA+gOYvtk0D+9bBFwVzoMmlUdgJfxgr6nPNkCxZWN2X8bWYrFq+br4PGE5vDswlIPK3wKwxC3WDDiQNJUOa+15dkAkQJfYex4tsGaFWYQdrHghvNi7vDzbGCvJvhc4VVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j4V3nrnCwSBhgXJjb+gWO55t00n+t5RJI61I78Qmy6w=; b=ZDhstl68/1DQh2rd8IcV676RUhgrdthiie9qfjRgyUxSKxASogeyBEL0EXew1XMkS4Wt3TCaW6z0mq+ROtiqZp/wRuQ/JOhVAk5gFBd/1xXxDplzJ8+1Ysy9VuYqfRTR4RATD2ojFjtXq8WFhq1HY5jFKQOTDa9W/MnPB4oM1jg= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) by DS7PR12MB5768.namprd12.prod.outlook.com (2603:10b6:8:77::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6156.29; Wed, 8 Mar 2023 09:54:58 +0000 Received: from CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::dd5a:8a5c:f493:9640]) by CH2PR12MB4294.namprd12.prod.outlook.com ([fe80::dd5a:8a5c:f493:9640%4]) with mapi id 15.20.6156.029; Wed, 8 Mar 2023 09:54:58 +0000 Message-ID: <6acbe8d1-d030-3fd7-7c17-ed8958fec4bb@amd.com> Date: Wed, 8 Mar 2023 09:54:51 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: Re: [PATCH v3] app/testpmd: fix secondary process not forwarding Content-Language: en-US To: "lihuisong (C)" , "He, ShiyangX" , "dev@dpdk.org" Cc: "Zhou, YidingX" , "stable@dpdk.org" , "Zhang, Yuying" , "Singh, Aman Deep" , "Burakov, Anatoly" , Matan Azrad , Dmitry Kozlyuk References: <20221230075554.25244-1-shiyangx.he@intel.com> <20230223144106.707999-1-shiyangx.he@intel.com> <6b982bad-f9e6-62be-7a0d-30c7431889ad@amd.com> <34b0ccca-f846-2ecd-61e3-0531037545de@amd.com> <59e12a4b-529a-96d5-616a-65899e52eee7@huawei.com> From: Ferruh Yigit In-Reply-To: <59e12a4b-529a-96d5-616a-65899e52eee7@huawei.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-ClientProxiedBy: LO4P123CA0674.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:351::6) To CH2PR12MB4294.namprd12.prod.outlook.com (2603:10b6:610:a9::11) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR12MB4294:EE_|DS7PR12MB5768:EE_ X-MS-Office365-Filtering-Correlation-Id: e77a4641-1b8c-4f9c-f4a6-08db1fbb2d1f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /vUaJzLZo262unxjWfnb9REN4lBC3dueUSnlM9sayPIJNfGWeUAq0tXyps/Svo58ClALVzNsm6fOJNJagg/s6WVNRt3RKVuoRvvZtZI7nMZDQDY2Csa5pIToSoTIzX2WTElVsMnJsG/VE4RYL6LTAmTUo0YDYP096RutJKgqidLe6mwUXXvEv0Tdr/BlvJDjhKERfJ4VYR4Sde3N5FwXbMqgBh45PDsVEpZOcqw2yqeXCTiw4dT8YpkyqPJz0jlnF+RMq61ADnHNClSK8UixmQCAO/6RNlEex0WeX0DmLprs1BzfIRNcvVbkv19BbkPIDaLb54vHaepcl4xWSW1q1cpjKurNigXvSEpnDMSOdjDDMuiXBITnCahzrHae2P0hj3pZotNcFvpJl/BA2mT/tzTKCW3cAJjNY3fDeRQP2l9IEmxFJ+p5YZaOKLWdaXOuMV4M7ZxKkl0RLwPR4jEHdqhCpD5eFFjyxGd8CmF1gzXMaeqcvhucTWvPU6lA3mJ4IvZuTGC2cVS3EpooaCZCZMdlek54k/kLWlkcZvblEB8AyzYPEzv67/pb5iv6EVgo8/0NMnbV09JcPjoNBXiqjU+iY3I4UMgbIwLaWUldH24F+e8+xzyu75Khdl4wHejlzjWiKXJlWVBjlY/9W7druvg7HdZFnlahFpA2BQ5kxprBli0+mtMtL5itJPRkGE1cy0RTq1cXC2Z+toF0dBIG3A+T9FQ2Q2vJH4hWUY5ZXOw= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR12MB4294.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230025)(4636009)(39860400002)(136003)(376002)(396003)(366004)(346002)(451199018)(83380400001)(31686004)(2616005)(44832011)(2906002)(66946007)(5660300002)(4326008)(66556008)(66476007)(7416002)(26005)(186003)(8676002)(41300700001)(36756003)(53546011)(6666004)(38100700002)(8936002)(6486002)(86362001)(478600001)(6506007)(6512007)(54906003)(31696002)(110136005)(316002)(43740500002)(45980500001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TUJsazNSNHBaWjBBdWtUVzNOQkdncU5VQTZwd2tLaC84NlBvRWdRcURyZC9k?= =?utf-8?B?MDV2UStrV1hjQWk1QTJGYk5iYTdxTDVqei9wcTJtSmYvSDk5NnpaWGJPazVG?= =?utf-8?B?V2dtbUplU2h2RjFmRWtwWU52N0loUSs0M05MaWRwSUVrMkNkTitiWG1NdEds?= =?utf-8?B?Rlh2RmwybXlSaXp3ZW42S3llRjJ6cDdYenEvaGczeGcrNE9Jc1FtcmI2VXdG?= =?utf-8?B?a1ZCb0cxN1UyRVltRTB2UEhFQlpaOFo0R3I5MktmbCtoeUtDcGV4a2ViblFS?= =?utf-8?B?M1Z0NFIrMzJLZjUwYWFqU1U5VXRHeVJlMTJDMFlVYmFveVpDLzlqczEwcExO?= =?utf-8?B?aDNqbUEvSWJvbFRWRFBLLzdiVWFhb3N5T1FSZjliRW9UTzVqd3d0N1JUb054?= =?utf-8?B?RW5IR0c2S0ZFVGNPdWJ1U2g4TXNRODBTTE5ZMjlSYStKWC92b0F0dDM2YlBE?= =?utf-8?B?d28xRE16c2dxM0MwK0o4MW5uQmV6aS9wdVVoOC8wQ0g2cTFmMnpXSm5LS2Ex?= =?utf-8?B?T216WXJjcVp5UHMzaUFPWjVydTNOcG9jUEd6U1RiVWhRellJNkdadU9xRWxZ?= =?utf-8?B?RzJzQXB2M2RPLzUyMTVwVThlQkc3bFJvU0NRTUpoTS8rOFhnbU1ScTZ3Nktq?= =?utf-8?B?UXIzL1JVNVd5eDEzcUhwWHZzdDRXOTFQR05WYmZSc25UdmIvcDJXYjNpd1RH?= =?utf-8?B?K29pMUxySWkxOHI4UkNESTRDMFJGdStaZ1JHelg1MFVkTW9wZFVYNHp5WDNj?= =?utf-8?B?K2RBdWpFRDRjZE5ScWo0K1IrTHBJVnJxMVZ4bG91RTZNelMxUDFHMW5mTlBT?= =?utf-8?B?Ynlhdkg4MWtQS3dyWkU5VmwvVXc1YXpVTEVLSCthcGxmdXV1QThhM2kwaXoz?= =?utf-8?B?aUl5UXUrbkdxeFgrd04vRy80eFQwWTFzNWJub2tRMjdxM0MzdVp0Kzg5ZG9S?= =?utf-8?B?WVpJRXYvSFZtRWdpS3pZUVBIZFNkVUFiMlN3MGxoa3YzMWc5Y01DWTBoTkZ6?= =?utf-8?B?SDFTQkNXN2tzZGRRa2RBTEkwZHBoUFZJVEJXaEVlRDRnVlRPVUhlbkpGTjdQ?= =?utf-8?B?MVBLRnJWdDNVUGYrWmowNElaNXZERkd2TmV1S1I5YWxCSHdUNkNxTGRpTjNF?= =?utf-8?B?MXVBd1hyZm1pMnZBcThvTjVUMElhTjMxcUdKbVczTlhDUUxzcXJCU0JlSjFI?= =?utf-8?B?SCtRVVE0UVd2VVlZeVYrUUtjZkRzK1N5SVhVZXNqOXVQVVpFZHN0SGN4SElL?= =?utf-8?B?ZjNaTHdRaDJHQVVua042Mi92V1l0LzZRa1Z5TlhvN3lYVkIwcmQydmkyelAv?= =?utf-8?B?K1ZLZ2N6ODNTSlZVMkxYLzJZQm5mam40MTJ0OTMvQjhaREFIYkY1Kzl3RXgy?= =?utf-8?B?bjFmRlpka2xpMURXdVRKQUFkQ3pKMXNqZ0d6UXpQR1F6cFlKYUwxclVkZ1Vz?= =?utf-8?B?dFMzUUhBbGVUcFBYWngva0ZQNFE0ZHYrbXRvSVdpaVM0NStMa2QzckZPT2xv?= =?utf-8?B?bDRKNkJiMURsbmp1Qnd0RTdLNnBaNUhmbFRnYm5Hc0hLb0x1S0lHRU02Rkth?= =?utf-8?B?eklyNEZTcDkwdXN1VFVJcEx4eEV1UDJuRU5MRzJzQ0twNDY3bmxQRDhWeVVV?= =?utf-8?B?cFozaWdFQkwxYU15a0lqZDJHU2FZU3ltSjBUL2s4RlhTaVh1VkRSd01wRFZ2?= =?utf-8?B?TUR6dFdMbS9vZUM0clZHNEhRNjNaTnZta0N2Z25UUDJNTi9JRGhTeG41eW8z?= =?utf-8?B?VTBUaWVnUVBOYW5JWlZKU0N3akQyR1FVRkhxK2t5Q0pKRER6TStaQXRBcEY0?= =?utf-8?B?NGh6TGhCdEtaVWhJMkIrTlFqc2Q2SEtrUy9jWEhYTlJnUnVjNWw5ZFZjcnRT?= =?utf-8?B?YmFWR2VKSmhVQmJ1NDVEV2dhak95VzMvU2ZTalRrZEVIbEM0UEpuNEYwdWJG?= =?utf-8?B?d2R2bEtzaG1JTGhYQkZpNVU4ZE5jUG9WOHcrNG1xazZhV1FTNHFvVEFyUncy?= =?utf-8?B?aStXMTJBdmNUODI1TnZDenJheGU4V21NN3V1cjF2SmpnWlBBNDY4K0YwN2pM?= =?utf-8?B?Mkx4ajJmVVVGSkswRGc4cml6ZGo3aGVIQ2xSQVE0SFVzTHU0ckdGUERPWklB?= =?utf-8?Q?wnPNOQZ9QA8WGHBG10h7JFyfD?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: e77a4641-1b8c-4f9c-f4a6-08db1fbb2d1f X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB4294.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Mar 2023 09:54:58.3064 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ODgRt8MI0cfqHeMbAVzXu+a+JiIv95QHCxRIcdA2iodRp54vjkQ1satU4l4+o4Y5 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB5768 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org On 3/8/2023 2:54 AM, lihuisong (C) wrote: > > 在 2023/3/8 10:05, He, ShiyangX 写道: >> >>> -----Original Message----- >>> From: Ferruh Yigit >>> Sent: Tuesday, March 7, 2023 7:41 PM >>> To: He, ShiyangX ; dev@dpdk.org >>> Cc: Zhou, YidingX ; stable@dpdk.org; Zhang, >>> Yuying >>> ; Singh, Aman Deep >>> ; Burakov, Anatoly >>> ; Matan Azrad ; Dmitry >>> Kozlyuk >>> Subject: Re: [PATCH v3] app/testpmd: fix secondary process not >>> forwarding >>> >>> On 3/7/2023 3:25 AM, He, ShiyangX wrote: >>>> >>>>> -----Original Message----- >>>>> From: Ferruh Yigit >>>>> Sent: Monday, March 6, 2023 11:06 PM >>>>> To: He, ShiyangX ; dev@dpdk.org >>>>> Cc: Zhou, YidingX ; stable@dpdk.org; Zhang, >>>>> Yuying ; Singh, Aman Deep >>>>> ; Burakov, Anatoly >>>>> ; Matan Azrad ; Dmitry >>>>> Kozlyuk >>>>> Subject: Re: [PATCH v3] app/testpmd: fix secondary process not >>>>> forwarding >>>>> >>>>> On 2/23/2023 2:41 PM, Shiyang He wrote: >>>>>> Under multi-process scenario, the secondary process gets queue state >>>>>> from the wrong location (the global variable 'ports'). Therefore, >>>>>> the secondary process can not forward since "stream_init" is not >>>>>> called. >>>>>> >>>>>> This commit fixes the issue by calling 'rte_eth_rx/tx_queue_info_get' >>>>>> to get queue state from shared memory. >>>>>> >>>>>> Fixes: 3c4426db54fc ("app/testpmd: do not poll stopped queues") >>>>>> Cc: stable@dpdk.org >>>>>> >>>>>> Signed-off-by: Shiyang He >>>>>> Acked-by: Yuying Zhang >>>>>> >>>>>> v3: Add return value description >>>>>> --- >>>>>>   app/test-pmd/testpmd.c | 45 >>>>>> ++++++++++++++++++++++++++++++++++++++++-- >>>>>>   1 file changed, 43 insertions(+), 2 deletions(-) >>>>>> >>>>>> diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index >>>>>> 0c14325b8d..a050472aea 100644 >>>>>> --- a/app/test-pmd/testpmd.c >>>>>> +++ b/app/test-pmd/testpmd.c >>>>>> @@ -2418,9 +2418,50 @@ start_packet_forwarding(int with_tx_first) >>>>>>       if (!pkt_fwd_shared_rxq_check()) >>>>>>           return; >>>>>> >>>>>> -    if (stream_init != NULL) >>>>>> -        for (i = 0; i < cur_fwd_config.nb_fwd_streams; i++) >>>>>> +    if (stream_init != NULL) { >>>>>> +        for (i = 0; i < cur_fwd_config.nb_fwd_streams; i++) { >>>>>> +            if (rte_eal_process_type() == RTE_PROC_SECONDARY) >>>>> { >>>>>> +                struct fwd_stream *fs = fwd_streams[i]; >>>>>> +                struct rte_eth_rxq_info rx_qinfo; >>>>>> +                struct rte_eth_txq_info tx_qinfo; >>>>>> +                int32_t rc; >>>>>> +                rc = rte_eth_rx_queue_info_get(fs->rx_port, >>>>>> +                        fs->rx_queue, &rx_qinfo); >>>>>> +                if (rc == 0) { >>>>>> +                    ports[fs->rx_port].rxq[fs- >>>>>> rx_queue].state = >>>>>> +                        rx_qinfo.queue_state; >>>>>> +                } else if (rc == -ENOTSUP) { >>>>>> +                    /* Set the rxq state to >>>>> RTE_ETH_QUEUE_STATE_STARTED >>>>>> +                     * to ensure that the PMDs do not >>>>> implement >>>>>> +                     * rte_eth_rx_queue_info_get can >>>>> forward. >>>>>> +                     */ >>>>>> +                    ports[fs->rx_port].rxq[fs- >>>>>> rx_queue].state = >>>>>> + >>>>>     RTE_ETH_QUEUE_STATE_STARTED; >>>>>> +                } else { >>>>>> +                    TESTPMD_LOG(WARNING, >>>>>> +                        "Failed to get rx queue >>>>> info\n"); >>>>>> +                } >>>>>> + >>>>>> +                rc = rte_eth_tx_queue_info_get(fs->tx_port, >>>>>> +                        fs->tx_queue, &tx_qinfo); >>>>>> +                if (rc == 0) { >>>>>> +                    ports[fs->tx_port].txq[fs- >>>>>> tx_queue].state = >>>>>> +                        tx_qinfo.queue_state; >>>>>> +                } else if (rc == -ENOTSUP) { >>>>>> +                    /* Set the txq state to >>>>> RTE_ETH_QUEUE_STATE_STARTED >>>>>> +                     * to ensure that the PMDs do not >>>>> implement >>>>>> +                     * rte_eth_tx_queue_info_get can >>>>> forward. >>>>>> +                     */ >>>>>> +                    ports[fs->tx_port].txq[fs- >>>>>> tx_queue].state = >>>>>> + >>>>>     RTE_ETH_QUEUE_STATE_STARTED; >>>>>> +                } else { >>>>>> +                    TESTPMD_LOG(WARNING, >>>>>> +                        "Failed to get tx queue >>>>> info\n"); >>>>>> +                } >>>>>> +            } >>>>>>               stream_init(fwd_streams[i]); >>>>>> +        } >>>>>> +    } >>>>>> >>>>> >>>>> Testpmd duplicates some dpdk/ethdev state/config in application >>>>> level, and this can bite in multiple cases, as it is happening here. >>>>> >>>>> I am not sure if this was a design decision, but I think instead of >>>>> testpmd storing ethdev related state/config in application level, it >>>>> should store only application level state/config, and when ethdev >>>>> related state/config is required app should get it directly from >>>>> ethdev. >>>>> >>>>> It may be too late already for testpmd, there is a mixed usage, but I >>>>> am for preferring this approach when there is an opportunity. >>>>> >>>>> >>>>> >>>>> For above issue, why queue state needs to be stored in application >>>>> level >>> 'port' >>>>> variable? >>>>> Where is this queue state used? >>>>> >>>>> Can it work to get queue state directly from ethdev where this state >>>>> is used, instead of storing it in the 'port' variable in advance? >>>>> >>>>> And perhaps testpmd 'port' variable can be updated there, both for >>>>> primary and secondary, for backward compatibility (other existing >>>>> users of this queue state). >>>>> >>>>> What do you think? >>>> Thanks for your comments! >>>> >>>> It is an effective method to get queue state directly from ethdev >>>> where this >>> state is used. >>>> I also don't know the design meaning of the 'ports' variable. If >>>> modification is needed, a higher level of design and more work are >>>> required. >>>> >>>> As a bug fix, apart from extracting the code block into a function, >>>> is the >>> solution feasible? >>> >>> Hi Shiyang, >>> >>> As a bug fix, this issue (testpmd stored state not being up to date for >>> secondary process) looks like have potential to occur many different >>> flavors, >>> that is why what about having a central update? >>> >>> I think 'start_port()' can be a good place for this kind of update: >>> >>> start_port() { >>> >>>     ... >>>     if (secondary) >>>         update_state() >>> } >>> >>> update_state() { >>>     update_queue_state() >>> } >>> >>> update_queue_state() { >>>      >>> } >>> >>> >>> Having secondary checks and updates in multiple places can make code >>> harder >>> to understand. >>> >>> What do you think to update as above? >>> >>> >>> >> Thanks for your reply! >> It is more reasonable to update the queue state in 'start_port()'. >> I'll send a new patch asap. > It's also necessary to update the queue state when start forwarding. > Because the state may be changed after start port. I was hoping updating on a single point can be sufficient, is this needed because of testpmd commands? > The state cannot be updated in real time(because of no notification), > but it's helpful for secondary.