From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 65E8EA04AB for ; Sun, 30 Aug 2020 23:19:48 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B12932B93; Sun, 30 Aug 2020 23:19:47 +0200 (CEST) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60078.outbound.protection.outlook.com [40.107.6.78]) by dpdk.org (Postfix) with ESMTP id B2C64255 for ; Sun, 30 Aug 2020 23:19:45 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Tk/tNnylB0bQiLgEhGwuU4n4iQBEAgG33QPnovFqmmlGjuy08cjPih5HTBLeQJ4q+AmEd7rZNxe7jU2UuLrsLNRyPwnXr+XrVSOEQAfqOKLPPwJlkW4KTorPXatT6mU0LFAZXR5Ylrmmx+oGMIQqFzkO7g/+hNkF9dju5nBTBXr8ltZ8E9ORVMiMaHFUeabE0ipznqCzDLFAnPOBGbOU9qiBSTcg7X+PcrbnBpeW1UuOjt+X6aXHWjReLlTY4RcYA8hc+0yqxBapTycWChdk0I3U/qPp1tJtKKM8hrCBBs9RBr0imXmoqq8eyL7370LvaBSdmIVWW1RmXataXwzbgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z9k7ksbHdFsVJ5fO4QcAFCTWcNgax5sjLLkb3jwzQd4=; b=KRaPBhlhoNnM5g91IGz2au+MYqdesZNzQ0Gm8tsShGNlrLqNxTz2koDeUMbmufW0dp57H4UDYpPmuS3hedQn8jiGK065JU/Y9IG2rTjfrVB+thu1d5i06z7wqA0Zr641YiLej/Zm0fP6J0W39sF0h6tvgbffNJ0DMjgevGl7I0LT2tt7C06hfKqqPi6wrqzj8q4Zk/3iUnOcTgevaz47vhDq5WY8Kx80n4Z+3VhTYVLAwfBMnYrJHu5rDI+/IQW7R5r+T6lJd5VBUDKsZhRl0VA3/pIviuMoSQhD7nONyKQ7WvV0R8DJIVOuttLA0z4ZAtuVPM51jUWFxIMKPiB16A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=b-ulltech.com; dmarc=pass action=none header.from=b-ulltech.com; dkim=pass header.d=b-ulltech.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tacirler.onmicrosoft.com; s=selector2-tacirler-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z9k7ksbHdFsVJ5fO4QcAFCTWcNgax5sjLLkb3jwzQd4=; b=mvLs2S2IVmAhI2SCk3rWYZ1ePKubHOKESGhNqopl3Wy/aDQbiTOIEzRi8uP18T2OmMgu/nxRy6ttG1xUGstSYRG9FZ8loeHnVmGFIIFKC6SoMDHmiEsK3uV1qvC1OcBjFspQUUt3yQOpOBrBPTgbh/6ifqidKPgex90B9Bawr7M= Received: from PR2PR08MB4777.eurprd08.prod.outlook.com (2603:10a6:101:26::19) by PR3PR08MB5660.eurprd08.prod.outlook.com (2603:10a6:102:8d::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3326.21; Sun, 30 Aug 2020 21:19:44 +0000 Received: from PR2PR08MB4777.eurprd08.prod.outlook.com ([fe80::b040:ef1e:a2cb:32ee]) by PR2PR08MB4777.eurprd08.prod.outlook.com ([fe80::b040:ef1e:a2cb:32ee%6]) with mapi id 15.20.3326.025; Sun, 30 Aug 2020 21:19:44 +0000 From: =?windows-1254?Q?Volkan_Atl=FD?= To: "users@dpdk.org" CC: =?windows-1254?Q?Demir_Yava=FE?= Thread-Topic: rte_hash with rcu usage Thread-Index: AQHWfxHzrl/2u12yjkqjgj/6iConwg== Date: Sun, 30 Aug 2020 21:19:44 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=b-ulltech.com; x-originating-ip: [88.237.220.195] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f67113e5-f363-4c35-f41a-08d84d2a6a8f x-ms-traffictypediagnostic: PR3PR08MB5660: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: NwqSFUnblP87PIC9APECBAFLnk2e/tfzunS7JB2HZwp3WqOC+sr2Jk9CpoKtxYjY/1rEN+vrrnl5qy2AoEtL2Xi10N6x2LKEwKItt8rTjYsKywQtdjv2ZquUXH12itryTr1KvdQLaJJ9ZBi8wEMW1rIq+UoZGpkLti7Nwas2+td5wTz572nPm+jzrAAMEWJHPAJddvndHLjhg1pyaT2FFKF+E/32QxkIYgLPIv1oEzDp8KIhqQnzoed3u4SqWFpfImdsm5jrc61QTXoIXXrq/g4vRZe8R/DJIuxWXqXE2hLtjD5nvn70dzidDgxp+YU+v4h6dpJZ5wnZgygL3Ugsa8TSE0Og6q0zO6M4kUGB2xJVOavdTsF6KxOyvBulIBBJSPh4xhr/YPGEHeo/LJCcCQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PR2PR08MB4777.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(39830400003)(136003)(346002)(376002)(396003)(366004)(55016002)(66574015)(6916009)(7696005)(107886003)(9686003)(8936002)(91956017)(76116006)(66446008)(2906002)(66556008)(21615005)(26005)(186003)(66476007)(64756008)(19627405001)(86362001)(66946007)(71200400001)(5660300002)(52536014)(33656002)(6506007)(478600001)(166002)(316002)(966005)(8676002)(4326008)(83380400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: HYMUDpXkPSbqEKwalthYe3AmHKFAOPhgdvI6pzxysMlz5MpLl2bN5O+CpgRP5S9IFoUCtcMmG70HiXjryGE353Svy3vKDF7o8qYqwJbb8ad0XVTgCfFwPpHB+2wz3H/rQNVrRsld/r9r8c9YpANbGb9xm9I15Kj0JwutlQvk4dR2+wzTPkSu6To8o1BuP68cJynNDiY0sW9jeGNos72wbH4aGxcw4FsJLK8Ppq6KkT2ursyphcj9+NW/QBANrZ4foJgTqHvt5lZBw6+3xEZ7+1n+3Cji1zzxkG4bmDvcz6iZBiEyN+TKJtFM6+cFfFQTaftOu6++VrfKAtnJQSiNCltlHAHgWZartIkDKPaNJHBcjD18MAAdEnbE0nK8Kb5cn60g2qNes3bhUHYyh/itkeHIQorYBx8GXLLWOT2oc5M61XuN5v6XaIluh40fcByZFXrDe+ihzksTSfEPsKd4PEHMqZ+HJcsblYxOjVkkp21RkNnH/rpJRroj+GTubl4aMwQD3dh4LZAzmBLRMOcSRjR1FETEnFLi9OC1ruKgLPJO5FFiIvAyHqQ12p135cE2EPanqKOOpIt0ZnHcLwfZTOkAC2tKepqRiU2JB/xBw86z309i7RcQ2lxkmiZMF/j74xmI+DO8xK8MfRAEUpCkaw== MIME-Version: 1.0 X-OriginatorOrg: b-ulltech.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: PR2PR08MB4777.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f67113e5-f363-4c35-f41a-08d84d2a6a8f X-MS-Exchange-CrossTenant-originalarrivaltime: 30 Aug 2020 21:19:44.2890 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 42185048-44ea-4757-bc9a-05cae8373e98 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: lc028yPev4kN0+Qjf+yQABQy1qArx/+3EiZnCCYr3U+JM1UPwZUSUCAI/EwUnWpEyorn5uKT0o5LRGEL3hkFF7+5j33WuWvbstBxKGWEAMc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR3PR08MB5660 Content-Type: text/plain; charset="windows-1254" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] rte_hash with rcu usage X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi Since there is not any example codes of how to use DPDK RCU [1] on the inte= rnet, I wrote a simple application to understand it (paste below) I'm trying to track the end of grace period for each struct ctx including a= n RCU-QS variable. I run the application as two processes, called primary (= or controlplane) and secondary (or dataplane). Actually it works as expecte= d. The primary process can add, delete or list the ctx(es). The secondary p= rocess can list, and also process only one ctx object by taking RCU and the= n sleeping 10 seconds to simulate datapath. The problem here is that since DPDK RCU APIs directly take the RCU itself a= s a parameter, I first need to lookup the corresponding ctx object in both = process command in secondary process and del command in primary process. He= nce, this is nonsense. If the both processes return the same ctx object fro= m hash lookup at the same time (race condition), del command may free the c= tx object without waiting grace period. I can use locks here but what is th= e meaning of RCU this time? Should I use a RCU-QS for the whole hash? How c= an I use it for each hash object? What is the correct use? Thanks in advance - Volkan [1] https://doc.dpdk.org/guides/prog_guide/rcu_lib.html [https://doc.dpdk.org/guides/_images/rcu_general_info.svg] 6. RCU Library =97 Data Plane Development Kit 20.08.0 documentation - DPDK<= https://doc.dpdk.org/guides/prog_guide/rcu_lib.html> 6.4. How to use this library. The application must allocate memory and init= ialize a QS variable. Applications can call rte_rcu_qsbr_get_memsize() to c= alculate the size of memory to allocate. This API takes a maximum number of= reader threads, using this variable, as a parameter. doc.dpdk.org ---------------------------------------------------------------- #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #define DATAPLANE_LCORE_ID 3 #define DEV_ID 0 /* Check condition and return an error if true. */ #define rcu_qsbr_exit_if(cond, str, ...) do { \ if (cond) { \ printf("ERROR %s:%d: " str "\n", __FILE__, __LINE__, ##__VA_ARGS__)= ; \ exit(EXIT_FAILURE); \ } \ } while (0) struct ctx { uint32_t tid; uint32_t ms_addr; struct rte_rcu_qsbr *rcu; }; struct dev { struct rte_hash *h_ctx; }; #define TOTAL_CTX_ENTRY (1024 * 32) static struct rte_rcu_qsbr *init_rcu(void) { size_t sz =3D rte_rcu_qsbr_get_memsize(RTE_MAX_LCORE); struct rte_rcu_qsbr *rcu_qsbr =3D rte_zmalloc(NULL, sz, RTE_CACHE_LINE_= SIZE); rte_rcu_qsbr_init(rcu_qsbr, RTE_MAX_LCORE); return rcu_qsbr; } static int uninit_rcu(struct rte_rcu_qsbr *rcu) { rte_free(rcu); return 0; } static struct ctx *init_ctx(uint64_t tid, uint32_t ms_addr) { struct ctx *ctx =3D rte_malloc(NULL, sizeof(struct ctx), 0); rcu_qsbr_exit_if(ctx =3D=3D NULL, "No memory"); ctx->tid =3D tid; ctx->ms_addr =3D ms_addr; ctx->rcu =3D init_rcu(); return ctx; } static struct dev *init_dev(int hash_id) { struct dev *dev =3D rte_malloc(NULL, sizeof(struct dev), 0); rcu_qsbr_exit_if(dev =3D=3D NULL, "No memory"); char hash_name[8]; sprintf(hash_name, "dev%d", hash_id); struct rte_hash_parameters hash_params =3D { .entries =3D TOTAL_CTX_ENTRY, .key_len =3D sizeof(uint32_t), .hash_func_init_val =3D 0, .socket_id =3D rte_socket_id(), .hash_func =3D rte_jhash, .extra_flag =3D RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY_LF, .name =3D hash_name, }; dev->h_ctx =3D rte_hash_create(&hash_params); rcu_qsbr_exit_if(dev->h_ctx =3D=3D NULL, "Hash create failed"); return dev; } static void dataplane(int dev_id) { char hash_name[8]; sprintf(hash_name, "dev%d", dev_id); struct rte_hash *hash =3D rte_hash_find_existing(hash_name); rcu_qsbr_exit_if(hash =3D=3D NULL, "Hash not found, please check primar= y process"); char *p; char line[100]; do { printf("ulak_dp> "); fflush(stdout); p =3D fgets(line, sizeof(line) - 1, stdin); char command[8]; uint32_t tid; int istatus =3D sscanf(line, "%7s %u", command, &tid); if (istatus =3D=3D 1 && (strcmp(command, "list") =3D=3D 0)) { const void *next_key; void *next_data; uint32_t iter =3D 0; if (rte_hash_count(hash) =3D=3D 0) { printf("no entry exits\n"); continue; } /* Iterate through the hash table */ uint32_t line_n =3D 0; while (rte_hash_iterate(hash, &next_key, &next_data, &iter) >= =3D 0) { printf("#%u tid=3D%u ms_addr=3D%u ptr(ctx)=3D%p\n", line_n+= +, *(uint32_t *)next_key, ((struct ctx *)next_data)->ms_addr, next_data); } } else if (istatus =3D=3D 2 && (strcmp(command, "process") =3D=3D 0= )) { uint32_t *pdata; if (rte_hash_lookup_data(hash, &tid, (void **)&pdata) !=3D -ENO= ENT) { struct rte_rcu_qsbr *temp =3D ((struct ctx *) pdata)->rcu; rte_rcu_qsbr_thread_online(temp, DATAPLANE_LCORE_ID); /* This API is provided to aid debugging, it will do nothin= g. */ rte_rcu_qsbr_lock(temp, DATAPLANE_LCORE_ID); printf("processing begin (tid =3D %u)\n", tid); rte_delay_us_sleep(10000000); printf("processing end (tid =3D %u)\n", tid); /* This API is provided to aid debugging, it will do nothin= g. */ rte_rcu_qsbr_unlock(temp, DATAPLANE_LCORE_ID); rte_rcu_qsbr_thread_offline(temp, DATAPLANE_LCORE_ID); /* Update quiescent state counter */ rte_rcu_qsbr_quiescent(temp, DATAPLANE_LCORE_ID); } else { printf("no entry found (tid =3D %u)\n", tid); } } } while (p !=3D NULL); } static void controlplane(struct dev *dev) { struct rte_hash *hash =3D dev->h_ctx; char *p; char line[100]; do { printf("ulak_cp> "); fflush(stdout); p =3D fgets(line, sizeof(line) - 1, stdin); char command[8]; uint32_t tid, ms_addr; int istatus =3D sscanf(line, "%7s %u %u", command, &tid, &ms_addr); if (istatus =3D=3D 1 && (strcmp(command, "list") =3D=3D 0)) { const void *next_key; void *next_data; uint32_t iter =3D 0; if (rte_hash_count(hash) =3D=3D 0) { printf("no entry exits\n"); continue; } /* Iterate through the hash table */ uint32_t line_n =3D 0; while (rte_hash_iterate(hash, &next_key, &next_data, &iter) >= =3D 0) { printf("#%u tid=3D%u ms_addr=3D%u ptr(ctx)=3D%p\n", line_n+= +, *(uint32_t *)next_key, ((struct ctx *)next_data)->ms_addr, next_data); } } else if (istatus =3D=3D 2 && (strcmp(command, "del") =3D=3D 0)) { uint32_t *pdata; if (rte_hash_lookup_data(hash, &tid, (void **)&pdata) !=3D -ENO= ENT) { struct ctx *ctx =3D (struct ctx *)pdata; int32_t pos =3D rte_hash_del_key(hash, &tid); printf("grace period (ctx->tid =3D %u)\n", ctx->tid); /* Start the quiescent state query process */ uint64_t token =3D rte_rcu_qsbr_start(ctx->rcu); /* Check the quiescent state status */ rte_rcu_qsbr_check(ctx->rcu, token, true); rcu_qsbr_exit_if(rte_hash_free_key_with_position(hash, pos)= < 0, "Failed to free the key #%d\n", tid); rte_free(ctx->rcu); rte_free(ctx); printf("deleted #%d\n", tid); } else { printf("no entry found (tid =3D %u)\n", tid); } } else if (istatus =3D=3D 3 && (strcmp(command, "add") =3D=3D 0)) { struct ctx *ctx =3D init_ctx(tid, ms_addr); rte_rcu_qsbr_thread_register(ctx->rcu, DATAPLANE_LCORE_ID); uint32_t *pdata; if (rte_hash_lookup_data(dev->h_ctx, &tid, (void **)&pdata) =3D= =3D -ENOENT) { if (rte_hash_add_key_data(dev->h_ctx, &tid, (void *)((uintp= tr_t)ctx)) < 0) { printf("Hash key add failed\n"); } } else { printf("Hash key (tid =3D %u) already exist\n", tid); } } } while (p !=3D NULL); printf("done\n"); } int main(int argc, char **argv) { int ret =3D rte_eal_init(argc, argv); if (ret < 0) rte_exit(EXIT_FAILURE, "Invalid EAL arguments\n"); if (rte_eal_process_type() =3D=3D RTE_PROC_PRIMARY) { struct dev *dev1 =3D init_dev(DEV_ID); controlplane(dev1); } else { dataplane(DEV_ID); } printf("exiting\n"); return EXIT_SUCCESS; } - Volkan