From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from demumfd001.nsn-inter.net (demumfd001.nsn-inter.net [93.183.12.32]) by dpdk.org (Postfix) with ESMTP id 36D60DE0 for ; Tue, 2 Dec 2014 10:55:56 +0100 (CET) Received: from demuprx016.emea.nsn-intra.net ([10.150.129.55]) by demumfd001.nsn-inter.net (8.14.3/8.14.3) with ESMTP id sB29tsWw027467 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Tue, 2 Dec 2014 09:55:55 GMT Received: from SGSIHTC002.nsn-intra.net ([10.159.225.19]) by demuprx016.emea.nsn-intra.net (8.12.11.20060308/8.12.11) with ESMTP id sB29tXkC026030 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Tue, 2 Dec 2014 10:55:54 +0100 Received: from SGSIMBX004.nsn-intra.net ([169.254.4.161]) by SGSIHTC002.nsn-intra.net ([10.159.225.19]) with mapi id 14.03.0195.001; Tue, 2 Dec 2014 17:55:29 +0800 From: "Chi, Xiaobo (NSN - CN/Hangzhou)" To: "Chi, Xiaobo (NSN - CN/Hangzhou)" , "dev@dpdk.org" Thread-Topic: [PATCH] add one option memory-only for those secondary PRBs Thread-Index: AQHQDhQcLwXdX9mh20CnxuPUpRKT8Jx8DF1Q Date: Tue, 2 Dec 2014 09:55:28 +0000 Message-ID: References: <1417513268-6040-1-git-send-email-xiaobo.chi@nsn.com> In-Reply-To: <1417513268-6040-1-git-send-email-xiaobo.chi@nsn.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.159.225.116] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-purgate-type: clean X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de X-purgate: clean X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate-size: 11921 X-purgate-ID: 151667::1417514155-0000658F-879712D6/0/0 Subject: Re: [dpdk-dev] [PATCH] add one option memory-only for those secondary PRBs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Dec 2014 09:55:56 -0000 Hi,=20 The similar functionality of this patch has been sent out for review on Nov= .12, email topic is "[dpdk-dev] one lightwight rte_eal_init() for SECONDARY= processes which only use sharedmemory". This time, I just change the impl= ementation method according to comments from Bruce Richardson.=20 Background: What we are doing now is port make telecom network element to be cloud base= d. For one of our product, DPDK is applied not only for fastpath/dataplan= e processing, but also for Distributed Message eXchange (DMX) between diffe= rent processes/applications which may located in different VM even differen= t host. for such a DMX system, in one VM, we have one DPDK based dmxdemo (= which acts as the PRIMARY) which is in charge of distribute message between= different applications, and dozens of applications (act as SECONDARY) to u= se DPDK based rte_tx_ring/rte_rx_ring/mempool/memzone to send receive messa= ges to dmxdemo. Problem: Here, these DPDK based SECONDARY processes need only the DPDK's hugepage ba= sed sharememory mechanism and it's upper libs (such as ring, mempool, etc.)= , they need not cpu core pinning, iopl privilege changing , pci device, tim= er, alarm, interrupt, shared_driver_list, core_info, threads for each core= , etc. Then, for such kind of SECONDARY processes, the current rte_eal_init= () is too heavy. =20 I have seen some others also met similar troubles.=20 Solution: One new EAL initializing argument, --memory-only, is added. It is only for = those SECONDARY processes which only want to share memory with primary proc= ess. When this argument is defined, users need not define those madentory a= rguments, such as "-c xx -n xx", due to we don't want to pin such kind of p= rocesses to any CPUs. I have tested on my test env, it works. Please kindly help to review and gi= ve comments. Thanks. -----Original Message----- From: chixiaobo [mailto:xiaobo.chi@nsn.com]=20 Sent: Tuesday, December 02, 2014 5:41 PM To: dev@dpdk.org Cc: Chi, Xiaobo (NSN - CN/Hangzhou) Subject: [PATCH] add one option memory-only for those secondary PRBs --- lib/librte_eal/common/eal_common_options.c | 18 +++- lib/librte_eal/common/eal_internal_cfg.h | 1 + lib/librte_eal/common/eal_options.h | 4 +- lib/librte_eal/linuxapp/eal/eal.c | 137 +++++++++++++++----------= ---- 4 files changed, 89 insertions(+), 71 deletions(-) mode change 100644 =3D> 100755 lib/librte_eal/common/eal_common_options.c mode change 100644 =3D> 100755 lib/librte_eal/common/eal_internal_cfg.h mode change 100644 =3D> 100755 lib/librte_eal/common/eal_options.h mode change 100644 =3D> 100755 lib/librte_eal/linuxapp/eal/eal.c diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/co= mmon/eal_common_options.c old mode 100644 new mode 100755 index e2810ab..5ab4b87 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c @@ -85,6 +85,7 @@ eal_long_options[] =3D { {OPT_XEN_DOM0, 0, 0, OPT_XEN_DOM0_NUM}, {OPT_CREATE_UIO_DEV, 1, NULL, OPT_CREATE_UIO_DEV_NUM}, {OPT_VFIO_INTR, 1, NULL, OPT_VFIO_INTR_NUM}, + {OPT_MEMORY_ONLY, 0, NULL, OPT_MEMORY_ONLY_NUM}, {0, 0, 0, 0} }; =20 @@ -126,6 +127,7 @@ eal_reset_internal_config(struct internal_config *inter= nal_cfg) internal_cfg->no_hpet =3D 1; #endif internal_cfg->vmware_tsc_map =3D 0; + internal_cfg->memory_only=3D 0; } =20 /* @@ -454,6 +456,10 @@ eal_parse_common_option(int opt, const char *optarg, conf->process_type =3D eal_parse_proc_type(optarg); break; =20 + case OPT_MEMORY_ONLY_NUM: + conf->memory_only=3D 1; + break; + case OPT_MASTER_LCORE_NUM: if (eal_parse_master_lcore(optarg) < 0) { RTE_LOG(ERR, EAL, "invalid parameter for --" @@ -525,9 +531,9 @@ eal_check_common_options(struct internal_config *intern= al_cfg) { struct rte_config *cfg =3D rte_eal_get_configuration(); =20 - if (!lcores_parsed) { - RTE_LOG(ERR, EAL, "CPU cores must be enabled with options " - "-c or -l\n"); + if (!lcores_parsed && !(internal_cfg->process_type =3D=3D RTE_PROC_SECOND= ARY&& internal_cfg->memory_only) ) { + RTE_LOG(ERR, EAL, "For those processes without memory-only option, CPU c= ores " + "must be enabled with options -c or -l\n"); return -1; } if (cfg->lcore_role[cfg->master_lcore] !=3D ROLE_RTE) { @@ -545,6 +551,11 @@ eal_check_common_options(struct internal_config *inter= nal_cfg) "specified\n"); return -1; } + if ( internal_cfg->process_type !=3D RTE_PROC_SECONDARY && + internal_cfg->memory_only) { + RTE_LOG(ERR, EAL, "only secondary processes can specify memory-only opti= on.\n"); + return -1; + } if (index(internal_cfg->hugefile_prefix, '%') !=3D NULL) { RTE_LOG(ERR, EAL, "Invalid char, '%%', in --"OPT_FILE_PREFIX" " "option\n"); @@ -590,6 +601,7 @@ eal_common_usage(void) " --"OPT_SYSLOG" : set syslog facility\n" " --"OPT_LOG_LEVEL" : set default log level\n" " --"OPT_PROC_TYPE" : type of this process\n" + " --"OPT_MEMORY_ONLY": only use shared memory, valid only for = secondary process." " --"OPT_PCI_BLACKLIST", -b: add a PCI device in black list.\n" " Prevent EAL from using this PCI device. The argume= nt\n" " format is .\n" diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/comm= on/eal_internal_cfg.h old mode 100644 new mode 100755 index aac6abf..68b982c --- a/lib/librte_eal/common/eal_internal_cfg.h +++ b/lib/librte_eal/common/eal_internal_cfg.h @@ -85,6 +85,7 @@ struct internal_config { =20 unsigned num_hugepage_sizes; /**< how many sizes on this system */ struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES]; + volatile unsigned memory_only; /**name); - solib->lib_handle =3D dlopen(solib->name, RTLD_NOW); - if (solib->lib_handle =3D=3D NULL) - RTE_LOG(WARNING, EAL, "%s\n", dlerror()); - } - - eal_thread_init_master(rte_config.master_lcore); - - RTE_LOG(DEBUG, EAL, "Master core %u is ready (tid=3D%x)\n", - rte_config.master_lcore, (int)thread_id); - - if (rte_eal_dev_init() < 0) - rte_panic("Cannot init pmd devices\n"); - - RTE_LCORE_FOREACH_SLAVE(i) { - - /* - * create communication pipes between master thread - * and children - */ - if (pipe(lcore_config[i].pipe_master2slave) < 0) - rte_panic("Cannot create pipe\n"); - if (pipe(lcore_config[i].pipe_slave2master) < 0) - rte_panic("Cannot create pipe\n"); - - lcore_config[i].state =3D WAIT; - - /* create a thread for each lcore */ - ret =3D pthread_create(&lcore_config[i].thread_id, NULL, - eal_thread_loop, NULL); - if (ret !=3D 0) - rte_panic("Cannot create thread\n"); - } - - /* - * Launch a dummy function on all slave lcores, so that master lcore - * knows they are all ready when this function returns. - */ - rte_eal_mp_remote_launch(sync_func, NULL, SKIP_MASTER); - rte_eal_mp_wait_lcore(); - - /* Probe & Initialize PCI devices */ - if (rte_eal_pci_probe()) - rte_panic("Cannot probe PCI\n"); - + if( !internal_config.memory_only ){ +#ifdef RTE_LIBRTE_IVSHMEM + if (rte_eal_ivshmem_obj_init() < 0) + rte_panic("Cannot init IVSHMEM objects\n"); +#endif =20 + if (rte_eal_alarm_init() < 0) + rte_panic("Cannot init interrupt-handling thread\n"); + + if (rte_eal_intr_init() < 0) + rte_panic("Cannot init interrupt-handling thread\n"); + + if (rte_eal_timer_init() < 0) + rte_panic("Cannot init HPET or TSC timers\n"); + + eal_check_mem_on_local_socket(); + + rte_eal_mcfg_complete(); + + TAILQ_FOREACH(solib, &solib_list, next) { + RTE_LOG(INFO, EAL, "open shared lib %s\n", solib->name); + solib->lib_handle =3D dlopen(solib->name, RTLD_NOW); + if (solib->lib_handle =3D=3D NULL) + RTE_LOG(WARNING, EAL, "%s\n", dlerror()); + } + + eal_thread_init_master(rte_config.master_lcore); + + RTE_LOG(DEBUG, EAL, "Master core %u is ready (tid=3D%x)\n", + rte_config.master_lcore, (int)thread_id); + + if (rte_eal_dev_init() < 0) + rte_panic("Cannot init pmd devices\n"); + + RTE_LCORE_FOREACH_SLAVE(i) { + + /* + * create communication pipes between master thread + * and children + */ + if (pipe(lcore_config[i].pipe_master2slave) < 0) + rte_panic("Cannot create pipe\n"); + if (pipe(lcore_config[i].pipe_slave2master) < 0) + rte_panic("Cannot create pipe\n"); + + lcore_config[i].state =3D WAIT; + + /* create a thread for each lcore */ + ret =3D pthread_create(&lcore_config[i].thread_id, NULL, + eal_thread_loop, NULL); + if (ret !=3D 0) + rte_panic("Cannot create thread\n"); + } + + /* + * Launch a dummy function on all slave lcores, so that master lc= ore + * knows they are all ready when this function returns. + */ + rte_eal_mp_remote_launch(sync_func, NULL, SKIP_MASTER); + rte_eal_mp_wait_lcore(); + + /* Probe & Initialize PCI devices */ + if (rte_eal_pci_probe()) + rte_panic("Cannot probe PCI\n"); + } return fctret; } =20 @@ -859,3 +861,4 @@ int rte_eal_has_hugepages(void) { return ! internal_config.no_hugetlbfs; } + --=20 1.9.4.msysgit.2