Re: [dpdk-dev] [RFC PATCH 0/5] graph: introduce graph subsystem

[Ray Kinsella <mdr@ashroe.eu>]

On 22/02/2020 10:24, Jerin Jacob wrote:
> On Sat, Feb 22, 2020 at 3:23 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>>
>> 22/02/2020 10:05, Jerin Jacob:
>>> On Fri, Feb 21, 2020 at 9:44 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>>>> 21/02/2020 16:56, Jerin Jacob:
>>>>> On Fri, Feb 21, 2020 at 4:40 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>>>>>> 21/02/2020 11:30, Jerin Jacob:
>>>>>>> On Mon, Feb 17, 2020 at 4:28 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>>>>>>>> On Mon, Feb 17, 2020 at 2:08 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>>>>>>>>> If we add rte_graph to DPDK, we will have 2 similar libraries.
>>>>>>>>>
>>>>>>>>> I already proposed several times to move rte_pipeline in a separate
>>>>>>>>> repository for two reasons:
>>>>>>>>>         1/ it is acting at a higher API layer level
>>>>>>>>
>>>>>>>> We need to define what is the higher layer API. Is it processing beyond L2?
>>>>>>
>>>>>> My opinion is that any API which is implemented differently
>>>>>> for different hardware should be in DPDK.
>>>>>
>>>>> We need to define SIMD optimization(not HW specific to  but
>>>>> architecture-specific)
>>>>> treatment as well, as the graph and node library will have SIMD
>>>>> optimization as well.
>>>>
>>>> I think SIMD optimization is generic to any performance-related project,
>>>> not specific to DPDK.
>>>>
>>>>
>>>>> In general, by the above policy enforced, we need to split DPDK like below,
>>>>> dpdk.git
>>>>> ----------
>>>>> librte_compressdev
>>>>> librte_bbdev
>>>>> librte_eventdev
>>>>> librte_pci
>>>>> librte_rawdev
>>>>> librte_eal
>>>>> librte_security
>>>>> librte_mempool
>>>>> librte_mbuf
>>>>> librte_cryptodev
>>>>> librte_ethdev
>>>>>
>>>>> other repo(s).
>>>>> ----------------
>>>>> librte_cmdline
>>>>> librte_cfgfile
>>>>> librte_bitratestats
>>>>> librte_efd
>>>>> librte_latencystats
>>>>> librte_kvargs
>>>>> librte_jobstats
>>>>> librte_gso
>>>>> librte_gro
>>>>> librte_flow_classify
>>>>> librte_pipeline
>>>>> librte_net
>>>>> librte_metrics
>>>>> librte_meter
>>>>> librte_member
>>>>> librte_table
>>>>> librte_stack
>>>>> librte_sched
>>>>> librte_rib
>>>>> librte_reorder
>>>>> librte_rcu
>>>>> librte_power
>>>>> librte_distributor
>>>>> librte_bpf
>>>>> librte_ip_frag
>>>>> librte_hash
>>>>> librte_fib
>>>>> librte_timer
>>>>> librte_telemetry
>>>>> librte_port
>>>>> librte_pdump
>>>>> librte_kni
>>>>> librte_acl
>>>>> librte_vhost
>>>>> librte_ring
>>>>> librte_lpm
>>>>> librte_ipsec
>>>>
>>>> I think it is a fair conclusion of the scope I am arguing, yes.
>>>
>>> OK. See below.
>>>
>>>>>> What is expected to be maintained, tested, etc.
>>>>>
>>>>> We need to maintain and test other code in OTHER dpdk repo as well.
>>>>
>>>> Yes but the ones responsible are not the same.
>>>
>>> I see your point. Can I interpret it as you would like to NOT take
>>> responsibility
>>> of  SW libraries(Items enumerated in the second list)?
>>
>> It's not only about me. This is a community decision.
> 
> OK. Let wait for community feedback.
> Probably we discuss more in public TB meeting in 26th Feb.
> 
>>
>>
>>> I think, the main question would be, how it will deliver to distros
>>> and/or end-users
>>> and what will be part of the dpdk release?
>>>
>>> I can think of two options. Maybe distro folks have better view on this.
>>>
>>> options 1:
>>> - Split dpdk to dpdk-core.git, dpdk-algo.git etc based on the
>>> functionalities and maintainer's availability.
>>> - Follow existing release cadence and deliver single release tarball
>>> with content from the above repos.
>>>
>>> options 2:
>>> - Introduce more subtrees(dpdk-next-algo.git etc) based on the
>>> functionalities and maintainer's availability.
>>> - Follow existing release cadence and have a pull request to main
>>> dpdk.git just like Linux kernel or existing scheme of things.
>>>
>>> I am for option 2.
>>>
>>> NOTE: This new graph and node library, I would like to make its new
>>> subtree in the existing scheme of
>>> things so that it will NOT be a burden for you to manage.
>>
>> The option 2 is to make maintainers life easier.
>> Keeping all libraries in the same repository allows to have
>> an unique release and a central place for the apps and docs.
>>
>> The option 1 may make contributors life easier if we consider
>> adding new libraries can make contributions harder in case of dependencies.
>> The option 1 makes also repositories smaller, so maybe easier to approach.
>> It makes easier to fully validate testing and quality of a repository.
>> Having separate packages makes easier to select what is distributed and supported.
> 
> If the final dpdk release tarball looks same for option1 and option2
> then I think,
> option 1 is overhead to manage intra repo dependency.
> 
> I agree with Thomas, it  is better to decide as a community what
> direction we need
> to take and align existing and new libraries with that scheme.
> 

+1 to Option 2.
As Jerin points out, it has allowed other larger communities to scale effectively.

> 
>>
>> After years thinking about the scope of DPDK repository,
>> I am still not sure which solution is best.
>> I really would like to see more opinions, thanks.
> 
> Yes.
> 
>>
>>