Libtpa: a DPDK based userspace TCP stack implementation

Libtpa: a DPDK based userspace TCP stack implementation

[Yuanhan Liu]

Hi all,

I'd like to share a new DPDK open source project, libtpa(Transport
Protocol Acceleration)[0], which is just another userspace TCP stack
implementation so far, written from scratch.

I started this project 3 years ago, while I was searching for a feasible
open source project with no luck. There were indeed quite a few options,
but none of them actually met my needs. I then started writing one. Likely,
there are still other guys out there looking for a high performance and
stable userspace TCP stack. This is what this email and libtpa for.

Libtpa is fast. To demonstrate that, we did a hacky redis integration. The
benchmark shows that libtpa can boost the performance more than 5 times,
from 0.21m rps to 1.14m rps[1]. Right, it can achieve 1 million rps just
with one CPU thread. Meanwhile, the p99 latency decreases from 0.815ms
to 0.159ms.

Regarding the stableness, I'd say it's not bad, all kudos to the
comprehensive testing. I've written more than 200 tests. Together with
the testing arguments matrix[2], it can result in a big variety of test
cases. Therefore, most of the bugs are captured before deployment.

Having said that, I'd still suggest you to do as much testing as you can
if you want to use it, for libtpa is still under active development and
it's just v1.0-rc0 being released. Tons of changes have been made since
the last stable release.

There is one more thing I'm a bit proud of about libtpa: as a DPDK based
project, libtpa has rich set of debug tools[3]. The sock tracing is
particularly handy on debugging that libtpa doesn't ship a tcpdump like
tool, simply for we don't really need one.

The TCP part then may not sound that exciting. It's basically just an
initial TCP implementation, with standard congestion avoid algorithm
(New Reno). Libtpa implements slightly more than that though, such as
SACK, congestion window validation, spurious retransmission detection,
keepalive, etc.

That's all. Comments, questions, patches and testing are all welcome!

Thanks,
Yuanhan Liu

---
[0]: libtpa: https://github.com/bytedance/libtpa
[1]: redis: https://github.com/bytedance/libtpa/tree/main/doc/redis.rst
[2]: matrix shell: https://github.com/bytedance/libtpa/tree/main/doc/internals.rst
[3]: user guide: https://github.com/bytedance/libtpa/tree/main/doc/user_guide.rst

Re: Libtpa: a DPDK based userspace TCP stack implementation

[Liang Ma]

Hi Yuanhan,
   Congratulations!  
Regards
Liang

On Mon, Dec 11, 2023 at 05:56:54PM +0800, Yuanhan Liu wrote:
> Hi all,
> 
> I'd like to share a new DPDK open source project, libtpa(Transport
> Protocol Acceleration)[0], which is just another userspace TCP stack
> implementation so far, written from scratch.
> 
> I started this project 3 years ago, while I was searching for a feasible
> open source project with no luck. There were indeed quite a few options,
> but none of them actually met my needs. I then started writing one. Likely,
> there are still other guys out there looking for a high performance and
> stable userspace TCP stack. This is what this email and libtpa for.
> 
> Libtpa is fast. To demonstrate that, we did a hacky redis integration. The
> benchmark shows that libtpa can boost the performance more than 5 times,
> from 0.21m rps to 1.14m rps[1]. Right, it can achieve 1 million rps just
> with one CPU thread. Meanwhile, the p99 latency decreases from 0.815ms
> to 0.159ms.
> 
> Regarding the stableness, I'd say it's not bad, all kudos to the
> comprehensive testing. I've written more than 200 tests. Together with
> the testing arguments matrix[2], it can result in a big variety of test
> cases. Therefore, most of the bugs are captured before deployment.
> 
> Having said that, I'd still suggest you to do as much testing as you can
> if you want to use it, for libtpa is still under active development and
> it's just v1.0-rc0 being released. Tons of changes have been made since
> the last stable release.
> 
> There is one more thing I'm a bit proud of about libtpa: as a DPDK based
> project, libtpa has rich set of debug tools[3]. The sock tracing is
> particularly handy on debugging that libtpa doesn't ship a tcpdump like
> tool, simply for we don't really need one.
> 
> The TCP part then may not sound that exciting. It's basically just an
> initial TCP implementation, with standard congestion avoid algorithm
> (New Reno). Libtpa implements slightly more than that though, such as
> SACK, congestion window validation, spurious retransmission detection,
> keepalive, etc.
> 
> That's all. Comments, questions, patches and testing are all welcome!
> 
> Thanks,
> Yuanhan Liu
> 
> ---
> [0]: libtpa: https://github.com/bytedance/libtpa
> [1]: redis: https://github.com/bytedance/libtpa/tree/main/doc/redis.rst
> [2]: matrix shell: https://github.com/bytedance/libtpa/tree/main/doc/internals.rst
> [3]: user guide: https://github.com/bytedance/libtpa/tree/main/doc/user_guide.rst

RE: [EXT] Libtpa: a DPDK based userspace TCP stack implementation

[Jerin Jacob Kollanukkaran]

> -----Original Message-----
> From: Yuanhan Liu <yliu@fridaylinux.org>
> Sent: Monday, December 11, 2023 3:27 PM
> To: libtpa@googlegroups.com
> Cc: dev@dpdk.org; Yuanhan Liu <liuyuanhan.131@bytedance.com>
> Subject: [EXT] Libtpa: a DPDK based userspace TCP stack implementation
> 
> External Email
> 
> ----------------------------------------------------------------------
> Hi all,
> 
> I'd like to share a new DPDK open source project, libtpa(Transport Protocol
> Acceleration)[0], which is just another userspace TCP stack implementation so
> far, written from scratch.
> 
> I started this project 3 years ago, while I was searching for a feasible open
> source project with no luck. There were indeed quite a few options, but none of
> them actually met my needs. I then started writing one. Likely, there are still
> other guys out there looking for a high performance and stable userspace TCP
> stack. This is what this email and libtpa for.

Great Yuanhan.

If you have time and willing to put effort, I suggest make this part of dpdk code base
as new library (tcp or so) and leverage + improve another existing library such ip_frag.

I believe, that is only way.
- This code soon won't soon outdated based on new DPDK version
- More community review and contributors
- More review and features from NIC vendors PoV.
- More arch and driver support.
- More quality

Just my 2c.

Re: [EXT] Libtpa: a DPDK based userspace TCP stack implementation

[Thomas Monjalon]

11/12/2023 12:32, Jerin Jacob Kollanukkaran:
> From: Yuanhan Liu <yliu@fridaylinux.org>
> > Hi all,
> > 
> > I'd like to share a new DPDK open source project, libtpa(Transport Protocol
> > Acceleration)[0], which is just another userspace TCP stack implementation so
> > far, written from scratch.
> > 
> > I started this project 3 years ago, while I was searching for a feasible open
> > source project with no luck. There were indeed quite a few options, but none of
> > them actually met my needs. I then started writing one. Likely, there are still
> > other guys out there looking for a high performance and stable userspace TCP
> > stack. This is what this email and libtpa for.
> 
> Great Yuanhan.
> 
> If you have time and willing to put effort, I suggest make this part of dpdk code base
> as new library (tcp or so) and leverage + improve another existing library such ip_frag.
> 
> I believe, that is only way.
> - This code soon won't soon outdated based on new DPDK version
> - More community review and contributors
> - More review and features from NIC vendors PoV.
> - More arch and driver support.
> - More quality

As Yuanhan said, there are many TCP stacks running on top of DPDK.
We should add this one to the list:
	https://www.dpdk.org/ecosystem/#projects
Also a discussion has started recently about integrating one in DPDK.
As Jerin suggests, libtpa looks like a very good candidate to focus efforts on it.

Regarding performance, how does it compare with F-Stack? TLDK? Seastar?

Re: [EXT] Libtpa: a DPDK based userspace TCP stack implementation

[Yuanhan Liu]

On Mon, Dec 11, 2023 at 11:32:16AM +0000, Jerin Jacob Kollanukkaran wrote:
> 
> 
> > -----Original Message-----
> > From: Yuanhan Liu <yliu@fridaylinux.org>
> > Sent: Monday, December 11, 2023 3:27 PM
> > To: libtpa@googlegroups.com
> > Cc: dev@dpdk.org; Yuanhan Liu <liuyuanhan.131@bytedance.com>
> > Subject: [EXT] Libtpa: a DPDK based userspace TCP stack implementation
> > 
> > External Email
> > 
> > ----------------------------------------------------------------------
> > Hi all,
> > 
> > I'd like to share a new DPDK open source project, libtpa(Transport Protocol
> > Acceleration)[0], which is just another userspace TCP stack implementation so
> > far, written from scratch.
> > 
> > I started this project 3 years ago, while I was searching for a feasible open
> > source project with no luck. There were indeed quite a few options, but none of
> > them actually met my needs. I then started writing one. Likely, there are still
> > other guys out there looking for a high performance and stable userspace TCP
> > stack. This is what this email and libtpa for.
> 
> Great Yuanhan.
> 
> If you have time and willing to put effort, I suggest make this part of dpdk code base
> as new library (tcp or so) and leverage + improve another existing library such ip_frag.
> 
> I believe, that is only way.
> - This code soon won't soon outdated based on new DPDK version
> - More community review and contributors
> - More review and features from NIC vendors PoV.
> - More arch and driver support.
> - More quality

Hi Jerin,

Thanks for you suggestion and these really are good points!

Although libtpa is currently designed as a libray, I doubt it would suit
well as a new library to DPDK. Just taking the code base an example,
libtpa so far is about 27K lines of code. The TCP part is only about
3K lines of code. All the rest are codes supporting the TCP part, such
as sock tracing, mem file, mem file auto archive, etc. You can look
more from the internals page (or even read the code ;)

   https://github.com/bytedance/libtpa/blob/main/doc/internals.rst

Thanks,
Yuanhan Liu


> 
> Just my 2c.
> 
> -- 
> You received this message because you are subscribed to the Google Groups "libtpa" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to libtpa+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/libtpa/BY3PR18MB478592AF236C7BBDB0285E3CC88FA%40BY3PR18MB4785.namprd18.prod.outlook.com.
> For more options, visit https://groups.google.com/d/optout.

Re: [EXT] Libtpa: a DPDK based userspace TCP stack implementation

[Yuanhan Liu]

On Mon, Dec 11, 2023 at 01:16:52PM +0100, Thomas Monjalon wrote:
> 11/12/2023 12:32, Jerin Jacob Kollanukkaran:
> > From: Yuanhan Liu <yliu@fridaylinux.org>
> > > Hi all,
> > > 
> > > I'd like to share a new DPDK open source project, libtpa(Transport Protocol
> > > Acceleration)[0], which is just another userspace TCP stack implementation so
> > > far, written from scratch.
> > > 
> > > I started this project 3 years ago, while I was searching for a feasible open
> > > source project with no luck. There were indeed quite a few options, but none of
> > > them actually met my needs. I then started writing one. Likely, there are still
> > > other guys out there looking for a high performance and stable userspace TCP
> > > stack. This is what this email and libtpa for.
> > 
> > Great Yuanhan.
> > 
> > If you have time and willing to put effort, I suggest make this part of dpdk code base
> > as new library (tcp or so) and leverage + improve another existing library such ip_frag.
> > 
> > I believe, that is only way.
> > - This code soon won't soon outdated based on new DPDK version
> > - More community review and contributors
> > - More review and features from NIC vendors PoV.
> > - More arch and driver support.
> > - More quality
> 
> As Yuanhan said, there are many TCP stacks running on top of DPDK.
> We should add this one to the list:
> 	https://www.dpdk.org/ecosystem/#projects
> Also a discussion has started recently about integrating one in DPDK.
> As Jerin suggests, libtpa looks like a very good candidate to focus efforts on it.
> 
> Regarding performance, how does it compare with F-Stack? TLDK? Seastar?

I think it should be fair to say (I haven't done the testing though; I
never tried to run those stacks), libtpa is the userspace tcp stack with
the best performance I'm aware of. The redis numbers showed in this email
thread is just one example. Libtpa also ships an performance benchmark,
tperf. With tperf write test (and without jumboframe), libtpa can achieve
200G linerate with only one physical core for write. The read test is not
that good though, because of missing hardware acceleration features like
TSO.

Although performance is very important to an userspace stack, I still
want to point out that, during the design, performance is not my major
goal. I spent most of my effort on shaping the testing system and the
debug-ablity initially. Libtpa has been deployed in bytedance for more
than two years, till now, there is no single TCP protocol bug reported.
(I do get very few bugs reported though, but most of them are related
to the OS environment, such as sigbus due to wrong API used when running
out of tmpfs).

Thanks,
Yuanhan Liu

Re: [EXT] Libtpa: a DPDK based userspace TCP stack implementation

[Jerin Jacob]

On Mon, Dec 11, 2023 at 5:48 PM Yuanhan Liu <yliu@fridaylinux.org> wrote:
>
> On Mon, Dec 11, 2023 at 11:32:16AM +0000, Jerin Jacob Kollanukkaran wrote:
> >
> >
> > > -----Original Message-----
> > > From: Yuanhan Liu <yliu@fridaylinux.org>
> > > Sent: Monday, December 11, 2023 3:27 PM
> > > To: libtpa@googlegroups.com
> > > Cc: dev@dpdk.org; Yuanhan Liu <liuyuanhan.131@bytedance.com>
> > > Subject: [EXT] Libtpa: a DPDK based userspace TCP stack implementation
> > >
> > > External Email
> > >
> > > ----------------------------------------------------------------------
> > > Hi all,
> > >
> > > I'd like to share a new DPDK open source project, libtpa(Transport Protocol
> > > Acceleration)[0], which is just another userspace TCP stack implementation so
> > > far, written from scratch.
> > >
> > > I started this project 3 years ago, while I was searching for a feasible open
> > > source project with no luck. There were indeed quite a few options, but none of
> > > them actually met my needs. I then started writing one. Likely, there are still
> > > other guys out there looking for a high performance and stable userspace TCP
> > > stack. This is what this email and libtpa for.
> >
> > Great Yuanhan.
> >
> > If you have time and willing to put effort, I suggest make this part of dpdk code base
> > as new library (tcp or so) and leverage + improve another existing library such ip_frag.
> >
> > I believe, that is only way.
> > - This code soon won't soon outdated based on new DPDK version
> > - More community review and contributors
> > - More review and features from NIC vendors PoV.
> > - More arch and driver support.
> > - More quality
>
> Hi Jerin,
>
> Thanks for you suggestion and these really are good points!
>
> Although libtpa is currently designed as a libray, I doubt it would suit
> well as a new library to DPDK. Just taking the code base an example,

I think, number of line won't be a concern for upstreaming

> libtpa so far is about 27K lines of code. The TCP part is only about
> 3K lines of code. All the rest are codes supporting the TCP part, such
> as sock tracing, mem file, mem file auto archive, etc. You can look

I think, key piece would be split the code as reusable library(like
mem file)and leverage
existing libraries like eal trace.

DPDK standardized  the new library addition process without doing a
lot of throw away code.
See https://doc.dpdk.org/guides/contributing/new_library.html

> more from the internals page (or even read the code ;)
>
>    https://github.com/bytedance/libtpa/blob/main/doc/internals.rst
>
> Thanks,
> Yuanhan Liu
>
>
> >
> > Just my 2c.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "libtpa" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to libtpa+unsubscribe@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/libtpa/BY3PR18MB478592AF236C7BBDB0285E3CC88FA%40BY3PR18MB4785.namprd18.prod.outlook.com.
> > For more options, visit https://groups.google.com/d/optout.