# orderedItems[].object.name
Extending the Vector Packet Processing Engine

# orderedItems[].object.summary
I've been building core networking components to leverage VPP more fully as a branch router. Here is an overview of that work.

# orderedItems[].object.content
Within the last couple of years, I spent some time working with a small regional ISP as CTO. We were focused on building out FTTP for our local community both to rural dwellings and to multi-dwelling units (MDUs).

One of the challenges that we had as a bootstrapped technology company was the cost of enterprise networking equipment. The founder of the company made the strategic decision to focus on delivering active Ethernet to our subscribers (no GPON). This allowed us to stick to the type of networking solutions that might be deployed in an enterprise campus rather than the proprietary solutions that drive more common FTTP deployments.

We also wanted more direct control over our network fabric. That led us to explore open networking solutions like SONiC and #VPP. VPP in particular found a home in our MDU deployments, allowing us to build very capable routing platforms using #OSS and no proprietary ASICs or other traditionally pipelined solutions.

There are some challenges with VPP, however. The system has a number of plugins (e.g., nat44-ed, ct6, nat64, acl-plugin, etc.) that were useful for our deployments. But some plugins interacted poorly with others (e.g., routing through an interface and using NAT together doesn't work well, using NAT64 and anything else doesn't work well, etc.) I came up with some tortured solutions to those problems that involved ACL-based forwarding, memif interfaces, and multiple VPP processes. But they weren't ideal.

Within the last month, I decided to take a stab at solving a problem in my home lab with ct6: ostensibly designed for #IPv6 connection tracking, it really didn't do much of anything. It's mostly a stub for future work that has never been completed. I decided to complete it and submitted a merge request upstream.

But in that process, I realized that I could do a lot more. Rather than just patch ct6, I could build a plugin from whole cloth that handled all of my translation needs from a common state table. And I could perform access control decisions with the state maintained as part of that capability. That became: https://github.com/justindthomas/sfw:

sfw zone internal interface lan
sfw zone external interface wan
sfw policy outbound from-zone internal to-zone external
sfw policy outbound default-action permit-stateful-nat
sfw policy inbound from-zone external to-zone internal
sfw policy inbound default-action deny
sfw nat64 pool add x.x.x.x/32 prefix 64:ff9b::/96

Along the way, I realized that I could take things further. I had been using Free Range Router (FRR) to handle things like BGP and OSPF. But it integrates with VPP via a shim called the Linux Control Plane. As the name implies, that only works on Linux, and I want to be flexible enough to support FreeBSD as well.

So I wrote https://github.com/justindthomas/vcl-rs to allow for Rust daemons to bind into VPP directly without going through the host stack. Then I created: ribd, bgpd, ospfd, and dnsd.

These daemons all have common characteristics; they each read a dedicated section from a YAML file (i.e., one file can be used to control all of them), they're designed to reload configuration on a SIGHUP, and they avoid using the host networking stack in favor of either vcl-rs or punt sockets (e.g., for OSPF).

That last daemon (dnsd) has me excited today. I'm very interested in facilitating the transition to IPv6. So things like #464XLAT, #NAT64, and #DNS64 are in my sights. But support for each of those protocols is challenging; VPP does not support router advertisements for #PREF64 (for NAT64) or for #RDNSS (to communicate IPv6 DNS information to clients).

To deal with that, I added in a couple of patches as part of sfw that hook into VPP's existing router advertisement functions. That expands the advertisements to include PREF64 and RDNSS:

00:00:00.000000 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 104)
    fe80::dcad:ff:fe00:64 > ff02::1: [icmp6 sum ok] ICMP6, router
    advertisement, length 104
  hop limit 64, Flags [managed], pref medium, router lifetime 600s,
      reachable time 0ms, retrans timer 0ms
    source link-address option (1), length 8 (1): de:ad:00:00:00:64
      0x0000: dead 0000 0064
    mtu option (5), length 8 (1):&nbsp; 1500
      0x0000: 0000 0000 05dc
    prefix info option (3), length 32 (4): xxxx:xxxx:10::/64,
          Flags [onlink, auto], valid time 2584092s, pref. time 596892s
      0x0000: 40c0 0027 6e1c 0009 1b9c 0000 0000 xxxx
      0x0010: xxxx 0010 0000 0000 0000 0000 0000
    unknown option (38), length 16 (2):
      0x0000: 0708 0064 ff9b 0000 0000 0000 0000
    rdnss option (25), length 24 (3):&nbsp; lifetime 90s,
          addr: xxxx:xxxx:10:0:ffff:ffff:ffff:fffe
      0x0000: 0000 0000 005a xxxx xxxx 0010 0000 ffff
      0x0010: ffff ffff fffe

This has enabled me to have a client network that is purely IPv6. My Mac clients see the router advertisements and enable their clat46 functionality:

en0: flags=88e3<UP,BROADCAST,SMART,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1500
    options=6460<TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
    ether 16:59:1d:c2:0d:c7
    inet6 fe80::1841:489:85ff:916f%en0 prefixlen 64 secured scopeid 0xb
    inet6 xxxx:xxxx:10:0:xxxx:xxxx:xxxx:xxxx prefixlen 64 autoconf secured
    inet6 xxxx:xxxx:10:0:xxxx:xxxx:xxxx:xxxx prefixlen 64 autoconf temporary
    inet 192.0.0.2 netmask 0xffffffff broadcast 192.0.0.2
    inet6 xxxx:xxxx:10:0:xxxx:xxxx:xxxx:xxxx prefixlen 64 clat46
    nat64 prefix 64:ff9b:: prefixlen 96
    nd6 options=201<PERFORMNUD,DAD>
    media: autoselect
    status: active

Interactions with IPv4 servers is automatically translated into 64:ff9b::/96 addresses and passed along to the servers over the router's IPv4 stack.

These daemons - even individually - constitute a formidable code base. And obviously I used an LLM to help create the tens of thousands of lines of code in total. The effort has been informed heavily by RFCs, industry-standard conformance testing, and integration testing against other routing platforms. And for some (like dnsd) a lot of iterations just to get performance to a solid level (I think DNS is more complex than we give it credit for - especially when DNSSEC is considered). And the fact that I'm avoiding the host networking stack complicates matters, because I have to handle the translation between Rust code and the C code that exposes the VCL capabilities in VPP.

I'm using the end product daily (pulled together in a larger router framework that I haven't released yet) and tweaking as issues reveal themselves. If you'd like to participate, please feel free to drop me a line.

# orderedItems[].object.source.content
Within the last couple of years, I spent some time working with a small regional ISP as CTO. We were focused on building out FTTP for our local community both to rural dwellings and to multi-dwelling units (MDUs).

One of the challenges that we had as a bootstrapped technology company was the cost of enterprise networking equipment. The founder of the company made the strategic decision to focus on delivering active Ethernet to our subscribers (no GPON). This allowed us to stick to the type of networking solutions that might be deployed in an enterprise campus rather than the proprietary solutions that drive more common FTTP deployments.

We also wanted more direct control over our network fabric. That led us to explore open networking solutions like SONiC and #VPP. VPP in particular found a home in our MDU deployments, allowing us to build very capable routing platforms using #OSS and no proprietary ASICs or other traditionally pipelined solutions.

There are some challenges with VPP, however. The system has a number of plugins (e.g., `nat44-ed`, `ct6`, `nat64`, `acl-plugin`, etc.) that were useful for our deployments. But some plugins interacted poorly with others (e.g., routing through an interface and using NAT together doesn't work well, using NAT64 and anything else doesn't work well, etc.) I came up with some tortured solutions to those problems that involved ACL-based forwarding, `memif` interfaces, and multiple VPP processes. But they weren't ideal.

Within the last month, I decided to take a stab at solving a problem in my home lab with `ct6`: ostensibly designed for #IPv6 connection tracking, it really didn't do much of anything. It's mostly a stub for future work that has never been completed. I decided to complete it and submitted a merge request upstream.

But in that process, I realized that I could do a lot more. Rather than just patch `ct6`, I could build a plugin from whole cloth that handled all of my translation needs from a common state table. And I could perform access control decisions with the state maintained as part of that capability. That became: [https://github.com/justindthomas/sfw](https://github.com/justindthomas/sfw):

```
sfw zone internal interface lan
sfw zone external interface wan
sfw policy outbound from-zone internal to-zone external
sfw policy outbound default-action permit-stateful-nat
sfw policy inbound from-zone external to-zone internal
sfw policy inbound default-action deny
sfw nat64 pool add x.x.x.x/32 prefix 64:ff9b::/96
```

Along the way, I realized that I could take things further. I had been using Free Range Router (FRR) to handle things like BGP and OSPF. But it integrates with VPP via a shim called the Linux Control Plane. As the name implies, that only works on Linux, and I want to be flexible enough to support FreeBSD as well.

So I wrote [https://github.com/justindthomas/vcl-rs](https://github.com/justindthomas/vcl-rs) to allow for Rust daemons to bind into VPP directly without going through the host stack. Then I created: [ribd](https://github.com/justindthomas/ribd), [bgpd](https://github.com/justindthomas/bgpd), [ospfd](https://github.com/justindthomas/ospfd), and [dnsd](https://github.com/justindthomas/dnsd).

These daemons all have common characteristics; they each read a dedicated section from a `YAML` file (i.e., one file can be used to control all of them), they're designed to reload configuration on a `SIGHUP`, and they avoid using the host networking stack in favor of either `vcl-rs` or `punt` sockets (e.g., for OSPF).

That last daemon (dnsd) has me excited today. I'm very interested in facilitating the transition to IPv6. So things like #464XLAT, #NAT64, and #DNS64 are in my sights. But support for each of those protocols is challenging; VPP does not support router advertisements for #PREF64 (for NAT64) or for #RDNSS (to communicate IPv6 DNS information to clients).

To deal with that, I added in a couple of patches as part of `sfw` that hook into VPP's existing router advertisement functions. That expands the advertisements to include `PREF64` and `RDNSS`:

```
00:00:00.000000 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 104) 
    fe80::dcad:ff:fe00:64 > ff02::1: [icmp6 sum ok] ICMP6, router 
    advertisement, length 104 
  hop limit 64, Flags [managed], pref medium, router lifetime 600s, 
      reachable time 0ms, retrans timer 0ms
    source link-address option (1), length 8 (1): de:ad:00:00:00:64
      0x0000: dead 0000 0064
    mtu option (5), length 8 (1):  1500
      0x0000: 0000 0000 05dc
    prefix info option (3), length 32 (4): xxxx:xxxx:10::/64, 
          Flags [onlink, auto], valid time 2584092s, pref. time 596892s
      0x0000: 40c0 0027 6e1c 0009 1b9c 0000 0000 xxxx
      0x0010: xxxx 0010 0000 0000 0000 0000 0000
    unknown option (38), length 16 (2):
      0x0000: 0708 0064 ff9b 0000 0000 0000 0000
    rdnss option (25), length 24 (3):  lifetime 90s,
          addr: xxxx:xxxx:10:0:ffff:ffff:ffff:fffe
      0x0000: 0000 0000 005a xxxx xxxx 0010 0000 ffff
      0x0010: ffff ffff fffe
```
This has enabled me to have a client network that is purely IPv6. My Mac clients see the router advertisements and enable their `clat46` functionality:

```
en0: flags=88e3<UP,BROADCAST,SMART,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1500
    options=6460<TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
    ether 16:59:1d:c2:0d:c7
    inet6 fe80::1841:489:85ff:916f%en0 prefixlen 64 secured scopeid 0xb
    inet6 xxxx:xxxx:10:0:xxxx:xxxx:xxxx:xxxx prefixlen 64 autoconf secured
    inet6 xxxx:xxxx:10:0:xxxx:xxxx:xxxx:xxxx prefixlen 64 autoconf temporary
    inet 192.0.0.2 netmask 0xffffffff broadcast 192.0.0.2
    inet6 xxxx:xxxx:10:0:xxxx:xxxx:xxxx:xxxx prefixlen 64 clat46
    nat64 prefix 64:ff9b:: prefixlen 96
    nd6 options=201<PERFORMNUD,DAD>
    media: autoselect
    status: active
```
Interactions with IPv4 servers is automatically translated into `64:ff9b::/96` addresses and passed along to the servers over the router's IPv4 stack.

These daemons - even individually - constitute a formidable code base. And obviously I used an LLM to help create the tens of thousands of lines of code in total. The effort has been informed heavily by RFCs, industry-standard conformance testing, and integration testing against other routing platforms. And for some (like `dnsd`) a lot of iterations just to get performance to a solid level (I think DNS is more complex than we give it credit for - especially when DNSSEC is considered). And the fact that I'm avoiding the host networking stack complicates matters, because I have to handle the translation between Rust code and the C code that exposes the VCL capabilities in VPP.

I'm using the end product daily (pulled together in a larger router framework that I haven't released yet) and tweaking as issues reveal themselves. If you'd like to participate, please feel free to drop me a line.