Linux + Starlink + IPv6

Adventures in connecting a local network with IPv6

A little history

Sometime around 1994 I was using Linux at home to login to my workplace network, using a dial-up modem. I then got an ISDN internet connection, as I recall with an external modem connected with ethernet. After a period of inactivity, the connection would drop, but by keeping pinging, it would stay up. I could then connect back to my home computer from work.

After a couple of years I got a cable modem from my cable TV supplier, again an external device with an ethernet connection. The ISP gave me a quasi-static IP address - it stayed the same for months, unless they upgraded their network or my computer lost power for more than a day. So I could use SSH to connect to my home PC from work, equally as well as connecting to work from home. I'd do a lot of work from home, or occasionally home-related things from work, such as calling insurance, and had a lot of files and documents on both computers. It didn't really matter which, since I could access either from anywhere the was internet. I gave my home PC a DNS entry and ran a webserver with all my personal photos, as well as email for a while.

When my kids needed computers in the 2000's, I got a second ethernet card and a network hub and set up masquerading - what is now called NAT - forwarding packets from the local network to the cable modem. Later I upgraded the hub to a switch, and later when I got a laptop and tablet with built-in WiFi, got an access point. I ran DHCP and DNS for my local network, so everything got a name and a static IP address. I've run the same configuration ever since, with a few hardware and software upgrades. The main desktop is now running CentOS6.

When I moved from the city to a rural area in 2019, I lost the cable modem. There's no fibre or cable or DSL. None of the ISP options are as reliable, and I could not get a static IPv4 address, even a quasi one, without paying more. The interface equipment does NAT or CGNAT internally so I can't get a public address of any kind.

Since I retired shortly before moving, that wasn't so much of a problem, but I had to move some of my notes and photos to my cloud server. However,there's not enough room for everything. When I was stuck in hospital for months, though, it became more annoying. I was able to run a reverse tunnel in SSH, but it would not stay up - the internet connection had odd drop-outs and the tunnel needed to be re-started by someone at home.

I had moved to Starlink to try and get a better internet connection, and found that it automagically supported IPv6, giving my computer a /64 quasi-static address. So in theory I could connect directly from the internet again. However, the hospital network only supported IPv4, and for that matter my primary cloud provider did not support IPv6 either. I had an Amazon ECS instance as a secondary DNS server, and switched that for a newer "Lightsail" instance that easily and by default supported IPv6. In fact, Amazon charge more for IPv4 connectivity. So by using SSH to connect to that via IPv4, I could login to my home PC over IPv6. Hooray. But if for some reason I wanted to connect to another device, I'd have to go through the dual-interface PC. I have a couple of IP cameras at home, and I was able to access them by firstly setting up a web proxy in Apache on my home PC, and then running Squid on the Amazon instance. By setting a proxy in Firefox, it would retrieve images via the IPv6 link from the webserver at home. Not exactly straightforward.

A real IPv6 network

Some weeks ago I thought I'd try and get a proper IPv6 connection for my home network. IPv6 has billions of possible addresses, enough for every tree on the planet, let alone every phone or refrigerator, so there's no need for NAT.

Per Starlink's documentation, Starlink provides

  • One public IPv4 address for the customer’s wide area network (WAN), provisioned via Dynamic Host Configuration Protocol (DHCP) for routers/firewalls using IPv4.
  • One IPv6 /64 prefix for the customer’s wide area network (WAN), provisioned via Stateless Address Auto Configuration (SLAAC) for routers/firewalls using IPv6.
  • One IPv6 /56 prefix for the customer’s local area network (LAN), provisioned to routers capable of issuing a DHCPv6-PD request.
(Unless you pay extra, the IPv4 address is a local address provisioned through CGNAT).
IPv6 Router Advertisements from Starlink have the "M" flag clear and the "Other" flag set; this means that the prefix and router addresses are in the RA packet but DNS should be obtained via DHCP.

How networking works in Linux

Networking is done in the kernel, with some helper programs. Kernel parameters are viewable in /proc/sys/net, and can be set at boot time in /etc/sysctl.conf.
Parameters are documented in the kernel source at Documentation/networking/ip-sysctl.txt
Some IPv6 parameters are compiled into the kernel, and may be found in /boot/config-<kernel-version>
Route information can be seen at /proc/net/route and /proc/net/ipv6_route; it's normal to interact with that using programs such as "ip" or "route".

In CentOS6, general network parameters are set in /etc/sysconfig/network and interface-specific ones in /etc/sysconfig/network-scripts/ifcfg-<interface-name>
Those are used by other scripts in network-scripts and documented in /etc/sysconfig/network-scripts/ifup-ipv6. Normally those are set by the NetworkManager process; in a desktop environment from the nm-connection-editor GUI.

IPv4 Networking

Addresses and routes in IPv4 are either static, set manually, or dynamic using DHCP. For DHCP, there's an entry "BOOTPROTO=dhcp" in ifcfg-<interface-name>. CentOS6 uses a DHCP client "dhclient", with a config file /etc/dhcp/dhclient.conf. Normally you'd request routers and domain-name-servers.
Out-of-the-box, normally this just works.

To get NAT for my local network, I have the following in an init script

echo 1 > /proc/sys/net/ipv4/ip_forward

IPv6 networking

Addresses and routes in IPv4 can be static, set dynamically by DHCP6, or set automatically by Router Advertisements in Stateless Address Auto Configuration (SLAAC). That's controlled by kernel parameters such as /proc/sys/net/ipv6/conf/all/accept_ra, and is normally the default. So IPv6 networking for a single device with dual stack "just works" on Starlink, getting DNS via DHCP4. IPv6 is by default the preferred protocol in Linux, so once IPv6 is configured you may start getting HTTP responses on IPv6 from e.g. google.com.

Prefix Delegation

Getting IPv6 running with global addresses (as opposed to Link-Local) on a local network is more complicated.

What is currently working for me is running dhcp6c (from the wide-dhcpv6-20080615 package) to get a delegated prefix from Starlink, and running radvd to advertise routes on the local network. I also need to set /proc/sys/net/ipv6/conf/all/forwarding, This normally means that router advertisements are ignored, but since kernel 2.6 accept_ra can be set to 2, per ip-sysctl.txt, to accept Router Advertisements even if forwarding is enabled.

accept_ra on eth0 causes a conflicting default route to be created with metric 1024 on eth0. That causes forwarding to dain, unless a route with a smaller metric is created on eth7. Setting accept_ra=2 on eth7 and accept_ra=0 on eth0 seems to work.

I have the following in /etc/wide-dhcpv6/dhcp6c.conf 

interface eth7 {
  send ia-pd 0;
  request domain-name-servers;
  script "/etc/wide-dhcpv6/my-dhcp6c-script" ;
};

id-assoc pd {
  prefix-interface eth0 {
    sla-id 1;
    sla-len 8;
  };
};
That sends a request for a prefix for eth0 to the Starlink router via eth7, and also requests a list of domain name servers. The named script is run when a prefix is obtained, and can be used to set DNS servers in /etc/resolv.conf.
The router responds with a prefix packet such as 
    Option: IA Prefix (26)
    Length: 25
    Preferred lifetime: 150
    Valid lifetime: 300
    Prefix length: 56
    Prefix address: 2605:****:****:****::
DNS recursive name server
    Option: DNS recursive name server (23)
    Length: 32
    DNS servers address: 2001:4860:4860::8888
    DNS servers address: 2606:4700:4700::1111

That shows up in stdout for dhcp6c running in foreground mode with debugging enabled, but is otherwise not obviously saved. It's used to set a global address for the local interface eth0.

I modified dhcp6c to write the prefix to /var/run/dhcpv6/prefix.dat, in a form 

Prefix 2605:****:****:****::/56 pltime=150, vltime=300
Using this information, I created /etc/radvd.conf as 
interface eth0 {
        AdvSendAdvert on;
        MinRtrAdvInterval 30;
        MaxRtrAdvInterval 100;
        AdvDefaultLifetime 300;
        prefix 2605:****:****:****::/64 {
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr off;
                AdvValidLifetime 300;
                AdvPreferredLifetime 150;
        };
};
(using /56 gets an error "invalid prefix length 56 + 16 + 64") ip6tables

By default, forwarding is blocked in ip6tables with 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all      anywhere             anywhere            reject-with icmp6-adm-prohibited
This block must be removed, ideally to allow only the configured global addresses access, or more crudely with 
ACCEPT     all      anywhere             anywhere
By default, the INPUT rule is set to something like 
Chain INPUT (policy ACCEPT)
target     prot opt   in     out     source      destination         
ACCEPT     all        any    any     anywhere    anywhere     state RELATED,ESTABLISHED 
ACCEPT     ipv6-icmp  any    any     anywhere    anywhere            
ACCEPT     all        lo     any     anywhere    anywhere            
ACCEPT     tcp        any    any     anywhere    anywhere     state NEW tcp dpt:ssh 
REJECT     all        any    any     anywhere    anywhere     reject-with icmp6-adm-prohibited
This allows outgoing traffic and replies to same, and incoming SSH traffic. For prefix delegation to work, it must also accept DHCPv6 packets. Again, I had temporarily set "ACCEPT all anywhere".

My current working configuration, concatenated from various sources, is 

/proc/sys/net/ipv6/conf/all/forwarding  1
/proc/sys/net/ipv6/conf/all/accept_ra   0
/proc/sys/net/ipv6/conf/all/autoconf    1
/proc/sys/net/ipv6/conf/all/disable_ipv6        0
/proc/sys/net/ipv6/conf/all/accept_redirects    0
/proc/sys/net/ipv6/conf/eth0/forwarding 1
/proc/sys/net/ipv6/conf/eth7/forwarding 1
/proc/sys/net/ipv6/conf/eth0/accept_ra  0
/proc/sys/net/ipv6/conf/eth7/accept_ra  2
/proc/sys/net/ipv6/conf/eth7/accept_redirects   1
/proc/sys/net/ipv6/conf/eth0/accept_redirects   1

ifconfig
eth0  inet6 addr: 2605:****:***:****:****:****:****:****/64 Scope:Global
      inet6 addr: fe80::****:****:****:****/64 Scope:Link
eth7  inet6 addr: 2605:****:***:****:***:***:****:****/64 Scope:Global
      inet6 addr: fe80::***:***:****:****/64 Scope:Link
fe80::/64 dev eth0  proto kernel  metric 256  mtu 1500
fe80::/64 dev eth7  proto kernel  metric 256  mtu 1500

ip -6 route
fe80::/64 dev eth0  proto kernel  metric 256  mtu 1500
fe80::/64 dev eth7  proto kernel  metric 256  mtu 1500
default via fe80::200:5eff:fe00:101 dev eth7  proto kernel  metric 1024  expires 193sec mtu 1500 hoplimit 64

/etc/sysconfig/network
NETWORKING=yes
IPV6FORWARDING=yes

/etc/sysconfig/network-scripts/ifcfg-Starlink-eth7
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=yes
NAME="Starlink eth7"
ONBOOT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
PEERDNS=yes
PEERROUTES=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

/etc/sysconfig/network-scripts/ifcfg-Local_eth0
TYPE=Ethernet
BOOTPROTO=none
IPADDR=192.168.2.14
PREFIX=24
DOMAIN=local
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=yes
NAME="Local eth0"
ONBOOT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes

Andrew Daviel