Part 5: Bonus! Use ZeroTier as mobile VPN.

Here is a bonus post. I am not going to go into deep details but should be enough to give a good idea on how to do this.

First thing I would suggest creating a second zerotier network from the routed LAN for all the mobile devices.  I would make sure the IP subnet is different than anything else, so following example, 10.10.0.0/24 for example.  Next set the range to .10-250 of the last octet under advanced.  The reason for this is we are going to set a bit on the routervms and turn them into nat routers on their own IPs.  Also use one of the higher ips (say .254 perhaps?) as floating IP between the VMs.  This way when we route, we route to the .254 and if something having problems, just move the .254 to another host and count to 30 and things should work again.

Now what will make this work for the clients.  You need to add routes in the zerotier network.  You can do two options and I will give comments on each.

  1. Add
    • 0.0.0.0/0 via 10.10.0.254
    • This is the most common most people would use.  It is a simple common route to push for default route to send everything.  The problem I ran into is not everything can take a default route like this out of the box.  So I gamed the system with the next one.
  2. Or Add
    • 0.0.0.0/1 via 10.10.0.254
    • 0.0.0.0/128 via 10.10.0.254
    • The reason this works is we can push two routes and neither is technically default.  The reason why this works is that they are more specific.  This means these routes would be picked over the default in most cases.

At this point it is time to edit all the routervms to enable nat routing and put the IPs on them.

      1. Join the routervms to the new VPN network
      2. Run the following to disable routes pushing to the routervms (that would cause some problems for routing).
        • sudo zerotier-cli set <network> allowManaged=false
          sudo zerotier-cli set <network> allowGlobal=false
          sudo zerotier-cli set <network> allowDefault=false
      3. Edit /etc/sysctl.conf and add the following to the bottom
        • net.ipv4.ip_forward = 1
      4. Edit /etc/sysconfig/iptables
        • *nat
          :PREROUTING ACCEPT [0:0]
          :INPUT ACCEPT [0:0]
          :OUTPUT ACCEPT [0:0]
          :POSTROUTING ACCEPT [0:0]
          -A POSTROUTING -o eth0 -s 10.10.0.0/24 -j SNAT --to-source eth.ip.goes.here
          COMMIT
          *filter
          :INPUT ACCEPT [0:0]
          :FORWARD DROP [0:0]
          -A FORWARD -i zt+ -s 10.10.0.0/24 -d 0.0.0.0/0 -j ACCEPT
          -A FORWARD -i eth0 -s 0.0.0.0/0 -d 10.10.0.0/24 -j ACCEPT
          :OUTPUT ACCEPT [0:0]
          COMMIT
      5. Now add .1-? to the routervms network configs.  So one gets .1 and another gets .2 and so on.  This way when they get rebooted they at least show up on the network.
      6. Now add .254 to only one of the routervms via,
        • ip addr add 10.10.0.254/24 dev <gobblygook network interface>

If everything was done right (and I did not skip a step) you should now be able to join a new device and approve it for your network.  It should then start sending all its traffic to the designated routervm and hopefully have interent connectivity.

Now the real reason for sending all traffic to the .254 and having it as a secondary ip.  I have yet to try to configure this but it should be possible to setup heartbeat or pacemaker to control the .254 address.  So that it auto moves for you between the hosts as the fail on the network.  Thus ensuring you will always have access across the “VPN”.

Part 4: Test everything including breaking it!

At this point if everything works you should be able to ping between the networks.  If that isn’t working then you need to troubleshoot what is breaking down communication wise.  This could be anything from the two local routers not sharing routes to the routers not talking over zerotier.  The point I am trying to push across is that there is no simple gotchas I can offer troubleshooting steps for.

Once connectivity across the LANs is working, the next good thing you want to do is break it of course.  First simple break I would suggest is just reboot the router vms.  Ideally if everything is setup right and once they are booted back up and the services start, routing should be restored and everything should work again.  Again if things do not work the next thing to do is troubleshoot and solve, as there are to many possibilities to mention.  I would even suggest break the routing demons and make sure you can repair them as needed.

At the end of this now you have a working multi-site lan that each site has its own working IP subnet.

Final comment: People have asked me is it possible to bridge LANs?  The answer is yes but that is an entirely different process and one I would not recommend.

Setup your own DNS over TLS

So I have gone a little crazy lately in my home lab. I have created a anycast address in the LAN of 10.10.10.10 that goes to the nearest pihole. (Local, Datacetner 1 or Datacenter 2)  While that was nice I still wanted a way to have pihole while on the go.  I thought about a VPN, that works but is not perfect for what I want.  A little more pondering and I found that Android 9 supports “Private DNS”.  Turns out that it is a simply DNS over TLS. (DoT)  That just makes this so much easier now.

How does one make a DoT server then?  Again answer is really simple, stunnel4 is all you need.  A quick bit of googling will get you to this page, which will walk through a more indepth setup of stunnel4.  Boiled down, and assuming you have a SSL cert handy/installed on the system:

# yum install stunnel4

# cat /etc/stunnel/dnstls.conf
[dnstls]
cert = /etc/letsencrypt/live/mindlesstux.com/fullchain.pem
key = /etc/letsencrypt/live/mindlesstux.com/privkey.pem

accept = 853
connect = 10.10.10.10:53


# systemctl enable stunnel4
# systemctl start stunnel4

After that, it was a matter of punching a hole in the firewall to allow 853 in to the server running the stunnel4 daemon.  Once that was up I added a DNS record for exampledot.mindlesstux.com to the ip where the firewall hole was punched.  Finally in my android Settings -> Network & Internet -> Advanced -> Private DNS.  I set that to “Private DNS provider hostname” and put the mentioned hostname in the text field.  Bam, now anywhere I go my phone is covered by one of my piholes.

Now to look into providing DNS over HTTPS (DOH) the same way…

Reference URL: https://kb.isc.org/docs/aa-01386
Reference URL: https://en.wikipedia.org/wiki/DNS_over_TLS
Reference URL: https://android-developers.googleblog.com/2018/04/dns-over-tls-support-in-android-p.html

Part 3: Setup router to use the routing VM(s)

With the VM building out of the way, on to configuring the VMs.

First a little more software install/prep work

yum install quagga
systemctl enable zebra
systemctl enable ospfd
systemctl disable firewalld
cp /usr/share/doc/quagga-*/zebra.conf.sample /etc/quagga/zebra.conf
cp /usr/share/doc/quagga-*/ospfd.conf.sample /etc/quagga/ospfd.conf
chown quagga.quagga /etc/quagga/*.conf
setsebool -P zebra_write_config 1
systemctl start zebra
systemctl start ospfd
systemctl stop firewalld
vtysh

Basically, install quagga, a routing daemon that works similar to cisco ios. From there enabling the two services we need for ipv4 connectivity. Zebra is the main quagga daemon, the ospfd is the daemon to run this as an OSPF router. There is an ospf6d daemon that I have yet to mess with to route the fd00::/8 ipv6 network.
In the process of setting this up, I found the default firewall is pointless and causes problems with the routing daemons unless you open the right ports and subnets. I chose to disable the firewall as these VMs are not directly exposed to the internet and are behind firewalls already.
The last two odd commands that may not be familiar:
The setsebool, allows the quagga daemon to modify the /etc/quaga/*.conf files.
The vtysh, this is how you get into the quagga software. Thus into the next bit.

routervm-site1# conf t
routervm-site1(config)# int eth0
routervm-site1(config-if)# ip address 10.10.9.2/29
routervm-site1(config-if)# ip ospf authentication message-digest
routervm-site1(config-if)# ip ospf message-digest-key 1 md5 quagga
routervm-site1(config-if)# exit
routervm-site1(config)# int zt-interfacename
routervm-site1(config-if)# ip address 10.10.10.251/24
routervm-site1(config-if)# ip ospf authentication message-digest
routervm-site1(config-if)# ip ospf message-digest-key 1 md5 quagga
routervm-site1(config-if)# exit
routervm-site1(config)# router ospf
routervm-site1(config-router)# ospf router-id 10.10.10.251
routervm-site1(config-router)# network 10.10.10.251/24 area 0.0.0.0
routervm-site1(config-router)# network 10.10.9.2/29 area 0.0.0.1
routervm-site1(config-router)# area 0.0.0.0 authentication message-digest
routervm-site1(config-router)# neighbor 10.10.10.252
routervm-site1(config-router)# neighbor 10.10.10.253
routervm-site1(config-router)# exit
routervm-site1(config)# ip forwarding
routervm-site1(config)# ipv6 forwarding
routervm-site1(config)# exit
routervm-site1# write

To sum this up:

    • Enter configuration terminal
    • interface eth0, (may need to change this to match, silly eno2345 names now days are possible.
    • Configure the IP address
    • Setup MD5 authentication and defines the key “quagga”
    • Drops back to config terminal, and then into the ZeroTier interface
    • A repeat of the eth0 interface, just different IP
    • Drops back to config terminal, and into the OSPF routing bit
    • Defines the OSPF router ID, I chose the ZeroTier IP.
    • Defines the networks OSPF is to run on and in what area
      • I went with the backbone area, as I don’t really want to get to complex
    • Defines the area to use MD5 authentication
    • Defines the OSPF neighbors,
      • In this case the other two VMs in the ZeroTier network.
    • Drops back and sets IP forwarding for IPv4 and IPv6
    • Drops back again and writes configs to disk

At each site where one of these VMs is set up, change the IP address as needed.  (network interfaces, neighbors, and router-id)

Continue reading

Part 2: Build the router VM(s)

To bring us into part 2, the next thing is to create the routing VMs.

I simply did a simple CentOS 7 minimal install with one network interface configured.  At the router, I created network 10.10.9.0/29 on the same LAN segment.  During setup of the VM, I configured the VM to be 10.10.9.2/29 with a gateway of 10.10.9.1/29.  I also did the same with IPv6, fd00:10:10:9::2/64 and fd00:10:10:9::1/64 respectively.  Now when I get to other sites to deploy, the network will change to be 10.10.9.8/29 & 10.10.9.16/29.  The reason I chose to go with a /29 here is simply due to I plan to layer in a “services” into these subnets.  This particular set of VMs I plan to layer on a Pi-Hole and Chrony NTP server.

If you have trouble installing CentOS 7 at a minimal install, then this series is not for you.

Once you have the VM created, the first thing to do is install ZeroTier.  Thankfully they make this dead simple.

If you want to verify things:
curl -s 'https://pgp.mit.edu/pks/lookup?op=get&search=0x1657198823E52A61' | gpg --import && if z=$(curl -s 'https://install.zerotier.com/' | gpg); then echo "$z" | sudo bash; fi

If you like to live risky:
curl -s https://install.zerotier.com/ | sudo bash

Run either as root and it is installed.

Other small things I have found that are needed after install,
In /etc/sysconfig/network-scripts/ifcfg-eth0, add the following: NM_CONTROLLED=”no”
In /etc/sysconfig/network, add the following: NETWORKING=yes
In /etc/selinux/config, change SELINUX to be permissive, SELINUX=permissive
Reboot a couple of times, make sure that default gateway and DNS servers show up in their respective places.

At this point, the next thing to do is create the ZeroTier network.  Simply log in and go the networks page and create a new network.  Once that is done, the next thing is to join the network.

zerotier-cli join abcd1234abcd1234

Finally, go authorize the new clients and label them in the “My Networks”.