Setting Up Servers in Countries with CGNAT

For organisations using an ISP that has CGNAT configured, it can be difficult to acquire a static IP for on-premise servers. To circumvent this, we can set up a VPS that will function as a proxy to our on-premise server. The concept behind this is as follows:

1. The on-premise server connects to the VPS and authenticates using a secret.

2. The VPS assigns the on-premise server an IP address on an internal VPN network.

3. Any incoming requests to the VPS can then be routed to the internal IP address of the on-premise server.

In this document we will specifically explain how we implemented this functionality for our on-premise back-up server. In implementing this solution, there were two main concerns:

1. We want to be able to SSH into the back-up server from anywhere in the world.

2. We want to proxy backup requests from our back-up client to our back-up server without knowing the IP of the back-up server.

As an aside: It might be useful to create an Ansible Playbook for similar use cases. This should not be too difficult to accomplish and will make setup a lot easier.

Let's begin.

Setting up the VPS proxy

We begin by installing WireGuard. WireGuard is a VPN program that will allow us to easily define our VPN structure and generate the necessary keys for the nodes in our VPN network.

Installing WireGuard

To start, we will want to install WireGuard onto the server. For an Ubuntu server, that will mean executing the following command:

sudo apt install wireguard

If you are using a different Operating System, please use the appropriate installation command.

Generating keys

We must now generate two keys for this machine that will allow authentication of the device on the network and encrypted communication between the different devices.

To generate the private key, use the following command:

wg genkey | sudo tee /etc/wireguard/private.key
sudo chmod go= /etc/wireguard/private.key

• wg genkey will generate the private key.

• sudo tee /etc/wireguard/private.key will save the private key to the file in /etc/wireguard/private.key and print the generated key to the shell.

• sudo chmod go= /etc/wireguard/private.key will make sure root permissions are needed to access this file.

Note down the private key that you just generated, it should haven been printed to the shell. In this document we will from now on refer to this private key as <vpn-server-private-key>.

To generate the public key, use:

sudo cat /etc/wireguard/private.key | wg pubkey | sudo tee /etc/wireguard/public.key

This will take generate a public key based on our private key and saves this key to /etc/wireguard/public.key.

Note down the public key. This one does not need to be hidden and will in fact be shared with other devices. In this document, we will refer to this key as <vpn-server-public-key>.

Figure out the correct public network interface

Run the following command:

ip route list default

This should return something resembling the following output:

default via 172.26.0.1 dev ens5 proto dhcp src 172.26.15.225 metric 100

The important part of this output is the 5th column, in our case this says `ens5`. This will be our public network interface which is the interface that allows us to access the public internet.

Note down this interface name. We will from now on refer to it as <public-interface>.

Make sure to allow IP forwarding

As we will be forwarding IP requests from one network interface to another, we need to first make sure to allow IP forwarding in the OS configuration. To do this, open the following file:

sudo nano /etc/sysctl.conf

And make sure this line is uncommented:

net.ipv4.ip_forward=1

This will enabled IPv4 forwarding. If you need to enabled IPv6 forwarding, check online.

Setting up the Wireguard configuration

It is not time to set up the WireGuard configuration. This configuration should accomplish 2 things:

1. It will route all traffic from port 2222 to our on-premise server's port 22. This is to allow us to remotely SSH into our on-premise server.

2. It will route all traffic from port 8000 to our on-premise server's port 8000. This is specific to our use-case, as port 8000 is the port on the on-premise server that the backup server is listening on.

This will result in the following configuration (/etc/wireguard/wg0.conf):

[Interface]
Address = 10.0.0.1/32
ListenPort = 51820
PrivateKey = <vpn-server-private-key>

# Allow forwarding between <public-interface> and wg0
PostUp = ufw route allow in on <public-interface> out on wg0
PostUp = ufw route allow in on wg0 out on <public-interface>

# Forward any packets arriving on public interface port 2222 to 10.0.0.2:22 on wg0
PostUp = iptables -t nat -I PREROUTING -i <public-interface> -p tcp --dport 2222 -j DNAT --to-destination 10.0.0.2:22
PostUp = iptables -t nat -I POSTROUTING -o wg0 -p tcp --dport 22 -d 10.0.0.2 -j SNAT --to-source 10.0.0.1

# Forward any packets arriving on public interface port 8000 to 10.0.0.2 on wg0
PostUp = iptables -t nat -I PREROUTING -i <public-interface> -p tcp --dport 8000 -j DNAT --to-destination 10.0.0.2
PostUp = iptables -t nat -I POSTROUTING -o wg0 -p tcp --dport 8000 -d 10.0.0.2 -j SNAT --to-source 10.0.0.1

# Disallow forwarding between <public-interface> and wg0
PreDown = ufw route delete allow in on <public-interface> out on wg0
PreDown = ufw route delete allow in on wg0 out on <public-interface>

# Stop forwarding any packets arriving on <public-interface> port 2222 to 10.0.0.2:22 on wg0
PreDown = iptables -t nat -D PREROUTING -i <public-interface> -p tcp --dport 2222 -j DNAT --to-destination 10.0.0.2:22
PreDown = iptables -t nat -D POSTROUTING -o wg0 -p tcp --dport 22 -d 10.0.0.2 -j SNAT --to-source 10.0.0.1  

# Stop forwarding any packets arriving on <public-interface> port 8000 to 10.0.0.2 on wg0
PreDown = iptables -t nat -D PREROUTING -i <public-interface> -p tcp --dport 8000 -j DNAT --to-destination 10.0.0.2
PreDown = iptables -t nat -D POSTROUTING -o wg0 -p tcp --dport 8000 -d 10.0.0.2 -j SNAT --to-source 10.0.0.1

[Peer]
AllowedIPs = 10.0.0.2/32
PublicKey = <on-premise-public-key>

Let's try to understand what is happening here:

Interface section

In the interface section we define the properties of the internal VPN node that VPN server will be running on.

Address = 10.0.0.1/32

We state that the internal IP address on this network is 10.0.0.1. The /32 means that his IP must be matched exactly! If this were 10.0.0.1/24 it could match any IP in the range of 10.0.0.1 - 10.0.0.254. This is not what we want as we will be using the exact 10.0.0.1 address later on to configure our IP routing tables.

ListenPort = 51820

This node listens on the 51820 port to allow any incoming WireGuard connections necessary to set up the VPN.

PrivateKey = <vpn-server-private-key>

This node will use the <vpn-server-private-key> private key to authenticate with other nodes and encrypt any data.

PostUp = ufw route allow in on <public-interface> out on wg0

PostUp = ufw route allow in on wg0 out on <public-interface>

PostUp = iptables -t nat -I PREROUTING -i <public-interface> -p tcp --dport 2222 -j DNAT --to-destination 10.0.0.2:22

PostUp = iptables -t nat -I POSTROUTING -o wg0 -p tcp --dport 22 -d 10.0.0.2 -j SNAT --to-source 10.0.0.1

PostUp = iptables -t nat -I PREROUTING -i <public-interface> -p tcp --dport 8000 -j DNAT --to-destination 10.0.0.2

PostUp = iptables -t nat -I POSTROUTING -o wg0 -p tcp --dport 8000 -d 10.0.0.2 -j SNAT --to-source 10.0.0.1

The PostUp commands are executed when the WireGuard interface is started up. We will use these to configure the firewall and allow forwarding connections across the <public-interface> and the wg0 internal interface, as well as define how this forwarding is supposed to work.

PreDown = ufw route delete allow in on <public-interface> out on wg0

PreDown = ufw route delete allow in on wg0 out on <public-interface>

PreDown = iptables -t nat -D PREROUTING -i <public-interface> -p tcp --dport 2222 -j DNAT --to-destination 10.0.0.2:22

PreDown = iptables -t nat -D POSTROUTING -o wg0 -p tcp --dport 22 -d 10.0.0.2 -j SNAT --to-source 10.0.0.1

PreDown = iptables -t nat -D PREROUTING -i <public-interface> -p tcp --dport 8000 -j DNAT --to-destination 10.0.0.2

PreDown = iptables -t nat -D POSTROUTING -o wg0 -p tcp --dport 8000 -d 10.0.0.2 -j SNAT --to-source 10.0.0.1

The PreDown commands are executed when the Wireguard interface is being taken down. We will use these to undo any of the changes we made with the PostUp demand.

Peer section

The peer section defines the peers it allows in the VPN.

AllowedIPs = 10.0.0.2/32

AllowedIPs defines the allowed IP addresses for the peers in the network that it can connect to. In our case this is again a singular IP address 10.0.0.2/32 because, as can be seen in the routing configuration above, this gets used explicitly for NAT.

Setting up the firewall configuration

We will now need to set up a firewall to protect the VPS, but also to allow access to the necessary ports. In particular we need to make sure that ports 22, 8000, 51820/udp are accessible.

To do this we will be using ufw.

sudo ufw allow OpenSSH
sudo ufw allow 2222
sudo ufw allow 8000
sudo ufw allow 51820/udp

If the VPS provider has its own firewall, make sure this one is configured accordingly.

To enable the Firewall run:

sudo ufw enable

To check if everything is operating correctly, use:

sudo ufw status

Running the WireGuard service

We now want to run the WireGuard service and make sure it gets started up on boot.

To enable it in SystemD for startup on boot use the following command:

sudo systemctl enable wg-quick@wg0.service

To start it up immediately, we now want to run:

sudo systemctl start wg-quick@wg0.service

The WireGuard interface should now be running. We can check this with:

ip a

To take this interface down you can use:

sudo wg-quick down wg0

and to bring it back up:

sudo wg-quick up wg0

Setting up the on-premise server

Install WireGuard

As on the VPS server, here we will need to install WireGuard. To do so run the same command:

sudo apt install wireguard

Generate WireGuard keys

We will also need to generate a set of keys for the on-premise server. Do this as follows:

wg genkey | sudo tee /etc/wireguard/private.key
sudo chmod go= /etc/wireguard/private.key

and

sudo cat /etc/wireguard/private.key | wg pubkey | sudo tee /etc/wireguard/public.key

Note these down, the private key will be refered to as <on-premise-private-key> and the public key as <on-premise-public-key>.

WireGuard configuration

We will now need to define the configuration for this device. This one will be substantially easier than the VPN server configuration and consists of the following:

[Interface]

PrivateKey = <on-premise-private-key>
Address = 10.0.0.2/32

[Peer]
PublicKey = <vpn-server-public-key>
AllowedIPs = 10.0.0.1/32
Endpoint = <vpn-static-ip>:51820
PersistentKeepalive = 25

What does this mean?

Interface section

We have already seen this interface definition before, the only thing missing here is the `ListenPort` argument which we can leave out here. This server does not need a specific port listening for connections as it will only server as a client, this means WireGuard is free to use whichever listening port it wants when needed.

Peer section

The peer section contains some differences though. We can see a few new fields, namely:

Endpoint = <vpn-static-ip>:51820

Here Endpoint defines how to connect to the VPN server peer by specifying the exact IP address and port where it will be listening for our connection.

PersistentKeepalive = 25

This PersistentKeepalive ensures that the on-premise server will automatically attempt to connect to the VPN server and keep the connection alive as long as the WireGuard interface is running. This is important because otherwise the VPN server will only know that the on-premise server exists when the on-premise server explicitly tries to connect to the VPN server first. There is not way for the VPN server to divine the location of the on-premise server otherwise.

Running the WireGuard service

As with the VPN server, use the following commands to start up the server and enable on boot:

sudo systemctl enable wg-quick@wg0.service

To start it up immediately, we now want to run:

sudo systemctl start wg-quick@wg0.service