~

Background

As we all know, exposing the SSH port directly on the internet has certain risks. While certificate-based authentication is fairly secure, whenever a port is exposed there will always be someone trying to attack the system through open SSH port. The same is true on Alibaba Cloud. Obviously, if you just need an SSH connection, you can use the password-free login on the web page to operate.

However, sometimes more than just an SSH port is needed. For example, YDJSIR found that Hadoop, Spark and other software will expose a lot of ports once they are started, and there is no authentication by default, which is obviously very dangerous. Especially for Spark, you always have to bind an IP that the EIP actually has, but that is the internal network IP of Alibaba Cloud and cannot be accessed directly (in fact, there is a workaround, first add an elastic network card to the current ECS instance, and then enable an elastic Public network IP, which can be bound in EIP network card visible mode). However, even with access, this is very insecure. In order to solve the above problems, the most direct idea is to find a way to make the equipment of the operation and maintenance/developer and the cloud server in the same intranet.

However, the threshold for related encapsulated services provided by Alibaba Cloud is quite high, and it is not suitable for small and micro developers. In this case, lets do it by ourselves. YDJSIR chose the open source OpenVPN (UDP+TUN solution) to solve such needs. It is modern, secure and efficient compared to traditional L2TP, PPTP, and has better cross-platform support and documentation (maybe?) than Wireguard. So why doesn’t YDJSIR use v2ray? YDJSIR knows that non-international routes are probably fine but still doesn’t want to trigger an alert. OpenVPN Alibaba Cloud has official tutorials on the domestic site, that’s it (references attached). After all, YDJSIR has been using OpenVPN since junior high school.

Architecture

image-20220421221916618

Developers and maintainers will connect to ECS0’s OpenVPN first, then access other resources in VPCs.

ECS0 only needs to be larger than 1C512M with x86_64 architecture.

Reference

No Title 链接 备注
1 How To Set Up and Configure an OpenVPN Server on CentOS 7 https://www.digitalocean.com/community/tutorials/how-to-set-up-and-configure-an-openvpn-server-on-centos-7 The main reference of this article
2 【干货】ECS服务器OPENVPN搭建,方便管理所有内网服务器 https://chowdera.com/2021/05/20210528104005167Q.html
3 在CentOS系统的ECS实例中如何配置OpenVPN https://help.aliyun.com/document_detail/42521.html Alibaba Cloud’s document of using OpenVPN in Alibaba Cloud
4 解决通过openvpn能ping通服务器,tcp连接不通的问题 https://www.icode9.com/content-4-148712.html solve the problem of being able to ping but TCP connection cannot be established
5 Linux实例常用内核网络参数介绍与常见问题处理 https://help.aliyun.com/document_detail/41334.html Alibaba Cloud’s explanation of reference4

Detailed Process

All commands are executed by the user provided by Alibaba Cloud’s web management tools, ecs-assist-user .

The ECS’s OS is CentOS 7.9. The security group has allowed UDP 1194 to go.

Install Software

1
2
3
4
sudo yum update -y
sudo yum install epel-release -y
sudo yum update -y
sudo yum install -y openvpn wget

Please pay attention to the version of OpenVPN below, and modify it accordingly in the paths involved later.

1
2
3
4
wget -O /tmp/easyrsa https://github.com/OpenVPN/easy-rsa-old/archive/2.3.3.tar.gz
tar zxvf /tmp/easyrsa
sudo mkdir /etc/openvpn/easy-rsa
sudo cp -rf /tmp/easy-rsa-old-2.3.3/easy-rsa/2.0/* /etc/openvpn/easy-rsa

Generating Certificates

1
sudo mkdir /etc/openvpn/easy-rsa/keys

Now, set the variables for later generation.

1
sudo vim /etc/openvpn/easy-rsa/vars
1
2
3
4
5
6
7
8
9
# /etc/openvpn/easy-rsa/vars
export KEY_COUNTRY="CN"
export KEY_PROVINCE="JS"
export KEY_CITY="Nanjing"
export KEY_ORG="DI"
export KEY_EMAIL="[email protected]"
export KEY_CN=ydjsir.com.cn
export KEY_NAME="server"
export KEY_OU="HW"

Make these variables effective.

1
source /etc/openvpn/easy-rsa/vars

Now generate the certificates.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cd /etc/openvpn/easy-rsa
./clean-all # init
./build-ca # CA

# server-side
./build-key-server server
./build-dh
sudo openvpn --genkey --secret /etc/openvpn/myvpn.tlsauth

# client-side
./build-key client

# distribute the config
sudo cp key/dh2048.pem key/ca.crt key/server.crt key/server.key /etc/openvpn
cp /etc/openvpn/easy-rsa/openssl-1.0.0.cnf /etc/openvpn/easy-rsa/openssl.cnf

Server-side Config

/etc/openvpn/server.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
local ${EIP_internal_IP}
port 1194
proto udp
dev tun
ca ca.crt
cert server.crt
dh dh2048.pem
key server.key
tls-crypt myvpn.tlsauth
server 10.8.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt
duplicate-cn
keepalive 10 120
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
log /data/logs/openvpn/openvpn.log
verb 3
push "route 172.26.128.0 255.255.240.0"
push "route 172.66.0.0 255.255.255.0"

Details

  1. ==OpenVPN binding address==. The actual IP displayed by ifconfig in the system must be written here. OpenVPN binds to a specific network card in this way. Of course, if you enable EIP visible mode, then this will be the public IP.

  2. ==Allows to assign multiple IP addresses for multiple connections to many users using same set of configurations==. If you want to reuse a configuration file, this is necessary, otherwise there will be IP address conflicts.

  3. ==Configure the Alibaba Cloud VPC network segment that you need to allow the client to access==. These network segments must first be the network segments that your server itself can access, which means that you will have to pay attention to security groups and whitelists. Please configure this part according to the actual needs.

  4. The same as No.23

Configure Routes

Set Forwarding Rules

Here we will use firewalld rather than iptables behind it to achieve forwarding, as the latter will have to reload configuration at every system boot. To allow data to flow freely between the OpenVPN network card and the target network, we need to configure routing and firewall rules. Line 5 maps the address under the OpenVPN subnet to the target network card by static NAT. In this way, our OpenVPN can connect to other intranets. Note that only IPv4 is forwarded here.

1
2
3
4
5
6
7
8
sudo systemctl enable firewalld
sudo firewall-cmd --permanent --add-masquerade # enable IP masquerade
sudo firewall-cmd --permanent --add-service=openvpn # enable the service
sudo firewall-cmd --add-port=1194/udp --permanent # enable the port
sudo firewall-cmd --permanent --direct --passthrough ipv4 -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source ${EIP_internal_IP}
sudo firewall-cmd --reload # reload the rules
sudo systemctl reload firewalld # reload the service
sudo systemctl restart firewalld # restart the service

Edit sysctl Preload/Configuration File

1
sudo vim /etc/sysctl.conf
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
vm.swappiness = 0
kernel.sysrq = 1

net.ipv4.ip_forward = 1 # Allow system-wide IPv4 forwarding
net.ipv4.neigh.default.gc_stale_time = 120

# see details in https://help.aliyun.com/knowledge_detail/39428.html (Chinese Only)
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2

# see details in https://help.aliyun.com/knowledge_detail/41334.html (Chinese Only)
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_slow_start_after_idle = 0

net.ipv4.tcp_timestamps = 0 # Disable TCP timestamps, otherwise you will be able to ping but enable to make any TCP connection
net.ipv4.tcp_tw_recycle = 0 # Disable fast recycling TIME-WAIT sockets

The last two lines of configuration cannot be absent. For detailed reason, please visit https://help.aliyun.com/knowledge_detail/41334.html (Chinese).

Execute the command below to make these configurations effective.

1
sudo sysctl -p

Start the OpenVPN Server

1
2
sudo systemctl enable [email protected]
sudo systemctl enable firewalld

Restart the server to verify that OpenVPN server will automatically launch during the system boot.

Client-side Config

Certification and Configuration Files

The files below should be platform independent.

  • client.opvn
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
client 
dev tun
proto udp
remote ${EIP_external_IP} 1194
remote-cert-tls server
resolv-retry infinite
nobind
persist-key
persist-tun
ca ca.crt
cert client.crt
key client.key
tls-crypt myvpn.tlsauth
comp-lzo
verb 3
  • ca.crt
  • client.crt
  • client.key
  • myvpn.tlsauth

Windows

  1. Install the OpenVPN install package provided by OpenVPN’s official website.

  2. Copy all the certification and configuration files mentioned above to OpenVPN‘s user config folder.

image-20220421000320402

  1. Start the OpenVPN.
image-20220421000336494

Linux

  1. Install OpenVPN. This can be installed directly from the software source, Ubuntu uses apt, CentOS uses yum… I won’t go into details here. The version should be between 2.4-2.5 as much as possible.

  2. Copy the configuration file and its dependent files to /etc/openvpn/client/config_v2

  3. Run the following command to start OpenVPN and connect.

1
sudo openvpn --daemon --cd /etc/openvpn/client/config_v2 --config client.ovpn --log-append /var/log/openvpn.log

Conclusions

How could I expected that a normal deployment activity would accidentally introduced a kernel parameter adjustment to disable TCP timestamps! It wasn’t until I couldn’t help searching with Google that I found out the solution. The network is complex, and it is necessary to understand the TCP/UDP connection mode. YDJSIR still has a lot to learn in computer networking.