PROBLEM: Everyone can test SSH access on my server
SOLUTION: Use ufw to allow connection only from specified network

Just use simple ufw command, where 10.5.0.0/24 is your e.g. VPN network (or you can use your public IP) and port number is desired port for SSH (can be adjusted in /etc/ssh/sshd_config).

root@server:~# ufw allow from 10.5.0.0/24  to any port 22