Security is an important aspect for any software technology company, the practice is to keep the systems not accessible over public networks and to achieve this VPNs are now an integral part of any organization. In today’s cloud computing world, most of the things are remote. Accessing some private network or services, they must be done securely and reliably. This is achieved using the VPN.
What is a VPN?
A Virtual Private Network (VPN) is a service that allows you to connect to the Internet via an encrypted tunnel to ensure your online privacy and protect your sensitive data. VPNs are commonly used to secure connections to public Wi-FI hotspots, hide IP addresses and make your browsing private. In case of private networks it can restrict your access.
One of our client had products both on the cloud and data centers and each product had their own VPN. For cloud products the VPN were earlier set up using OpenVPN. The OpenPVN client profile was shared among the users. They also had their solutions deployed on their clients Datacenter so these clients provided their custom VPN setup to access the datacenter network, which was again shared among the users accesing it. But there are many challenges with the shared VPNs.
The Challenges
- Repeated Effort: Each user had to set up the VPN on his machine. If he is working on multiple products, he need to setup and manage those many VPNs on his machine.
- Impersonation: In shared VPN, the credentials are shared with all the users, so users can easily impersonate others.
- Security and compliance issues: Security is at risk with this approach. Employees might share the credentials outside of the company or carry with him/her when no more works with the organization. There is no easy way to have control over these things.
- No Audit: No way can administrator audit logs of users activities or monitor the VPN traffic.
- Rotating Credentials: VPN credentials are rotated frequently for security. But the new credentials has to be shared again with all the users again. Waste of time for them.
- Set up for non-technical people: Non-technical has to struggle more than anyone else to use VPN. It hamper their productivity as they need to manage VPNs on their machine.
- Difficult for Devops: From an SRE or Administrator’s perspective such individual setups on each machine makes his/her life worse. They needs to address issues individually and fix them.
The Solution
We solved the above problems by setting up a Pritunl cluster which would be the central hub for routing all the VPN traffic. The solution we implemented reduced the VPN related issues by more than 90%. The main intention was to have a central control over the company wide VPN(s). So instead of each user setting their own VPN, we set that particular VPN once and just route users’ traffic through these machines. There are three major parts of this solution: Pritunl client, Pritunl Cluster and the VPN facilitators.
Pritunl Client
The pritunl client needs to be installed at user end. It is readily available for all platforms and OS. Users need to authenticate themselves with the Pritunl cluster and get a list of VPN profiles that they have given access to. Pritunl has various integrations for authentication like Google, Github, Auth0 etc. so only valid users can register and download their VPN profiles. How does the users get access to particular VPNs I will explain in the next section. These profiles may also have two-step authentication. Each user at the time of registration and every he logs in, is presented with a unique QR code which is the seed for the 2FA token. This seed or the QR code can be used in token generating apps like Google Authenticator, Authy etc. So there is a double check before a user tries to access a network behind VPN. Since the client talks to the Pritunl cluster, the load is distributed among the nodes in the cluster, so nodes are choked with the incoming traffic. The client automatically tries to switch to another node if the first node is not reachable or some network issue.
So on the users end it is pretty straightforward they just need to install the client and don’t have to worry about the nitty-gritty behind it. Here’s a sample screenshot of the pritunl client.
In the above screenshot, the us-west-production, us-west-development and us-east-production are the VPN profiles a user has access to.
Pritunl Cluster
Pritunl is designed for high availability with no master servers and no single point of failure. All servers in a Pritunl cluster are independent and do not rely on other servers. Pritunl is designed to keep the configuration simple. All hosts in a cluster are equal and there is no master server. Adding hosts to a cluster only requires connecting the new host to the database used by the other host/hosts. So in the above architecture diagram of ours simply adding a new server which talks to MongoDB will be automatically added to the cluster. We used three VMs to create clusters for HA (High Availability) as all these three VMs are placed in different AZs.
Pritunl provides a nice dashboard for managing the users, accessing the logs, creating new VPN profiles etc. Here we create different VPN profiles to access different services. Each profile has the routes or the CIDRs of the network behind the VPN. A profile has multiple tags and users are also added with the same tag, they get access to the VPN profile, which they can download by logging on the Pritunl dashboard. Pritunl dashboard lets you access the various and monitor the traffic on the hosts. One can also set up pritunl monitoring with an influx db. We had our own monitoring pipeline which we used to monitor the cluster.
Till now, we have just talked about routing the traffic of a network, but we haven’t yet set any VPN and forwarded this incoming traffic over this VPN.
VPN Facilitators
The red color VMs present in the diagram above are the VPN facilitators, meaning we actually set up VPNs on these machines. On these machines all the necessary VPNs are set up and over these the traffic will be routed to the destination. We had mostly two types of VPNs, one where we had to access third party services in their private DC, which means they provided us with their VPN set up process and second where we had to access our own private network deployed on clouds.
The third party VPNs are the ones where we don’t much control so we just do that once on a single machine. Earlier each user would have to set up this on his machine and debug the issues around it versus here it just needs to be done right once. We used nomad schedulers to run the VPN process reliably on these nodes. Most third party VPNs disconnect every few hours for safety and the clients need to re-connect. Using nomad we achieved that automation and ran these VPN services reliably.
For the second type of VPNs, since the services were deployed by itself, we had more control over it. We used the shuttle for this purpose. As it says, sshuttle is a transparent proxy server that works as a poor man’s VPN 🙂 Shuttle is easy to set up. Now from these machines the services are accessible but still there is disconnect between these VMs and the pritunl cluster. So we just need to add appropriate routes in the VPC router. For example, all traffic of 10.54.28.0/24 we routed by lets say the VPN1 VM. So whenever a user accesses any service from the above network, the traffic is routed from his machine to one of the Pritunl cluster nodes to the VPN1 node and then to the destination network.
All VPNs that are set up on these machines are monitored by our monitoring pipeline. If any of the process goes down we immediately get a PagerDuty.
Impact
- Easy and seamless to setup and connect to VPN.
- Reduce VPN related issues by 90%.
- Enahanced security with 2-Factor authentication.
- Ease to manage users, give access and revoke
- Abstract VPN setup from developers which enhanced their productivity.
- Audit logs and monitor for unsual user activities over VPN
