Low cost SAN with cluster to achieve Failover
I have recently been window shopping for some networking stuff on Amazon. Amazingly, 10GbE NICs and switches of name brands and Cat7 cables are getting cheaper than ever. I started to wonder if I could build a mail server on iSCSI SAN and cluster to make it HA-capable? Since my concept is not totally automated, perhaps I should call it MA (medium availability) rather than HA (high availability)? LOL…Name Brand Solution
For the past two decades or so, mail server solution on HA cluster must have definitely been available from big names, such as Microsoft Exchange Server, IBM Lotus Domino, Oracle Veritas Cluster Server/Messaging Server and etc. But, the implementation cost of such solutions is rocket-high, like 10 grand, 20 grand or more. They all come with top hardware specs, server license, cluster license, cluster node license, mail server connection license, directory service connection license, tech support tickets, man labor and blah blah.Amazon Solution
In 2019, prices on SSD and 10GbE parts are dropping and getting more accessible than ever. DIY iSCSI SAN and HA cluster implemntation makes this solution no longer for enterprise only. My DIY idea comes from equipping mail server with more advanced but less expensive NAS gadgets as well as lower cost PCs for cluster purpose.Theory
According to 10GbE spec sheet, the 10Gb/s transmitting speed is roughly 1250MB/s, which is 1.25GB/s. For SATA-3, it is about 6Gb/s and that turns out to be 750MB/s. |
| Dual-port 10GbE NIC |
Next, we will discuss about software and hardware spec that compose the iSCSI SAN & HA Cluster infrastructure.
Network Speed
In the past, 1GbE NIC used to be good enough upgrading from 100MbE NIC. At that time, perhaps only data center could afford 10GbE equipments. Nowadays, these gadgets have become so affordable for regular users. To achieve 10GbE transmission speed, it requires SFP+ (copper or fiber) or RJ-45 (Cat6 or Cat7 cable) for cabling between Cluster Server and NAS.SSD storage
Just about one year ago, a 1TB SSD used to be over USD$400. Recently, 1TB SATA-3 SSD is running about USD$150 and it is dirt cheap comparing what it used to cost. For my personal brand preference, I use either Micron or Intel: |
| Crucial MX500 2TB |
- Crucial/Micron MX500 2TB: USD$289 / 1TB: USD$134
- Intel 545S 1TB: USD$222
Synology NAS
Due do gradual price drop on SSD drives over time, building a RAID-5 or -6 based on 4 Micron 2TB or 1TB SSD is a great news for IT staff because users will less likely complain about poor disk performance. For non-IT people, NAS is an easier solution. When it comes to Synology NAS, I think DS1517+ is not a bad choice. Also, to achieve my design of iSCSI NAS plus HA Cluster Mail Server, DS1517+'s hot-swappable HDD capability and expandability to 10GbE E10G18-T2 (dual-port high-speed 10GBASET add-in card) are must-have. By the way, upgrading memory module to 16GB for the best performance is a great plus. |
| Synology DS1517+ |
Cluster PC
- Hardware: Buy two computers with specs of Intel i7, 16GB RAM, and one 120GB SSD. Equip each PC with 10GbE NIC for connections with NAS. The built-in 1GbE NIC is used to connect to switch or router. Then, do we still need additional NIC for cluster heartbeat? We will discuss about it later in this article.
- Software: Install Windows 10 and EVO Mail Server on each of PC described above. Then, do we need to deploy VMWare Workstation to virtualize the mail services? Also, we will discuss about it later in this article.
Cluster Heartbeat Myth
Generally, a separate NIC will be installed on each member PC of a cluster group for heartbeat purpose. Windows Server 2008, 2012 and 2016 and Linux Server have this built-in feature of hearbeat. But, there is no such feature on Windows 10, which is a part of my design for money saving. I also googled it but found no answer. If you happen to know any such tool software available for Windows 10, please let me know and thanks in advance.The purpose of hearbeat feature works along with quorum disk to automatically determine whether the backup server should take over the primary server or not. I think such automated decision makings should be used on mission-critical servers, such as web servers or database servers.
For mail server, from another point of view, I do not think mail server operation is that mission-critical. With a bit higher fault-tolerance, IT staff can take a a few minutes to diagnose what went wrong and then judge if backup server should take over the primary server. Factors on decision-making:
- Resolving issues on primary server and continue?
- Immediate action of taking over primary server?
- Hardware: mainboard, memory module, harddisk, power, NIC, network cable – If parts replacement is not readily available, then switch to backup server asap.
- Software: Bugs on OS or mail server software – If opearting system is breaking down and cannot boot up, then switch to backup server asap. If a bug can be fixed within 10 minutes or so, such as a version update or parameter adjustment, then there is no need to switch.
Necessary to deploy VMWare?
Whether or not deploying VMWare is a debatable issue. I think if there is sufficient fund, then it is always better to prepare additional contingency plan other than a cluster backup server by installing VMWare Workstation on the primary server. |
| VMWare Workstation 15 |
- On the primary server, install a virtual Windows 10 under VMWare Workstation. Let’s call it VM A.
- Once VM A runs mail service properly, shut it down.
- Install another 120GB SSD on primary server.
- Copy VM A image file to the new SSD and make it VM B.
- Let’s assume that the primary server hardware is working. Under the circumstance that VM A image is broken, we can immediately start VM B to continue mail services.
Backup server taking over primary server
In the case of deploying EVO Mail Server, since all data of mail, certificate and configuration and etc are all located on NAS, the role of cluster PCs is solely for running mail server services, accessing NAS data, and communicating with LAN/WAN. Each IT staff has his or her own preference on switching from primary server to backup server. I list some of the ways below for you to make decision on which one fits your need:- If servers are inside LAN:
- Log in router and change the LAN IP under Port Forwarding page (some router calls it Virtual Server) to backup server's internal IP.
- If you always bring primary server offline first before throwing backup server online, then we can disconnect backup server from the network by turning off its NIC while standing by. When the switch-over is about to take place, we turn backup server’s NIC back on. In this scenario, backup server’s LAN IP can be configured as the same as primary server’s. The downside of it is that we cannot perform the swtich-over remotely and have to do it by actually logging in the backup server.
- For the above method, there is a workaround. While backup server is standing by, its NIC is enabled and assigned with an separate internal IP other than the primary server’s IP to avoid conflict. Before switching-over, disconnect primary server from the LAN first. Then, log in the backup server and change its LAN IP to primary server’s IP.
- If the primary server is equipped with VMWare and if server’s hardware is working normally but VM A image is corrupted, launch VM B image to take over mail services. (Note: VM A and VM B have the same public IP)
- If servers are in DMZ or directly connect to Internet:
- Log in DNS hosting service or your own DNS server first. Then, modify IP of mail server host name’s (literally MX record) A record to backup server’s public IP. Such DNS record modification will usually take mintues to a few hours to kick in.
- If the primary server is equipped with VMWare and if server’s hardware are working normally but VM A image is corrupted, run VM B image to take over mail services. (Note: VM A and VM B have the same public IP)
- Before switching-over, if the primary server is still accessible, simply log in the server and de-activate EVO Mail Server license key first.
- Before switching-over, if the primary server already inaccessible, use other email account to contact EVO staff and let them manually de-activate license key. Then, activate the license key again on backup server.
- To make switch-over quicker, purchase two EVO Mail Server licenses, one for primary and one for backup, and activate them separately. Once backup server is active, copy its license.dat file to the license key folder location on NAS and overwrite the existing one. Then, start mail service.
- If VM images solution is deployed, there is no need to de-activate and re-activate since both VM A and VM B are actually identical. However, make sure not to boot up both images at the same time. Otherwise, both Windows license and mail server license will conflict each other.
- The last one is even easier. Install identicaal hot-swappable drive bays on both servers. If any hardware parts fails but the 120GB OS SSD. Then, simply take out drawer containing the OS SSD. Stuff it into backup server's drive bay and boot up immediately. However, it is recommended to prepare a spare ready-to-boot 120GB OS SSD aside.
Build NAS, iSCSI SAN and Cluster PC together
Let’s assume EVO Mail Server is operating normally. For installing, configuring and testing EVO Mail Server, please refer to this article.- On Synology NAS, create iSCSI storage:
- If NAS storage volume has not been created, follow instruction here. If already created, go to next step.
- Now, build TARGET and build LUN. If they have been configured, then skip this part or make slight modification if necessary.
- Open iSCSI Manager.
Synology iSCSI Manager - Let’s start from scratch.
iSCSI overview - Click on Target to the left and click on Create.
Create target - Enter name and IQN as well as making decision to enable CHAP or not.
Create target setting - The next step is mapping LUN. Here, we will click on Create a new iSCSI LUN.
Create new LUN mapping - Define capacity of iSCSI LUN.
Determine volume size - Confirm settings and click on Apply to save settings.
Target confirm settings - Click on Target to the left again to double-check settings.
Target summary - Click on LUN to the left again to confirm if target mapping is connected.
LUN summary - Click on Settings to the left to increase I/O Queue Depth value. Setting a greater value will improve 10GbE network connectivity and SSD performance.
I/O Queue Depth Value - Finally, click on Overview to confirm whether iSCSI has already been properly configured.
- Enable iSCSI initiator in Windows 10.
- There are a few articles that you can read through on what can be done for iSCSI optimzation on Windows end.
- How to optimize your iSCSI service for Windows
- How to use iSCSI Targets on Windows computers with Multiple Connections per Session (MC/S) support
- How to Use iSCSI Targets on Windows Computers with Multipath I/O
- Next, search iSCSI initiator and open it in Windows 10.
Search for iSCSI initiator - In iSCSI initiator window, under Targets tab, enter NAS IP and click on Quick Connect… button to initiate a quick search and connection.
Enter iSCSI target IP - In Quick Connect window, available target is listed and the state is connected.
Quick Connect window - Upon closing Quick Connect window, Discovered targets will list the connected NAS as well.
Discovered target - Search Computer Management and open it in Windows 10.
Search for Computer Management - Click on Storage/Disk Management and the newly created disk is already listed.
Convert to GPT - In the screenshot above, mouse right-click on Disk 2 area and click on Convert to GPT Disk. Why GPT? GPT enables creating partition beyond 2.2TB.
- Next, mouse right-click on area of newly created partition to continue creating volume and formatting it.
New Simple Volume Wizard - Check the formatted disk.
New Simple Volume Wizard - Move mail server related data to iSCSI disk.
- Open EVO Mail Server GUI. Click on Status page and scroll down to Server setting status.
EVO Mail Server Server setting status - In the screenshot above, click on arrow loop symbol to the right of Mail Data Folder. There will open a window browsing folder tree.
EVO Mail Server Server setting status - Once a folder is selected, click on OK. There will be a Reminder popping up and inform actions being taking place.
EVO Mail Server Server setting status - Once data migration is complete, another Reminder will pop up and inform actions taken place.
Mail Data Relocation Summary - At the moment, Mail Data Folder is displaying new folder location.
Mail folder location relocated - Finally, perform testing routines described in the setting up mail server article I mentioned previously.
Price breakdown
Nowadays, running a company with 100 to 200 employees is never an easy task. IT infrastructure investment ought to be carefully assessed. Here is a rough price quote for what it costs to build this solution:- Intel i7, 16GB RAM, 120GB SSD, 10GbE NIC (options are fiber and RJ-45), 450W power supply, Windows 10 computer x 2, USD$2000
- Synology NAS DS1517+ x 1, USD$1000
- Synology DDR3 Memory Module, USD$300
- Synology E10G18-T2 or E10G17-F2 NIC x 2, USD$270 or USD$257
- Micron MX500 2TB SSD x 4, USD$1200
- WD 4TB HD (Mail Archive HD) External (if 5th drive bay is used for SSD cache) or Internal (if using 5th drive bay) x 1, USD$99 or USD$169
- VMWare Workstation 15 x 1, USD$249 (optional if VMWare VM A and VM B solution is deployed)
- NIC: If possible, use fiber NIC. You will be rewarded with much better performance.
- NAS: Other than DS1517+, DS1817+ (8GB RAM version, USD$1400) is a good choice. It has two 10GbE ports built-in. Yet, two drive bays can be used for building SSD cache while there are still six drive bays for constructing RAID-5 or -6. Any idea on sizes for SSD cache drives? Consult your SSD dealer or check out this article. Though DS1817+ is USD$200 more expensive than DS1517+, RAM modules and 10Gbe NIC are taken care of. The overall expenses on NAS is USD$200 less at least. Nevertheless, hardware spec on NAS varies from one to another. It is up to you to decide.
- NAS SSD: Why I strongly recommend on getting 2TB SSDs? Based on technical aspect, the bigger capacity the faster speed it is. Hence, if I were you, I would get 2TB SSDs once for all and it will definitely last longer.
Conclusion
It is a long article and it took me a long time to wrap it up. I just happened to write this article because I thought that we can benefit from huge price drop on both 10GbE NIC and TB-grade SSD. There are definitely other ways of implementation giving even faster performance, such as stuffing several PCI-e and NVME SSDs directly into mail server machine itself.But, my main focus this time is providing a low-budget solution that offer fault tolerance for mail server. For exqmple, NAS running on RAID-5 or -6 provide fault-tolerance for storage while cluster deployment provides fault-tolerance for hardware failure.
At last, this article is for your reference when you happen to build a fault-tolerance mail server. However, each company has different IT environment. Perhaps you can survey on all left-over IT assets that are sitting around first. Then, purchase what is missing. Eventually, you will save even more for yourself or for your boss. Hope your server keep running and lasting. Yet, you can have sweet dreams every night with no emergency call from IT colleagues.
