Yesterday we (and I imagine a significant portion of the rest of the world) were alerted to a vulnerability found in the OpenSSL software. OpenSSL is an integral piece of software as it is effectively involved in all encryption. When accessing a website with a certificate via HTTPS, OpenSSL is involved. When accessing cPanel via HTTPS? OpenSSL. SSH, SFTP, Secure IMAP, Secure POP3? Again, all OpenSSL. When an OpenSSL vulnerability is discovered it’s a really big deal. In this instance, the vulnerability allowed attackers to potentially obtain sensitive information from a server. This includes the risk of obtaining usernames and passwords, SQL queries and more.
Once alerted to the vulnerability we applied the system patches as soon as they were available (8th April). As we also use Litespeed, we applied the necessary updates as soon as they were released and we were notified these were available. I can confirm at the time of writing, all of our servers have had the necessary patches applied though we are currently in the process of completing reboots on some of our clients servers, and some of our lesser used infrastructure.
Why did you reboot today and not yesterday?
After the patches were applied yesterday, we restarted all of the services we believed to use OpenSSL so that they could use these new patches and no longer be vulnerable. We also ran several tests against the software on our servers to see if the vulnerabilities were detected. All of these tests came back negative (we were patched) so we saw no need to reboot.
Today we received additional information regarding patching and the vulnerability, and decided that to be completely safe we would reboot all of our servers. OpenSSL is tightly integrated into a lot of the services we run, and a lot of the software that runs these services. A reboot effectively ensures that all software is terminated and started again, making sure everything is now running from the newly patched OpenSSL and should leave no potential room for exploitation.
We do not know if this was 100% necessary, and were confident that the previous steps taken were adequate, however to protect our network and our customers we felt that the brief downtime was necessary to be completely sure. We are sorry for the inconvenience caused by the downtime, but hope you understand our reasons for doing this without prior notice and without warning. Again, to reiterate and confirm, we are now fully patched against this vulnerability.