I just thought I’d write a blog post outlining todays DNS issue, for clarification.
It seems that some (not all) DNS records in our cluster had been reset to cPanel defaults. cPanel defaults in this instance meaning a standard cPanel template, where the MX records default to the local server and with the most basic of cPanel A and CNAME records. As such, any modifications made from this default template were removed.
Why did this happen/Why were custom changes removed?
To prevent a fully corrupt record from occuring, cPanel appears to reset the DNS zone back to default on what it considers a critical error. This actually makes sense, because of the DNS record is corrupt to the point of not working, then no DNS would be resolved at all. Resetting it back to a template will at least obtain basic functionality.
A “critical error” in this instance does not mean user error, and we are in no way stating that these records were broken or corrupt due to end user modification. Unfortunately it’s impossible to tell exactly why these records were reset, because nothing is logged by cPanel in this instance and all other diagnostics are proving futile. We are in communication with them to try and determine the cause and to obtain information so we can try and prevent this in the future.
Custom changes were removed because when cPanel resets a DNS zone, it does so based on very basic information formed from the account. Typically, the IP address that’s allocated to the site on the server and any addon/subdomains that were setup on the account. Things like MX changes are not stored anywhere at all so cPanel cannot rebuild a record containing this information.
Why didn’t you e-mail me/tell me?
Firstly, this issue did not affect all of our customers, so a global mailout was deemed excessive. Secondly, we simply had no way of knowing which records were affected. From a server point of view, records appeared to be fine and standard. Additionally, when discovered that MX entries were also reset we decided that attempting to mail those with potentially affected MX entries would prove illogical. How would they receive the e-mail if the MX entries are pointing to the wrong place?
Some of you may have missing or incorrect A, CNAME, MX or other DNS records from your domains. We strongly recommend that you recreate these records from within cPanel or WHM as soon as you can in order to restore service and resolve problems with those.
Since clients have full control over their DNS, ThisWebHost staff will not be able to assist you with the recreation of this data.
What are you going to do to stop this from happening again?
We understand your frustrations, and we apologise for the inconvenience if you were one of those affected. However, this is a very very rare and isolated incident (something I and even others have never experienced before in our duration of the hosting industry) and we anticipate that we’ll never experience this again. That said, we are taking steps to implement additional backup functionality for DNS records so that if for whatever reason we need to, DNS data can be restored relatively easily on a global scale.
You suck/I hate you/I’m leaving for another host/Etc
Once again, we’re sorry for the inconvenience here. Problems do happen from time to time in hosting, and in our case we feel that whilst this incident was problematic, our track record and reputation speak for themselves as a quality hosting provider. Many of you have been with us for several years, and I honestly can’t recall very many (any?) situations that have globally and negatively impacted our customers. It would be a shame of you to consider this one incident your primary view and experience of our service.