Regarding Today’s DNS Issue

POSTED BY Doug ON July 14, 2011 AT 7:03 pm UNDER announcements
closeThis post was published 1 year 10 months 12 days ago which means the content may no longer be applicable or relevant to the service we offer today. If in doubt, please contact us.

I just thought I’d write a blog post outlining todays DNS issue, for clarification.

What Happened?
It seems that some (not all) DNS records in our cluster had been reset to cPanel defaults. cPanel defaults in this instance meaning a standard cPanel template, where the MX records default to the local server and with the most basic of cPanel A and CNAME records. As such, any modifications made from this default template were removed.

Why did this happen/Why were custom changes removed?
To prevent a fully corrupt record from occuring, cPanel appears to reset the DNS zone back to default on what it considers a critical error. This actually makes sense, because of the DNS record is corrupt to the point of not working, then no DNS would be resolved at all. Resetting it back to a template will at least obtain basic functionality.

A “critical error” in this instance does not mean user error, and we are in no way stating that these records were broken or corrupt due to end user modification. Unfortunately it’s impossible to tell exactly why these records were reset, because nothing is logged by cPanel in this instance and all other diagnostics are proving futile. We are in communication with them to try and determine the cause and to obtain information so we can try and prevent this in the future.

Custom changes were removed because when cPanel resets a DNS zone, it does so based on very basic information formed from the account. Typically, the IP address that’s allocated to the site on the server and any addon/subdomains that were setup on the account. Things like MX changes are not stored anywhere at all so cPanel cannot rebuild a record containing this information.

Why didn’t you e-mail me/tell me?
Firstly, this issue did not affect all of our customers, so a global mailout was deemed excessive. Secondly, we simply had no way of knowing which records were affected. From a server point of view, records appeared to be fine and standard. Additionally, when discovered that MX entries were also reset we decided that attempting to mail those with potentially affected MX entries would prove illogical. How would they receive the e-mail if the MX entries are pointing to the wrong place?

What now?
Some of you may have missing or incorrect A, CNAME, MX or other DNS records from your domains. We strongly recommend that you recreate these records from within cPanel or WHM as soon as you can in order to restore service and resolve problems with those.

Since clients have full control over their DNS, ThisWebHost staff will not be able to assist you with the recreation of this data.

What are you going to do to stop this from happening again?
We understand your frustrations, and we apologise for the inconvenience if you were one of those affected. However, this is a very very rare and isolated incident (something I and even others have never experienced before in our duration of the hosting industry) and we anticipate that we’ll never experience this again. That said, we are taking steps to implement additional backup functionality for DNS records so that if for whatever reason we need to, DNS data can be restored relatively easily on a global scale.

You suck/I hate you/I’m leaving for another host/Etc
Once again, we’re sorry for the inconvenience here. Problems do happen from time to time in hosting, and in our case we feel that whilst this incident was problematic, our track record and reputation speak for themselves as a quality hosting provider. Many of you have been with us for several years, and I honestly can’t recall very many (any?) situations that have globally and negatively impacted our customers. It would be a shame of you to consider this one incident your primary view and experience of our service.

19 Responses to “Regarding Today’s DNS Issue”

  • July 14th 2011 07:05:49
     Reply 

    Ahh so thats why my Google mail wasnt working! Thanks for the heads-up. Got it sorted anyway.

    P.S. I liked you old site design more.

  • July 14th 2011 07:41:19
     Reply 

    Seriously guys not what I would call a proffessional response to the problem, and to be refered to this blog post by way of an explanation is also pretty poor especially as it was 15 plus hours after the event. I also had difficulties contacting you as my password was not working, and without the password you cannot submit a support ticket, and if you are not receiving emails you cannot reset your password. GET A PHONE NUMBER ON THE SITE!

  • July 14th 2011 07:52:29
     Reply 

    @David: How is this not a professional response to the problem? What is the issue with being referred to this blog post, exactly?

    You had an issue opening a ticket because you told us that your web developer was the one who had all of the passwords, not yourself. They should’ve left you with adequate login credentials to contact us if needed. We do not provide telephone communication, please see our FAQ’s.

  • July 14th 2011 09:16:31
     Reply 

    This has been really tough for us. Our email has been down for an entire day without explanation and I just now was able to figure out the problem and update the DNS settings (which will now take up to 24 hours to propagate). I’m sure you can understand the frustration of having your email down for over 24 hours.

  • July 14th 2011 09:22:50
     Reply 

    @Colby: If you’d submitted a ticket, we would’ve informed you what was happening and would’ve gotten your e-mail up and running as soon as possible. The same of course applies to everyone else.

    The purpose of this blog post is not to provide initial communication of the problem, it’s to provide a summary for those who were already affected and knew about it, and to inform those who may not be aware of any issues that they should check their DNS accordingly. Just to be clear; anyone who submitted a ticket today about this issue was informed there was a DNS problem and that they would need to fix their records. This is not our first communication of the issue.

  • July 14th 2011 10:13:28
     Reply 

    Let’s assume you had sent out a notification e-mail. A customer who was not affected would have spent 5 minutes checking his/her account, realized there were no problems, and would’ve thought “Phew, I’m unaffected, even though I wasted some time it’s good that This* took the time to keep me updated”. A customer who /was/ affected would have been able to re-modify his/her DNS settings and suffered a minor inconvenience with minimized downtime. Neither situation seems bad enough to justify not sending out an e-mail. Most affected people are probably annoyed with the issue, but will understand that problems do happen. I would wager most of them are much more disappointed by the lack of e-mail notification than the issue itself. Your affected-MX justification for not sending one out seems disingenuous; certainly most of your clients have a contact e-mail address on file that isn’t routed through This*.

    I don’t receive a lot of e-mail to my domains, and so when I received no mail to my Google Apps e-mail I wasn’t alarmed. In fact, I probably would’ve gone several days before noticing. The only reason I checked my MX records and caught the problem was because I follow this blog in my RSS reader. I can imagine you have other clients in a similar situation. The correct thing to do would’ve been to send out a notification e-mail. Even though now it would be late, it should still be sent out.

  • July 14th 2011 10:20:00
     Reply 

    @Raymond: Problems do indeed happen, and given our track record and the situation I’m confident that this is a rarity.

    Regarding the mailouts, with respect I strongly disagree. We have a good idea on the number of clients who use us for e-mail and you would be surprised how many route all of our communications directly through their main domains with us. Given this information we know that those affected would likely not receive the mailout and not know of the situation. This is why we actively use Twitter and our blog to communicate these significant issues and have also put a notification without the client area.

  • July 14th 2011 10:42:31
     Reply 

    Even accepting the possibility some clients wouldn’t receive the e-mail, there isn’t a good reason to not send it out anyway. I have a gmail address as my primary contact, and accordingly would have: 1. received such an e-mail; 2. been able to correct the problem with shorter downtime as I wouldn’t have had to wait for this blog post to appear; 3. had a generally better feeling about the whole situation.

    We receive scheduled downtime notices, invoices, etc. through e-mail; there’s no reason something of this magnitude shouldn’t receive an e-mail notification also. You are well within your rights to stand by your decision not to send it, but if you were to poll your clients, I’m positive most would have wanted you to at least try.

  • July 14th 2011 10:45:23
     Reply 

    @Raymond: We’re going to have to agree to disagree.

    You’re assuming most people utilise a third party e-mail address, when the reality of the situation is sadly much different. For any global issues that affect clients we would normally send a mailout, and have done in the past (see our recent ns2 IP address update for an example). In this instance since some e-mail was affected, it was deemed (and quite correctly in my opinion) that this would not be the most reliable method of communication.

  • Heinrich
    July 15th 2011 04:41:11
     Reply 

    Not cool guys but like you said, given your track record so far I can let it go. I’ve now got to try and remember the MX record settings for my domain :/

  • July 15th 2011 11:44:47
     Reply 

    Re your ‘What now?’ paragraph…I’m a photographer and not a web specialist and since the issue was of your doing surely you could offer help to rebuild this ‘missing or incorrect A, CNAME, MX or other DNS records from your domains’? I wouldn’t know how to accomplish that and wouldn’t know if I’ve been affected by this notification.

  • July 15th 2011 01:24:43
     Reply 

    and….? seriously, things like this shouldnt be happening, at least i would like to recieve an email explaining this… i just had to figure out by myself why i had 0 emails this morning…

  • July 15th 2011 01:32:12
     Reply 

    I agree with Luis. To hang everyone out to dry without backups is pretty irresponsible; especially without any type of notification that it happened. I had a client emailing me saying they weren’t receiving emails, and I don’t even offer support like this — but I felt obligated to fix it, since I set it up.

    Also to offer no assistance is also irresponsible. Your system broke, so you should help fix it.

  • July 15th 2011 03:31:52
     Reply 

    We didn’t “break” anything, and we are not responsible for cPanel resetting DNS records.

  • Ryan Scherf
    July 15th 2011 04:45:19
     Reply 

    Is cPanel not our your system? Is this something I set up on my own when I created my account? Do I have a pre-installed alternative?

    Maybe you can’t prevent issues with cPanel, but to take no responsibility for it is ridiculous. “We don’t know what happened” isn’t a good enough answer for people whose businesses were interrupted because of this.

    Without a single apologetic word in your post (except for to those you think will leave your service), I find it hard to believe you actually care about your customers.

  • July 15th 2011 05:09:17
     Reply 

    @Ryan: cPanel is on our systems yes, but you’re accusing us of causing this issue when we did nothing of the sort. We cannot take responsibility for cPanel, much in the same line of we cannot take responsibility for products such as WordPress, etc. If WordPress were to cause corruption of your website, would that be our fault?

    I understand some users are angry and frustrated, but lashing out at us for something that is beyond our control is not productive. Yes, this is a cPanel issue, yes this is something we’re looking into it. No, we do not know what caused it. Would you rather we lied about the cause, just so you would have some finality to the situation? Sorry, but we’re not in the habit of lying to our customers.

    “We understand your frustrations, and we apologise for the inconvenience if you were one of those affected.” is that not apologetic? Of course we care about our customers, but lashing out and demanding we do the impossible is not going to get us anywhere.

  • Hector
    July 16th 2011 03:30:20
     Reply 

    My website went down on the 14th of July but I have no idea how to get it back up again? My other domains hosted on the same WHM reseller that I have still work, and my nameserver setting is ns1.thiswebhost.com ? Dissapointing that the way I found out about this was checking my google analytics after two days and seeing traffic fall to zero..

  • Hector
    July 16th 2011 03:53:06
     Reply 

    As well as this, support tickets that I have been making are never actually created. What I thought was just lousy support was actually a broken system.. not sure which is worse..

  • July 16th 2011 04:46:38
     Reply 

    @Hector: Our support ticket system works fine. Please ensure you read the pages carefully when submitting a ticket.