Friday 26th November at 23.00 CET the Mobypicture.com, Moby.to and 147 other domains Mathys van Abbe registered got suspended by Sitelutions, the company we’ve been using for many years for domain name registration and DNS hosting.
A Mobypicture user has supposedly uploaded a copyrighted photo on our website. The company owning the rights of this picture has made a complaint and sent us a DMCA notice through Sitelutions.
Sitelutions received this complaint and undertook some steps. (facts)
- They tried to contact us via email, but unfortunately that email never reached us.
- Sitelutions sent us a reminder on the same email address.
- When we didn’t reply to the request and the deadline to delete the copyrighted material passed, they decided to suspend ALL the domains related to the account which also registered Mobypicture.com. (other clients, companies and projects), instead of only the mobypicture.com domain.
We contacted Sitelutions via email and phone within minutes after we encountered the suspension.
Sitelutions replied pretty fast by mail with a copy of the previously sent request to delete the copyrighted material, a notice that they had enabled our account again and that we had a window of twelve hours to delete the copyrighted material.
Sitelutions had changed the DNS records for all domains and subdomains to 127.0.0.1, to make sure no domain was reachable, during the suspension. For some reason though Sitelutions had changed the Time To Live for the 127.0.0.1 records on ALL the domains to 24 hours. This means that when Sitelutions enabled the account again, it took up to 24 hours before all DNS servers were sending out the correct IP addresses instead of the 127.0.0.1 address, when someone wanted to visit mobypicture.com, moby.to or on of our other domains. Normally we set the TTL somewhere between 10 minutes and an hour, to make sure changes are propagated quickly to all clients. If Sitelutions had set a low TTL on their temporary A records as well, it could have all been solved in 30 minutes, it was their TTL of 24 hours that caused the severe downtime.
This all resulted in a lot of work, downtime/irreliable service for our 500.000 users and damages for our customers and the 900 applications built on our API.
Today we started to investigated how this could have happened and found out the following disturbing things:
- Sitelutions didn’t NOT try to contact the domain holder email (whois lookup) but a different one.
- Sitelutions did not try to reach us in any other way, Mathys’ phone number is also in the whois (please don’t tell anybody…), so they could have given us a call.
- Sitelutions changed the TTL to 24 hours, the TTL was set to 10 minutes by us originally.
This leaves us with a couple of questions:
- Why didn’t they try to reach us using the right contact information?
- Why did they suspend 147 different domains as well?
- Why did Sitelutions change the TTL to 24 hours?
We sent this blogpost to Sitelutions prior to publication and got some emails back explaining the situation. They told us that they will try to reach their customers in more ways, before suspending, but that’s too late for us. Suspending our whole account instead of only the mobypicture.com domain is a technical limitation in their backend and not legally obligatory. It is still unknown to us why they set a TTL of 24 hours on temporary DNS records to disable domain names.
There are several things we undertook to prevent this in the future.
First we make sure a DMCA notice sent to any email address used by Mobypicture arrives at the right person, so they can be dealt with swift and correctly.
Second, we moved away from Sitelutions as a DNS provider. We suffered almost 12 hours of downtime when Sitelutions was hit by a DDOS a week earlier – without any communication from Sitelutions to their customers -, so we were already in the process of moving. We have moved to Dynect, one of the major Enterprise DNS providers with over 16 locations world-wide to lower the latency and better withstand attacks – among other advanced services. They contacted me when I was looking for an alternative during the DDOS last week and have been helpful ever since.
Third, we are moving the registration of our .com domain names to a Dutch registrar, for directer ways of communication.
We apologize for all the trouble this caused. We already made sure there were as less single-points-of-failure as possible in our architecture, but did not suspect DNS would be the cause of such severe downtime twice in 2 weeks. We will make sure this won’t happen again.