The Moby troubles of Friday 26 November. This is what happened.

Friday 26th November at 23.00 CET the Mobypicture.com, Moby.to and 147 other domains Mathys van Abbe registered got suspended by Sitelutions, the company we’ve been using for many years for domain name registration and DNS hosting.

A Mobypicture user has supposedly uploaded a copyrighted photo on our website. The company owning the rights of this picture has made a complaint and sent us a DMCA notice through Sitelutions.

Sitelutions received this complaint and undertook some steps. (facts)

  1. They tried to contact us via email, but unfortunately that email never reached us.
  2. Sitelutions sent us a reminder on the same email address.
  3. When we didn’t reply to the request and the deadline to delete the copyrighted material passed, they decided to suspend ALL the domains related to the account which also registered Mobypicture.com. (other clients, companies and projects), instead of only the mobypicture.com domain.

We contacted Sitelutions via email and phone within minutes after we encountered the suspension.

Sitelutions replied pretty fast by mail with a copy of the previously sent request to delete the copyrighted material, a notice that they had enabled our account again and that we had a window of twelve hours to delete the copyrighted material.

Sitelutions had changed the DNS records for all domains and subdomains to 127.0.0.1, to make sure no domain was reachable, during the suspension. For some reason though Sitelutions had changed the Time To Live for the 127.0.0.1 records on ALL the domains to 24 hours. This means that when Sitelutions enabled the account again, it took up to 24 hours before all DNS servers were sending out the correct IP addresses instead of the 127.0.0.1 address, when someone wanted to visit mobypicture.com, moby.to or on of our other domains. Normally we set the TTL somewhere between 10 minutes and an hour, to make sure changes are propagated quickly to all clients. If Sitelutions had set a low TTL on their temporary A records as well, it could have all been solved in 30 minutes, it was their TTL of 24 hours that caused the severe downtime.

This all resulted in a lot of work, downtime/irreliable service for our 500.000 users and damages for our customers and the 900 applications built on our API.

Today we started to investigated how this could have happened and found out the following disturbing things:

  1. Sitelutions didn’t NOT try to contact the domain holder email (whois lookup) but a different one.
  2. Sitelutions did not try to reach us in any other way, Mathys’ phone number is also in the whois (please don’t tell anybody…), so they could have given us a call.
  3. Sitelutions changed the TTL to 24 hours, the TTL was set to 10 minutes by us originally.

This leaves us with a couple of questions:

  1. Why didn’t they try to reach us using the right contact information?
  2. Why did they suspend 147 different domains as well?
  3. Why did Sitelutions change the TTL to 24 hours?

We sent this blogpost to Sitelutions prior to publication and got some emails back explaining the situation. They told us that they will try to reach their customers in more ways, before suspending, but that’s too late for us. Suspending our whole account instead of only the mobypicture.com domain is a technical limitation in their backend and not legally obligatory. It is still unknown to us why they set a TTL of 24 hours on temporary DNS records to disable domain names.

There are several things we undertook to prevent this in the future.

First we make sure a DMCA notice sent to any email address used by Mobypicture arrives at the right person, so they can be dealt with swift and correctly.

Second, we moved away from Sitelutions as a DNS provider. We suffered almost 12 hours of downtime when Sitelutions was hit by a DDOS a week earlier – without any communication from Sitelutions to their customers -, so we were already in the process of moving. We have moved to Dynect, one of the major Enterprise DNS providers with over 16 locations world-wide to lower the latency and better withstand attacks – among other advanced services. They contacted me when I was looking for an alternative during the DDOS last week and have been helpful ever since.

Third, we are moving the registration of our .com domain names to a Dutch registrar, for directer ways of communication.

We apologize for all the trouble this caused. We already made sure there were as less single-points-of-failure as possible in our architecture, but did not suspect DNS would be the cause of such severe downtime twice in 2 weeks. We will make sure this won’t happen again.

13 thoughts on “The Moby troubles of Friday 26 November. This is what happened.

  1. Good explanation, but wondering if you are working on free upload of possible copyrighted material you could have been thinking of possible DMCA Notices and still there isn’t a possibility yet to ‘Report abuse’ at every single image?

    Would have helped the owner of the brand i suppose? Just a thought..

  2. @Gerben, we are working very hard on completing what we call ‘MobyNext’, a complete new version of Mobypicture. It has among other things a report abuse button on every page.

    It was already on our mind, but hadn’t had the time to implement in the current version and moved it to our new major release

  3. I would setup 3 VPS DNS machines, I would control.
    US Europe Asia.
    Like http://whois.domaintools.com/wikipedia.com Wikipedia.
    No 3rd party to kill my business.
    All the big-dogs do this except Amazon.
    Local player Hyves controls it also themselves.
    ISP’s cache DNS queries so load is low, and global stuff is done by them. They even override your TTL (Belgacom).

    Good luck @Kjeld

  4. I really think it’s a matter of personal preference. Facebook is running its own DNS, Twitter isn’t. Microsoft is, Amazon isn’t. Hyves is, Relatieplanet isn’t.

    We prefer to outsource a lot of stuff like Storage, CDN, DNS and our hardware because we do not have the capacity or knowledge to build these systems at scale, with low cost and high reliability. We rather focus on the things we ARE good at. That gives us a head start and a more reliable and better operating service.

  5. Pingback: Cloud based services and Platform as a service (PaaS) | Void Station

  6. Pingback: Jouw internetprovider als rechter en beul « Bits of Freedom

  7. I have been having trouble getting a reply from Moby on a DMCA that I sent to the site twice since June 19th, 2013. How can I get in contact with someone to get a photo removed?

Leave a Reply