September 9, 2024 - DNS Outage
September 13, 2024
On September 9th, our team was notified of an outage at approx. 11am ET. This interruption was a result of a DNS outage. The cause of this outage was identified to be a result of the transfer of ownership of the ButterCMS domain. As you may know, ButterCMS was recently acquired by Tiugo Technologies, and during the transfer of ownership, domain ownership was also transferred resulting in an unexpected outage.
The domain transfer process was initiated on August 25th. The domain has been successfully transferred to the AWS account but is not verified. Unfortunately, the domain verification email was sent from an address that is sometimes treated as spam; in our case, this email went to spam, and no one noticed it. Unfortunately, AWS Route53, after 15 days without verifying the email, suspends the domain. The email about the domain suspension was sent from another email and was noticed on September 9 at 2:42 PM UTC. After this step, we immediately verified the email and the domain was verified correctly. For several minutes, the domain was suspended, which resulted in the domain being unavailable in several regions of the world; not all locations were affected. On September 9 at 3:24 PM UTC we received confirmation that the domain was completely unsuspended.
We understand how important uninterrupted service is to your business and are pleased to share that the issue was quickly identified and resolved. As part of the post-mortem analysis, we have concluded and are introducing mechanisms to protect against such a mistake. One of the action points is the improvement of incident reporting processes using our status page.
Since the outage did not affect all regions, our monitoring unfortunately did not record system unavailability (because it checked the system from an insufficient number of regions). In the future, we will expand this and improve our monitoring system.
We apologize for any inconvenience this may have caused and appreciate your continued support. If you have any questions or need further information, our support team is here to assist.