Engineers have been alerted to delays in notification processing. We are currently investigating and will follow-up with further details.

At 9:37 AM EDT an incident occurred which delayed incident notification processing. The incident was fully resolved by 2:30 PM EDT. More details on the root cause will follow.  Subscribers would still see status pages as expected, but incident notification would have been delayed.

StatusCast engineers noted a background processing degradation that caused delays in some actions from being completed in a timely fashion. 

 This has been resolved. We apologize for any inconvenience.

StatusCast engineers identified a backup in its background processing that is causing delays in some actions from being completed in a timely fashion. 

Engineers have determined the root cause of the backup in processing. During this time we have scaled out the service that is responsible to help clear out the remaining items faster.

Services should be operating as expected.

In keeping with our commitment to security and transparency we wanted to inform you that our provider for SMS, Twilio, has alerted our security team to an incident that occurred. Below is the statement we received from Twilio which gives a detailed account of the incident and what has been done:

—--------------------------------------

Twilio has been notified by one of our backup carriers, iBasis, that IdentifyMobile inadvertently exposed certain SMS-related data sent by iBasis publicly on the internet that included personal data. We are writing to inform you that some of your personal data and non-personal data (such as data related to marketing campaigns) was accessed by a security research group while it was publicly exposed by IdentifyMobile.

Although this incident was outside Twilio’s control, we take it very seriously and are committed to helping you understand the full impact. 

What do you need to know? 

In order to deliver messages in specific regions, Twilio relies on numerous carriers to maximize deliverability to their final destinations. Twilio was notified that iBasis (a Twilio backup carrier) had used IdentifyMobile (iBasis's further backup carrier) who inadvertently enabled public access on an AWS S3 Bucket during development work. Information contained in this bucket was made public from May 10-15, 2024, and accessed between May 13-14, 2024. Based on a joint investigation between IdentifyMobile and Amazon AWS, we learned that a portion of this data was accessed by the Chaos Computing Club (CCC). CCC is a security research group that identifies security issues; CCC has confirmed that they are not holding any data downloaded from the AWS S3 Bucket. We do not have evidence that allows us to confirm that no other third party accessed the data. 

Twilio does not own this bucket, and none of our systems have been compromised in connection with this data exposure. This incident was the result of actions taken by IdentifyMobile and outside of Twilio’s control.

While we continue collaborating with these carriers to bring you the most accurate information regarding this exposure, the portion of data exposed by IdentifyMobile related to SMS sent between January 1, 2024 and May 15, 2024, and included: 

• Mobile number 

• SMS message content 

• SMS Sender ID 

• SMS Timestamp

What have we done so far?

• Twilio initiated our incident response process to rapidly investigate this matter.

• Twilio escalated this issue to the iBasis executive team; subsequently, we’ve done an analysis on the data logs that were compromised to provide you with as much information as possible.

• Out of an abundance of caution, we have ceased sending traffic to iBasis where possible. iBasis informed Twilio that it has stopped routing with IdentifyMobile.

• We will continue working with our 3rd party carriers to get you any additional details that may arise from this incident.

What do you need to do?

We recommend reviewing the SMS traffic you sent between January 1, 2024 and May 15, 2024, discussing the implications of an exposure with your internal team(s) and deciding if you need to engage with impacted individuals. If you need additional information regarding this incident, we are here to support you throughout this situation.

We deeply regret any inconvenience this may cause and appreciate your understanding and cooperation.

—--------------------------------------

What has StatusCast done?

Once StautsCast’s security team was alerted an audit was performed to try and determine the potential impact this had on our service. Based on Twilio’s report it is possible that some notifications sent through StatusCast’s service would have been sent using the iBasis carrier. Based on the report data would have included the recipient's phone number as well as the SMS message. SMS notifications in StatusCast are designed to be brief and to drive users to the status page. SMS messages do not include recipients name or other personal identifiable information(PII). 

As always in StatusCast you have the right to choose whether SMS notifications are sent or not. SMS options can be controlled in the Settings and Integration sections of the application if you wish to review your configurations. At this time there are no additional action items for StatusCast to take in regards to its application or infrastructure, however we will maintain close communication with Twilio and if any additional information is released we will pass it along through our own status page.

StautsCast has partnered with Twilio for years and they have maintained a strong commitment to security and transparency. We will take that as well as this incident into consideration as we continue to offer SMS notifications in the application. If you have any additional questions please reach out to us at support@statuscast.com

We are very excited to announce that StatusCast has been acquired by 4Me! Since 2013 we have been working hard to close the gap between service outages and those who are impacted, and this acquisition is one large step further in our journey of providing critical information to those who need it most. 

The inclusion of StatusCast's features will aid 4Me in it's mission to modernize service management for organizations. Click here to read more!

StatusCast engineers identified a backup in its background processing that resulted in some actions being delayed from being completed in a timely fashion. Once the backup had been identified engineers worked swiftly to correct the issue and to release a patch update to help prevent this type of backup from happening again in the future. At this time StatusCast should be operating fully as expected, if you continue to have any further issues please contact us at support@statuscast.com

The StatusCast team will be performing a maintenance on December 17, 8:00am EST, the estimated duration is 2h. We do not expect any impact to your service but in some cases there may be a brief interruption.

This maintenance has been completed.

Starting August 30th, 2023 for Public Status Pages that allow SMS subscriptions StatusCast will now require that a valid email address be confirmed before a person can fully establish a new SMS subscription.

This change in subscription workflow is to help prevent malicious parties from attempting to commit SMS fraud which has become a growing concern for many SaaS companies dealing with mass notifications. We here at StatusCast have witnessed this trend, in the past 6 months the quantity of malicious traffic attempting to commit SMS fraud has increased drastically. While we have continued to implement industry best practices to safeguard against this sort of activity, ultimately real user confirmation is the most effective way to prevent such unwanted attention.

StatusCast's engineers were alerted that schedule maintenance events created from StatusCast's legacy application("V2") were not properly auto-closing after their estimated duration had been reached. After an initial investigation engineers have confirmed the cause on the service responsible and a patch was performed to correct the error. Any maintenance that was overdue for closure should have been resolved and StatusCast's engineers will continue to monitor the legacy process for this to ensure no other issues occur. 

StatusCast’s engineers were alerted that some SMS and email notifications had an unusual delay in their delivery. The team immediately began investigating and determined that one of the notification processors was experiencing a failure that resulted in a queued backup of notifications. This did not impact all notifications sent through StatusCast, only a subset that were assigned to the instance in question.

Once the instance was isolated the backup of requests were offloaded and the instance itself taken out of rotation. Even though the instance itself did not enter an unhealthy state our engineers have re-evaluated certain health checks to account for queue backups as well as other performance attributes. Additionally, our engineers have scaled out the number of instances to help reduce the load on any single processor at a given time. At this time all notification services are running as expected but we will continue to monitor the system closely.