All features and functionality have been restored. The systems have been monitored for an extended timeframe without any further incidents.
Posted Dec 20, 2023 - 07:23 PST
Update
The restoration of the eV6 (stratafax.net) historical fax images has completed and the images should be available. All services and functionality restored for both eV5 (fax.stratafax.com) and eV6 (fax.stratafax.net) Monitoring for 24 hours.
Posted Dec 13, 2023 - 06:42 PST
Update
eV6 (fax.stratafax.net) maintenance has been completed. An inability of some users to submit faxes via email due to the eV6 (stratafax.net) server(s) not accepting TLS was corrected. The restoration of the eV6 (stratafax.net) historical fax images is still in progress.
Posted Dec 12, 2023 - 22:09 PST
Update
We will be performing maintenance to add additional resources to the eV6 (fax.stratafax.net) servers tonight between 8:00pm Pacific - 9:00pm Pacific. The eV6 (fax.stratafax.net) servers will not be available during this time.
This maintenance will not affect the eV5 (fax.stratafax.com) servers.
Posted Dec 12, 2023 - 15:15 PST
Update
The restoration of the eV5 (stratafax.com) historical fax images has completed and the images should be available. An inability of some users to submit faxes via email due to the eV5 (stratafax.com) server(s) not accepting TLS was corrected. Additional eV6 (stratafax.net) server(s) have been reinitialized to bring it back to full capacity. The restoration of the eV6 (stratafax.net) historical fax images has been started. An inability of some users to submit faxes via email due to the eV6 (stratafax.net) server(s) not accepting TLS is in the process of being addressed.
Posted Dec 12, 2023 - 13:25 PST
Update
Additional eV5 (stratafax.com) servers have been reinitialized to bring it back to full capacity. eV5 (stratafax.com) ATA Connector is online and functioning The restoration of the eV5 (stratafax.com) historical fax images is still in progress.
Posted Dec 10, 2023 - 22:45 PST
Update
The delay in delivering email notifications has been resolved. All outstanding email notifications have been processed and delivered.
Posted Dec 08, 2023 - 15:54 PST
Update
There is currently a delay in delivering email notifications. We are currently working on the issue.
Posted Dec 08, 2023 - 08:36 PST
Monitoring
stratafax eV5 (fax.stratafax.com) All queued inbound faxes has been delivered. Notification queues are all caught up and inbound fax notifications have been sent.
Outbound queues have also been caught up from the previous backlog.
Next step is to add additional nodes to reduce queue time during peak times.
Posted Dec 08, 2023 - 00:59 PST
Update
We are continuing to work on a fix for this issue.
Posted Dec 07, 2023 - 14:36 PST
Update
stratafax eV5 (fax.stratafax.com) The stratafax eV5 (fax.stratafax.com) services have been restored for inbound and outbound faxing and notifications moving forward.
*To expedite the transition and restore functionality, the images for inbound/outbound faxes from prior to 12/05/2023 at 10:30pm Pacific Time were not included in the initial data transfer and will be migrated over time to lessen the impact to the servers.
Current Status: Inbound Faxes: All inbound faxes that were received since 12/05/2023 at 10:30pm Pacific Time have been pulled in and are currently being queued for email delivery; notifications have been sent, and the data/images are available in the web client. All inbound faxes that were received prior to 12/05/2023 at 10:30pm Pacific Time have been delivered, notifications have been sent, and the data is available in the web client. Images for these historical faxes will not be available at this time.*
Outbound Faxes: All outbound faxes that were sent via email-to-fax since 12/05/2023 at 10:30pm have been delivered, notifications have been sent, and the data/images are available in the web client. All outbound faxes that were sent prior to 12/05/2023 at 10:30pm Pacific Time have been delivered, notifications have been sent, and the data is available in the web client. Images for these historical faxes will not be available at this time.*
Posted Dec 07, 2023 - 14:08 PST
Update
stratafax eV6 (fax.stratafax.net) The data for stratafax eV6 (fax.stratafax.net) has been replicated in the DR location and services have been restored for inbound and outbound faxing and notifications moving forward.
*To expedite the transition to the DR location and restore functionality, the images for inbound/outbound faxes from prior to 12/05/2023 at 10:30pm Pacific Time were not included in the initial data transfer and will be migrated over time to lessen the impact to the servers.
Current Status: Inbound Faxes: All inbound faxes that were received since 12/05/2023 at 10:30pm Pacific Time have been delivered, notifications have been sent, and the data/images are available in the web client. All inbound faxes that were received prior to 12/05/2023 at 10:30pm Pacific Time have been delivered, notifications have been sent, and the data is available in the web client. Images for these historical faxes will not be available at this time.*
Outbound Faxes: All outbound faxes that were sent via email-to-fax since 12/05/2023 at 10:30pm have been delivered, notifications have been sent, and the data/images are available in the web client. All outbound faxes that were sent prior to 12/05/2023 at 10:30pm Pacific Time have been delivered, notifications have been sent, and the data is available in the web client. Images for these historical faxes will not be available at this time.*
stratafax eV5 (fax.stratafax.com) Current Status: The data for stratafax eV5 (fax.stratafax.com) is still being replicated in the DR location due to the size/amount of data and ongoing I/O issues on the SAN. A similar update as above will be posted for the stratafax eV5 (fax.stratafax.com) environment once the data replication has finished.
Posted Dec 06, 2023 - 22:26 PST
Update
The root cause of the issue has been determined to be a failing interface on the core SAN. New hardware is in the process of being moved to the datacenter and installed to resolve the issue.
In the interim, the data transfer to the DR location is also still ongoing. However, the I/O latency is slowing down this process as well. This data transfer will be completed regardless of whether the new hardware is in place prior to completion, in order to ensure data integrity and prevent future lengthy outages.
In regard to the current status of inbound and outbound faxes, please see the information below:
Outbound Faxes: Users will not receive an error message from stratafax for outbound faxes submitted via email, since the emails are not currently reaching the stratafax servers. The email servers will keep retrying the delivery. Once services are brought back online (if the email has not reached its non-delivery timeout), they will get delivered and processed by our stratafax servers.
Inbound Faxes: Inbound faxes have continued to be received by etherFAX. These faxes will get pulled down and processed for delivery once the stratafax services are reestablished.
Posted Dec 06, 2023 - 10:09 PST
Update
Service is down while the data is being migrated to the DR location. This affects all communications including email to to fax and web access. Inbound jobs will be queued up and will be delivered when services are restored. During this process, you will not be able to login to stratafax and will be redirected to host main page: https://www.ibusinesstech.com/.
Updates will be posted to this site as more information is available.
Posted Dec 05, 2023 - 22:27 PST
Identified
An issue has been identified with the SAN/LAG/Switches which is causing I/O latency. Inbound and outbound faxes have been processing throughout the day at a slower rate.
We are currently putting a plan into effect to migrate the data to a DR location. This will require the servers to be brought down periodically during this process.
Posted Dec 05, 2023 - 14:52 PST
Investigating
We apologize for the inconvenience. It appears that the same issues have occurred again causing I/O Latency. Our team is back onsite at the datacenter and working with the hardware/software vendors to to diagnose/resolve the issue.
Posted Dec 05, 2023 - 07:29 PST
Monitoring
All functionality has been restored to both fax.stratafax.com and fax.stratafax.net. We will be monitoring the system to ensure stability and are currently investigating the root cause of the issue.
All inbound faxes are available in the web client and notifications are being sent.
Outbound email to fax jobs sent during the outage appear to be processing as well. Please verify to confirm if there are any that need to be resubmitted.
Posted Dec 05, 2023 - 00:34 PST
Update
The team on site at the data center is currently engaged with the hardware and software vendors working to restore full functionality to the stratafax environment. We are also working on deploying servers in a separate environment to restore access to the stratafax application for interim access via the Web Client, Email, and API.
Posted Dec 04, 2023 - 15:16 PST
Update
We are still investigating the cause of the issue. It has been narrowed down to an extreme latency problem at the datacenter and someone is on site currently to work on diagnosis and resolution.
Posted Dec 04, 2023 - 11:23 PST
Update
We are continuing to investigate the issue. All features/functions are now affected including telephony. Inbound faxes have been received and will be accessible once the issue is resolved.
Posted Dec 04, 2023 - 09:15 PST
Investigating
We are currently investigating this issue.
Posted Dec 04, 2023 - 07:48 PST
This incident affected: StrataFax eV5 (StrataFax eV5 Web API Endpoints, StrataFax eV5 Web Application, StrataFax eV5 Telephony, StrataFax eV5 Email Submission and Delivery, Stratafax eV5 ATA Connector) and StrataFax eV6 (StrataFax eV6 Web API Endpoints, StrataFax eV6 Web Application, StrataFax eV6 Telephony, StrataFax eV6 Email Submission and Delivery).