Update 03/12/15: We have written a detailed update on what happened with ecbiz97, our current plans in action, and how we plan to prevent it in the future: here.
| Issue: | Emergency Filesystem Maintenance on ecbiz97 |
|---|---|
| Status: | Our Systems team is having to perform emergency filesystem maintenance on ecbiz97. |
| Who is impacted? | Users on the ecbiz97 server. This will affect all services (email, websites, database access) You can find what server you are on here. |
| Estimated time until resolution: | We currently have no specific ETA. |
UPDATE 3/9/2015
On 3/4/2015, our ecbiz97 shared server experienced a catastrophic file system error following emergency maintenance to replace a failing RAID controller. As a result, a file system check was necessary to verify integrity of data and prevent data loss for all accounts housed on this server. Unfortunately, critical errors persisted following the completion of an extended file system check. Now, we are currently performing a 3rd attempt to repair the file system.
Accounts have been restored from available backups running on a temporary server until this emergency maintenance is completed.
We are aware of the disruption this causes, and assure you that our top system administrators are working diligently to restore normal operations as quickly as possible. Thank you for your understanding and continued patience.
UPDATE 3/6/2015
Currently, we are running all customer accounts on ecbiz97 from a backup system. While we do this, customers may experience a slight decrease in performance due to the virtualization process used. Also since the backups that are being used are approximately 56 days old, you may notice certain behaviors uncharacteristic of your standard hosting account. Any work including emails, files, and database changes may not be present if they occured after January 9, 2015. Please keep in mind, you may need to recreate email accounts that were created in that time. As the file system check is completed and we switch from the back up to actual file system, the data will be merged. This may result in duplicated emails and other possible anomalies. Thank you for your continued patience and please accept our apologies as we work to resolve this issue as quickly as possible.
We apologize for the delay. A fsck (file system check) is running on the hard drive of the server. This process checks the hard drive for filesystem errors and is necessary to ensure there is no data-loss.
Unfortunately there is no ETA at the moment due to the nature of how the hard drive estimates the checking of all the files on the system. We will continue to update this article as soon as we have more info to share.
