| Issue: | Emergency Filesystem Maintenance on biz100 |
|---|---|
| Status: | Our Systems team is having to perform emergency filesystem maintenance on biz100. |
| Who is impacted? | Users on the biz100 server. You can find what server you are on here. |
| Estimated time until resolution: | Updated at 3:32pm. As our Systems team is being thorough, it is taking longer than originally estimated. There is currently no estimated time for completion. |
Update (7:08 EDT)
The filesystem check has completed and all services have been restored.
Update (4:25PM EDT)
We apologize for the delay. A fsck (file system check) is running on the hard drive of the server. This process checks the hard drive for filesystem errors and is necessary to ensure there is no data-loss.
Unfortunately there is no ETA at the moment due to the nature of how the hard drive estimates the checking of all the files on the system. We will continue to update this article as soon as we have more info to share.
I just noticed that the status on this has not yet been updated. My site and email have been back up for at least 12 hours now.
Is the problem ongoing, or has it been resolved?
There is no way to estimate this as the file system check could take an hour, or it could take several hours. From there, if a disk needed to be replaced or other possible issues showed up, there could be additional time added. When a filesystem check is run, there is no possible way to estimate the amount of time it will take.
If we provide an estimated timeframe for it to be completed, we want to hold up to that estimate. If we were to estimate an hour for completion, and it took a full day, we do not want to provide an unrealistic expectation.
We understand that it is indeed frustrating when there are issues, and those issues affect us just as much as they affect you. We want to make sure that if we set an expectation, that those expectations are able to be satisfied.
In reference to previous downtime, as the server is just like any other piece of technology, it can have unforeseen issues from time to time. This is especially true of shared hosting where there are multiple users all sharing the same server. One way to minimize issues would be a dedicated environment. With a dedicated server, far less issues can occur because the environment is yours alone. If the same checks were required on a dedicated server, which would be far less possible, it would be much quicker as it would only be checking on your specific drive and not a much larger drive that stores the data of every other user on the server as well.
The issue has been resolved. I’ll update the article now to avoid any confusion by anyone else who comes across it.
It never ceases to amaze me how companies beg for feedback from customers and then try to school us when we do so.
Any estimate would have been helpful. Considering my site was down for three days last year, I wanted to know what to expect this time.
I asked only to be given an “hours, days or weeks” answer, and even that was not possible? Come on.
When your customers tell you they want an ETA, your response should be: “Fine. We’ll give you an ETA. It may not be accurate, but you asked for it.”
Instead, we’re told why it’s too much to expect. If InMotion doesn’t know whether the process is likely to take 8 hours or 8 days or 8 weeks, they don’t deserve to remain in business.
Yes, most of us are business owners or IT Consultants managing email/websites for clients. Because of yesterday’s lack of communication, we are moving our services to providers who are tried and true. Though they cost more, if you tally up the cost of lost business in one day, only just one business transaction more than pays for the difference in a year’s worth of service. Just not worth the aggravation, lack of communication, unprofessionalism, and downtime.
InMotionHosting should have provided an ETA no matter how terrible. Business owners and IT consultants need to have this information in order for them to come up with a plan B. Having no ETA due to “We don’t know how long it’s going to take” is a lame business excuse. Underpromise, overdeliver.
No ETA has been provided because it is impossible to determine how long the filesystem checks will take. As we do not want to set expectations that would be impossible to guarantee, we are unable to provide an exact time frame for completion. This is the same across any hosts that may run into this issue and it is completely unavoidable as all hosts can go through this with any of their servers. I understand this can be frustrating, but I can assure you that this is possible with any host and nobody will be able to provide you with an estimated time that a full filesystem check would be completed.
You may want to inform people that the server is back up. At least my sites and email are working.
During the downtime, any senders would have received a bounceback notice stating that the mail was unable to be delivered. Their mail server, however, may continue to attempt to send the mail over time in which you may get some of those messages that initially failed.
Hello,
I am very concerned-where are all my emails from 6:30am PDT and 4PM PDT? There is a huge gap? Please tell me data wasn’t lost…
-Joyce
Thank you for working hard and answering all of our comments. Obviously downtime has a negative impact on everyone… especially business like ours.
That being said, I think most of us take uptime for granted these days. Generally speaking, we have received a high level of performance from our many shared hosting plans. At the price of a shared plan, we are more than happy with the quality of service!
Again, thanks for working diligently on this problem for all of us!
-c
Hello Chris,
Thanks for the kind words! Still, the down time really does upset us. We are working to make sure that when these events to happen, that we can use the experience to help improve our business. As per the comments above by Steve, we definitely have room for improvement in the way we communicate the issue.
That being said, thanks again for the positive spirit and thanks! We look forward to being a more improved host and helping you with your website needs.
Regards,
Arnel C.
DON’T INSULT OUR INTELLIGENCE!! Obviously whatever backups and other maesures you have in place are not working. Looking at these posts, you can see we are all business owners. It’s 11 hours down now and still counting. I get it that sometimes the best plans just don’t pan out and things do break. But, wen we call or chat for answers, you need to be honest with us – tell us what is really going on and a realistic time frame to repair. Your lack of information leaves us helpless and makes us look bad to our customers
Hello Steve,
We’re not insulting you’re intelligence. If you want an example of how difficult it is give an ETA on file recovery on a computer, think on this example. When you try to transfer a large group of files on a computer from one place to another, you will get an estimated time, but if you watch that estimation of time, it will continually change. It’s a number that is continually being “re-calculated”. It is the same in the case of the hosting server, except we’re dealing with a large number of files. So, I do apologize that you think we’re trying lie or be dishonest when we’re really trying our best to be honest with you when we say “we don’t have a definite ETA.”
As I said in my earlier post, we really do care for our customers and I have gone to the Customer Experience manager to explain EXACTLY what you’re saying – that our handling the matter needs to be improved. The problem is that when we try to be general about the issue, it comes off as if we’re not providing enough information. If we try to be technical about the issue – as per the explanations in the posts below, it just becomes a technical quagmire for those not familiar with file systems/servers.
And you’re right that in this case the redundancy that is currently in place did not help in this (other than to make sure that data was NOT lost). Redundancy for immediate return to live status is an expensive venture – it’s typically much more than what would be available for a shared server account. We do try to provide a stable working environment that provides security for server data while also providing a cost effective solution for small businesses and individual a certain level of website.
I hope this to provide an explanation behind our communications and our efforts to resolve the issue. I understand the frustration you’re feeling – and believe me you’re not the only one affected by this issue. We are always working to improve things in order to keep any downtime to a minimum.
Kindest regards,
Arnel C.
Hello again Steve,
As per my reply to you above, if you want an analogy similar to the server, trying to give a realistic frame of repair is difficult due to the many files involved and the time it takes for the server to work through these files. We are not trying to be dishonest with you. We are working with Customer Experience to find a more informative way of explaining this type of problem. Though we would rather not see this type of event occur again. As the process is improved our technical support staff will be involved in communicating the problem in a more constructive and helpful manner.
Regards,
Arnel C.
Don’t insult our intelligence! This is a group of business owners. We understand that stuff happens- things break. We get that. What really ticks me off is the way you handle the situation. Hours before we get any info. Then it’s always ” no ETA”. It’s insulting and counter productive. All we ask is for some honesty so we can run our businesses and talk to our customers. Obviously the redundency you claim to have is just not working. This is not the first time and surely will not be the last.
Hello Jonathan,
Unfortunately with web hosting there are a lot of things to keep up and running, and looking at your account notes it does appear you had problems last year in March with your site due to some power outage problems at the data center. This of course is a more isolated problem just to the physical server that houses your account, but regardless either way your website has been down before and we apologize for that.
We have been updating this guide as we have info, but unfortunately we didn’t have much to share. As when a file system check is running on the hard drive, it does take some time to complete and there’s not an accurate way to guess how long it will take.
The latest update we’ve received is that it does appear to be in the final stretch of the filesystem check, so hopefully services will start popping up soon. We are looking into a proactive email notification system for instances like this, unfortunately that is not available at this time yet.
Sorry for any inconvenience this outage is causing you and your business, it hopefully shouldn’t be much longer!
– Jacob
This is not the first time something like this has happened with my shared hosting account. Last time my site was down for three days.
Even if you don’t have and ETA, is would be nice to know if we’re looking at hours, days or weeks of downtime.
Some proactive acknowledgement of the problem via email would have been nice, too. I had to contact Customer Service to find out the problem wasn’t on my end.
Happens about once a year: I get an email about an upcoming “short” interruption and it takes hours to recover! Brilliant.
Hello Dave,
Sorry if you’ve been experiencing longer than normal service interruptions once a year or so since being with us. We really appreciate you staying with us through them.
We have been making improvements to our network and server configurations to minimize any downtime at all, but unfortunately it’s not perfect yet. I apologize for any inconvenience this current issue is causing you!
– Jacob
This has been a major deal for my business as well.I’m on a crucial deadline, and I haven’t had access to email for over 8 hours. I cannot believe that there is no redundant system in place.
Hello Brad,
Apologies for the downtime! We do have redundant back up servers, but unfortunately this typically requires the transfer of all the user data off the old server’s hard-drives, or from the latest automatic backup that is stored on our separate backup servers.
The process of allowing a server’s fsck or file system check to complete running is typically far less than the average total data transfer time, and you also are recovering with more recent data from databases and the like because it’s not from a data backup possibly 24-36 hours old.
We do understand the importance of server uptime and sincerely apologize for the issues it has caused.
Regards,
Arnel C.
Hello Chris,
Apologies for the problems. Again (as per the answers above), all of the services on the server are affected by this downtime. We do not have a current ETA on the resolution of the issue. Visit the InMotion Server Status page for any updates.
Regards,
Arnel C.
This appears to affect our email as well. This is causing a fairly serious situation at the moment! Any update on when this will be resolved?
-c
Isn’t there a redundant back up server that could take over in these cases, it seems that your system is really vunerable.
We are ending our Crowd source funding project in the next 36 hours… we just did and major push to our site and now we cannot capitalize on our efforts. MAJOR fail.
Dean
Hello Chris,
Apologies, but again, there is no current ETA on the issue. This is what I can tell you…our systems team is running a recovery program on the server and because there are millions of files on it the actual ETA for their recovery ambiguous. For example, we may have a bunch of graphic files that take 10 minutes to recover, but then you have another 10 database files that require more time because they have more data. It’s not a uniform thing which leads to uncertainty for the recovery time.
I’m trying to give you some information on the issue, but the bottom line is there is no true ETA on the recovery of the server at this point. Please understand that we are working as fast as possible to get things operational again.
Regards,
Arnel C.
Hello Dan,
It is a bit strange to say “emergency maintenance”, in this case we’ve updated this announcement a bit more to be more specific. The server had to be rebooted in an emergency situation and upon coming back up needed to start a filesystem check to ensure no data-loss.
Typically server outages are very short in nature, usually a few hours at most. Unfortunately in this case when dealing with a large hard drive there are a lot of files on the filesystem for the server to check and verify for integrity and that’s why this problem is taking longer than normal to recover from.
I definitely empathize with both yourself and our other customers. This isn’t a fun position to be in for either side, and we have made a lot of strides to ensure things like this happen few and far between for our customers.
Sorry for all the inconvenience I know this is bringing to your business. I hope you can rest assure that once this maintenance is complete we’ll be back to our normal uptimes.
– Jacob
Hello Dean, sorry about the issues.
Sorry that these rare problems happened for you at a very inopportune time for your crowdfunding. That definitely is a major fail on our behalf and I’m sorry nothing can be done right this instant to get your site back up and running.
We do have redundant back up servers, but unfortunately this typically requires the transfer of all the user data off the old server’s hard-drives, or from the latest automatic backup that is stored on our separate backup servers.
The process of allowing a server’s fsck or file system check to complete running is typically far less than the average total data transfer time, and you also are recovering with more recent data from databases and the like because it’s not from a data backup possibly 24-36 hours old.
Again really sorry for the troubles Dean, we don’t have an exact ETA but are hoping it will be back up just as soon as possible!
– Jacob
Please provide an ETA at least. You’ve been down since at least 8:15am CST today. Thank you,
Chris Van Horrick
Lab Interoperability Collaborative
I think “emergency maintenance” is an oxymoron!
Having said that, I have been an InMotion customer for years and the only other downtimes were very, very short. I have been a satisified customer.
But today has been a killer for my small businesses, too. I empathize with Ms. Zenner.
Hello Douglas,
Apologies for the current status. The earlier ETA was based on a rate that has since slowed resulting in a much longer recovery period. At this time, we do not have an ETA, but we will update the announcement as soon we have something more solid. The systems team is working as fast as possible to resolve this issue. We do apologize for the ambiguous answer, but we honestly can’t give a completion time right now.
Apologies again for the inconvenience!
Regards,
Arnel C.
When do you expect this service to be completed? It’s now after 3pm EDT and I still am unable to send or receive my company email. Thanks.
Doug Flinn
meraco.com
Hello Baris,
I’m sorry that you believe the issue to be a scandal. InMotion Hosting does not make it a practice to have down times with our server. In fact, no hosting company would want this as part of their service. Servers do have mechanical parts and thus they are susceptible to breaking down. We do regularly maintain our server fleet and manage what needs to be replaced or upgraded in the appropriate time as circumstances allow. However, there are situations when something breaks down or a network issue occurs that is beyond our control. It is during these times that we work hard to keep these periods of downtime to a minimum. Unfortunately, it’s also during these times that some people are negatively affected due to these losses of service. We do recognize this and we do apologize for those affected.
We regularly review our procedures and we make an attempt to improve them. Currently, there are three separate departments who advocate for our customers within InMotion – Customer Care, Customer Experience and Community Support. We do truly care for our customers and we are always working to make things better.
So, please don’t call it a “scandal” as we never intentionally want any type of down time for the services that we provide. Any hosting provider who is doing that is doing a disservice to their customers and to their own bottom line.
Kindest regards,
Arnel C.
One word : SCANDAL
Hello Joyce Zenner,
I’m sorry to hear that you faced the brunt of the server downtime and that it has affected your business. You can be compensated, which is typically done by contacting our customer care department. However, I have already reached out to them for you and they should be touching base with you soon.
The entire server fleet regularly undergoes maintenance and upgrades. We do understand the value of uptime for our customers and we are always working to improve our systems. There are redundancy systems in place and improvements underway, but downtime due to server/internet connections can still happen. We strive to keep these problems to them to a minimum and apologize that you were affected in this instance.
If you have any further questions or comments, please let us know.
Regards,
Arnel C.
Hi,
This has been a HUGE issue for me and my small business. Prime morning hour and it’s just unacceptable about once a year there is an issue. It cost me answers and getting some tracktion and a contract signed today.
It makes a small business look rinky dink when this happens. I hope you find a way to compensate us for the issues.
How could this have been prevented? What lead up to it-don’t you get regular notifications or have redundant systems?
It’s very very frustrating to want to count on this service and be red faced to customers.
-Joyce