Hello All
Some of you may be wondering why World of Trainz has been unavailable for the last day or so. Here is a copy of the information from the host;
Regards
Phil
Some of you may be wondering why World of Trainz has been unavailable for the last day or so. Here is a copy of the information from the host;
So rest assured World of Trainz will returnDear Valued Customers,
We apologize for the troubles you are facing as of now. Here is the detailed information on Data Center outage in DC staff words.
======================== 07:36 PM CDT =================
The Planet is currently experiencing an outage which is effecting a number of customers' servers. This issue may also be affecting customers' ability to get through to our call center.
We are doing everything we can to remedy this problem as quickly as possible. And will place an updated notice here as soon as there is new information.
=========================07:36 PM CDT ==================
Today at approximately 5:45 p.m., a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down.
We are working with the fire department, with our facilities staff on site, to assess the situation.
=========================07:51 PM CDT ===================
We have determined that no servers in the data center have been damaged. Nonetheless, they are down because power is out. Teams across the board are working to take appropriate action.
========================08:17 PM CDT ====================
In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center.
======================= 10:46 PM CDT ===============
From Data Center CEO ,
This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.
We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.
This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.
We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well. There is no impact in any of our other five data centers.
I am sorry that this accident has occurred and I apologize for the impact.
========================= 01:56 AM ===============
Few of Ehostpros.com servers back online. EHostPros Staff checking servers status and working with DC to bring back remaining servers online.
=========================02.30 AM ===============
Nearly 80% of ehostpros.com servers back online. All web sites on those 80% servers should be working fine.
=========================08:54 AM================
We do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our data center staff and management continue to work through the night and morning.
======================== 01:45Pm......
To keep you up-to-date, here is the latest information about the outage in our H1 data center.
We expect to be able to provide initial power to parts of the H1 data center beginning at 5:00 p.m. CDT. At that time, we will begin testing and validating network and power systems, turning on air-conditioning systems and monitoring environmental conditions. We expect this testing to last approximately four hours.
Following this testing, we will begin to power-on customer servers in phases. These are approximate times, and as we know more, we will keep you apprised of the situation.
================================08.20pm============
DC continue to work to restore power to the data center and bring all affected customer servers online.
=================================11:14 PM==================
We hope remaining 20% servers will be online with in 5 hours, here is the latest information about the outage in our H1 data center. From Data Center CEO ,
There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.
We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.
Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.
We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.
===================================12.54 AM======
Next updates from Data Center...
After the fire marshall inspected the H1 location, we were given the green light to bring power back to the facility. The generators have been turned on, and we are receiving power on the second floor. The generator power restoration is the first step in the full restoration of service to the data center.
From here, we will begin the process of cooling the DC floor, which could take a few hours. As soon as the power integrity is confirmed and the DC floor is ready for operation, we will be restoring power and checking server hardware on a rack-by-rack basis.
======================================01:55 AM==========
Following the restoration of power to the second floor of the data center, we've cooled the data center floor and are now in the process of systematically restoring power to racks.
We've got a full staff in the data center to power up racks in sections and verify that the server hardware starts up successfully. This process may take a few hours to restore service to all customer servers on the second floor.
Thanks for your cooperation.
Regards
Phil
