Proton Servers Host - Email Server Outage – Incident details

The server is back online.

This incident has been resolved.

Our upstream provider has provided us with the following update:

Our onsite team has energized the primary electrical equipment that powers the site, enabling us to bring our mechanical plant online. We are currently cooling the facility.

As we monitor for stability, we are focused on bringing up our electrical systems. In starting this process, we have identified an issue with powering up our fire panel as well as power systems that were powered by UPS3. While this will cause us a delay, we are working with our vendors for remediation.

We are currently at 25% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 7 hours. We are still planning for an evening time frame when clients will be able to come back on-site. We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted.

Update directly from the data center:

The EWR data center remains powered down at this time per the fire marshal.

We have just finished the meeting with the fire marshal, electrical inspectors, and our onsite management. We have made great progress cleaning and after reviewing it with the fire marshal, they have asked us to clean additional spaces and they have also asked us to replace some components of the fire system. They have set a time to come back and review these requests at 9am EDT Wednesday. We are working to comply completely with these new requests with these vendors and are bringing in additional cleaning personnel onsite to make the fire marshal's deadline.

In preparation for being able to allow clients onsite, the fire marshal has stated that we need to perform a full test of the fire/life safety systems which will be done after utility power has been restored and fire system components replaced. We have these vendors standing by for this work tomorrow.

Update directly from the data center:

Our remediation vendor and our team has worked through the night to clean the UPS' at the request of the fire marshal. They have made significant progress and we hope to have the cleaning completed by mid-day, at which time we will engage the fire marshal to review the site. Following their review, we hope to get a sign off from them so that we can start the reenergizing process. The reenergizing process can take 4-5 hours, as we need to turn up the critical infrastructure prior to any servers.

Update directly from the data center:

We expect the cleanup to take place at 8am and the fire marshall to inspect afterward, We will keep you updated along the way.

Update directly from the data center:

After reviewing the site, the fire Marshall is requiring that we extensively clean the UPS devices and rooms before they will allow us to re-energize the site. We have a vendor at the site currently who will be performing that cleanup

To be clear the fire was not in the data center floor and was contained to an electrical room.

——————————

We are in constant communication with the data center. We do expect an update by 12 PM UTC

Update directly from the data center:

Power remains off at our data center in EWR1 per the local fire marshal. We have had an electrical failure with one of our redundant UPS' that started to smoke and then had a small fire in the UPS room. The fire department was dispatched, and the fire was extinguished quickly. The fire department subsequently cut power to the entire data center and disabled our generators while they and the utility verify electrical system. We have been working with both the fire department and the utility to expedite this process. We are currently waiting on the fire marshal and local utility to reenergize the site. We are completely dependent upon their inspection and approval.

We are still awaiting news from the facility in relation to the resolution of this issue.