Jul 11, 20:43:00 GMT+0
Monitoring -
Update directly from the data center:
The EWR data center remains powered down at this time per the fire marshal.
We have just finished the meeting with the fire marshal, electrical inspectors, and our onsite management. We have made great progress cleaning and after reviewing it with the fire marshal, they have asked us to clean additional spaces and they have also asked us to replace some components of the fire system. They have set a time to come back and review these requests at 9am EDT Wednesday. We are working to comply completely with these new requests with these vendors and are bringing in additional cleaning personnel onsite to make the fire marshal's deadline.
In preparation for being able to allow clients onsite, the fire marshal has stated that we need to perform a full test of the fire/life safety systems which will be done after utility power has been restored and fire system components replaced. We have these vendors standing by for this work tomorrow..
Jul 12, 18:28:07 GMT+0
Monitoring -
Our upstream provider has provided us with the following update:
Our onsite team has energized the primary electrical equipment that powers the site, enabling us to bring our mechanical plant online. We are currently cooling the facility.
As we monitor for stability, we are focused on bringing up our electrical systems. In starting this process, we have identified an issue with powering up our fire panel as well as power systems that were powered by UPS3. While this will cause us a delay, we are working with our vendors for remediation.
We are currently at 25% for completion toward bringing the site back online and the revised ETA for bringing up the critical infrastructure systems is approximately 7 hours. We are still planning for an evening time frame when clients will be able to come back on-site. We will send out additional information regarding access to the facility and remote hands assistance and we will notify you once client access to the facility is permitted..
Jul 13, 00:18:00 GMT+0
Resolved -
The server is back online.
This incident has been resolved..
Jul 10, 23:08:00 GMT+0
Investigating -
Our email server is currently experiencing an outage due to a power issue impacting the entire datacenter.
The facility are investigating and have shared the following notice:
"We are currently experiencing a sitewide power issue related to our UPS systems at EWR1. We have site personnel investigating the issue. We have called our UPS vendor who is currently in route to the site to assist us as well.".
Jul 11, 01:20:00 GMT+0
Investigating -
We are still awaiting news from the facility in relation to the resolution of this issue..
Jul 11, 05:08:00 GMT+0
Identified -
Update directly from the data center:
Power remains off at our data center in EWR1 per the local fire marshal. We have had an electrical failure with one of our redundant UPS' that started to smoke and then had a small fire in the UPS room. The fire department was dispatched, and the fire was extinguished quickly. The fire department subsequently cut power to the entire data center and disabled our generators while they and the utility verify electrical system. We have been working with both the fire department and the utility to expedite this process. We are currently waiting on the fire marshal and local utility to reenergize the site. We are completely dependent upon their inspection and approval. .
Jul 11, 05:40:00 GMT+0
Monitoring -
Update directly from the data center:
After reviewing the site, the fire Marshall is requiring that we extensively clean the UPS devices and rooms before they will allow us to re-energize the site. We have a vendor at the site currently who will be performing that cleanup
**To be clear the fire was not in the data center floor and was contained to an electrical room.**
——————————
We are in constant communication with the data center. We do expect an update by 12 PM UTC.
Jul 11, 11:44:00 GMT+0
Monitoring -
Update directly from the data center:
We expect the cleanup to take place at 8am and the fire marshall to inspect afterward, We will keep you updated along the way..
Jul 11, 15:03:00 GMT+0
Monitoring -
Update directly from the data center:
Our remediation vendor and our team has worked through the night to clean the UPS' at the request of the fire marshal. They have made significant progress and we hope to have the cleaning completed by mid-day, at which time we will engage the fire marshal to review the site. Following their review, we hope to get a sign off from them so that we can start the reenergizing process. The reenergizing process can take 4-5 hours, as we need to turn up the critical infrastructure prior to any servers..
Apr 11, 20:15:00 GMT+0
Identified -
Our France datacenter is currently replacing their routing network infrastructure to address recurring issues.
They are expecting a single interruption of 5 - 10 minutes in the public network.
Unfortunately, we were only notified of this incident about an hour ago and were unable to keep our customers aware of this earlier.
We apologize for any inconvenience this may cause and assure you that we are working closely with the datacenter to minimize any potential impact on our services..
Apr 11, 20:20:00 GMT+0
Monitoring -
Our datacenter has implemented a fix and are currently monitoring the result..
Apr 11, 20:33:00 GMT+0
Identified -
Our datacenter is currently facing difficulties and the outage is lasting longer than expected. Their teams are actively working to complete the operation as soon as possible. We apologize for the inconvenience and thank you for your patience..
Apr 11, 21:03:00 GMT+0
Resolved -
The maintenance is now complete. We apologize for the longer service outage than initially communicated. We confirm that the France network has now been migrated to a new network stack and the concerns noted in recent days should not reoccur in the future..
Apr 6, 15:55:09 GMT+0
Completed -
Maintenance has completed successfully..
Apr 6, 09:40:10 GMT+0
Identified -
Maintenance is now in progress..
Apr 6, 11:26:39 GMT+0
Identified -
Due to some unforeseen issues, the maintenance is taking longer than initially anticipated. We will update this incident once we have completed the maintenance..
Apr 6, 09:30:00 GMT+0
Identified -
During our scheduled maintenance window we will be rebooting all web hosting servers to load in updated kernels. Several of our servers have been up for over 2 years without a reboot and while we do use KernelCare to apply kernel level security updates without reboots, it is time to load newer kernels to all hosting servers.
We expect the downtime for each server to be around 10-15 minutes. If we run into any further issues, we will update this incident with further details..
Mar 2, 06:51:09 GMT+0
Identified -
We are currently migrating our email service to standalone infrastructure to ensure that all emails sent by us reach our customers' inboxes. During this process, all emails generated on our billing website including password reset, order confirmation, invoice payment confirmation, and ticket response emails will be queued in our email system until the migration is complete.
Please note that any emails sent to us during the migration will not be delivered and will be discarded. We apologize for any inconvenience caused and appreciate your understanding..
Mar 2, 14:16:25 GMT+0
Completed -
Our email service has been successfully migrated to standalone infrastructure.
All emails from our billing website that were queued during the maintenance will be sent out within the next 30 minutes.
We are also pleased to announce that our email infrastructure is now fully operational, and we are able to receive emails as normal..
Mar 26, 18:31:00 GMT+0
Resolved -
Everything is back online..
Mar 26, 18:29:00 GMT+0
Monitoring -
Server has been rebooted to fix a minor issue with the networking configuration..
Mar 25, 04:37:00 GMT+0
Completed -
Maintenance has been completed successfully..
Mar 25, 04:31:00 GMT+0
Identified -
Maintenance is now in progress..
Mar 25, 04:30:00 GMT+0
Identified -
We will be upgrading server software/packages and rebooting our Bangalore-1 and Proton Servers Host Panel during the above time frame for increased reliability. We do not expect any downtime for more than 3 minutes but in the case, you encounter any issues even after the scheduled 15-minutes window feel free to open a support ticket and our support representatives will be glad to assist you..
Mar 17, 00:00:00 GMT+0
Monitoring -
We have successfully reinstalled and configured control panels on the respective servers. Websites are being restored as we post this update and are seeing most of the websites return to their original state.
We are currently monitoring the result and will post our next update when all of the websites have been restored..
Mar 15, 10:25:00 GMT+0
Investigating -
We are currently investigating elevated faults on the file system of our United Kingdom dedicated server..
Mar 16, 18:05:00 GMT+0
Identified -
We have detected that file system on our United Kingdom dedicated server has been corrupted which is due to the recent reconfiguration of iDRAC (Integrated Dell Remote Access Controller) by our datacenter; it is a software that allows us to update and manage our Dell systems.
There is no conclusive evidence to support any data loss, and as a result, over the next few hours, the websites will be restored to their original state before corruption after reinstall and configuration of the respective control panels.
We are continuing to work on a fix for this incident.
If you have any questions, or wish to have any technical details, implications, and protocols that we do here at Proton Servers Host answered/explained, feel free to open a ticket on our portal asking about this.
Additionally, for the time that the server is down, you are entitled to our SLA compensation. However, we are willing to personally offer a 72 hours/3 days extension to all web hosting customers on this server and their active web service with us. You'll need to do this on our portal, located at [https://portal.protonservershost.com](https://portal.protonservershost.com/submitticket.php?step=2&deptid=4)
Note: There is no data loss
Thank you for your understanding .
Mar 17, 04:25:00 GMT+0
Resolved -
We just resolved this issue!.
Feb 4, 14:31:00 GMT+0
Identified -
Maintenance is now in progress..
Feb 5, 12:30:00 GMT+0
Completed -
Maintenance has been completed successfully..
Feb 4, 14:30:00 GMT+0
Identified -
During the mentioned timeframe, our datacenter's engineering team will be performing maintenance on redundant power units for our resources in our BLR-1 region, in order to improve reliability.
We don't expect to see any downtime as power to Hypervisors is redundant, however, in the very unlikely situation that we lose power to both the supplies, we may experience downtime of 5 minutes to maximum 1 hour. We will endeavor to keep this to a minimum for the duration of the change..
Nov 12, 14:30:00 GMT+0
Identified -
During the mentioned timeframe, our datacenter's engineering team will be performing maintenance on redundant power units for our resources in our BLR-1 region, in order to improve reliability.
We don't expect to see any downtime as power to Hypervisors is redundant, however, in the very unlikely situation that we lose power to both the supplies, we may experience downtime of 5 minutes to maximum 1 hour. We will endeavor to keep this to a minimum for the duration of the change..
Nov 13, 12:30:00 GMT+0
Completed -
Maintenance has been completed successfully..
Nov 12, 14:31:00 GMT+0
Identified -
Maintenance is now in progress..
Apr 12, 22:01:00 GMT+0
Investigating -
We are currently investigating this incident..
Apr 13, 05:38:00 GMT+0
Resolved -
We just resolved the issue!.
Apr 13, 00:17:00 GMT+0
Identified -
A recent cPanel update that came out a few hours ago seems to have a few bugs present it which affected the DNS server causing a Proxy Error.
We are continuing to work on a fix for this incident.
Please note: None of the customers web servers or websites have been affected by this issue and it is only our website that is affected at the moment. .
Oct 2, 00:00:00 GMT+0
Completed -
Maintenance has been completed successfully..
Sep 14, 00:01:00 GMT+0
Identified -
Maintenance is now in progress.
Sep 14, 00:00:00 GMT+0
Identified -
During the mentioned Time Frame, our Datacenter will be rolling out a global software update to improve networking performance and stability on our hypervisors. There should be minimal downtime on the affected hypervisor during the rollout.
This maintenance is being performed due to one of our Provider's Engineers discovering a bottleneck in our networking software stack. After the update, our hypervisors will be better able to handle higher network loads without network disruption.
The expectation is that most users will observe no impact, though some users with a sustained high-packets-per-second or high-throughput workload will see a small number of dropped packets..