Some systems are experiencing issues

About This Site

Maintenances and incidents regarding our services, networks and datacenters can be followed here.

Scheduled Maintenance
[DC3] Electrical maintenance

We will do the annual maintenance of the inverters of electrical chains A, B, C and D from 24 to 28 September 2018.

The inverters of these electrical chains supply racks of IT halls. Operations will be performed by our maintainer under our permanent supervision, according to the following previsional schedule :

  • 24/09 Electrical chain A

  • 25/09 Electrical chain B

  • 26/09 Electrical chain C

  • 27/09 Electrical chain D

  • 28/09 Reserve if planning delayed

This maintenance requires isolating each UPS in or der to intervene on the device. The downtime of an UPS is estimated to take 30 minutes. The operation will be done UPS by UPS and will have no impact on your equipments housed in our Datacenter : the others UPS of the electrical chain are sized to recover the load from the offline UPS.

[DC3] CRAC Units proactive maintenance

We will carry out the semi-annual preventive maintenance of CRAC units from September 3rd to October 4th, 2018. CRAC units will be shut down one at a time, so as not to impact the cooling of halls. Each IT zone has CRAC unit called «backup»: each CRAC unit is doubled. Each MVLB hall has a main air hanling unit (AHU).

The maintenance will be done according to the following schedule:

  • MLVBsand operators halls: 03 to 07/09
  • IT halls n°4 : 10 to 14/09
  • IT halls n°3 : 17 to 21/09
  • IT halls n°2 : 24 to 28/09
  • IT halls n°1: 01 to 04/10

Impacts on services : Loss of cooling redundancy during the preventive maintenance operation of aCRAC unit (max 1h). No impact on the cooling of facilities. The maximum cut-off time per CRAC unit is otherwise defined at 1 hour

Thanks for your understanding

[DC5-A] Low voltage maintenance TGBT1

We are scheduling an electrical maintenance on TGBT1. It should not have any impact on availability, but some power redundancy will be down.

Maintenance will start on October 11th 2018, at 11:00PM UTC, and last around 7 hours (until October 12th, at 6:00AM UTC).

Past Incidents

Thursday 20th September 2018

No incidents reported

Wednesday 19th September 2018

[DC5-A] RPN switch update racks C9 / C10, scheduled 1 day ago

We plan to update one of our RPN router in DC5-A on 09-19-18 at 7am UTC. A 15 minutes downtime is expected on RPN services in racks C9 and C10.

We will keep you updated through this status.

Tuesday 18th September 2018

[DC5] Major Maintenance - Relocation of hall 1-T - Sept. 18th, scheduled 2 days ago

The launch of our DC5 datacenter requires us to perform a major maintenance operation on Tuesday September 18th, from 6 am to 8 am (Paris time - UTC+2).

DC5 is a 24 MW datacenter of over 16,000 m² (over 170,000 sqft) with a targeted power usage effectiveness (PUE) of only 1.09.

During this intervention, servers will be temporarily unavailable as they will be disconnected and moved from room 1-T to room 1. Operations will be done by batches and we expect a one hour downtime.

To ensure the integrity of your data, we encourage you to backup your disks and shutdown your systems.

More information will be available in the upcoming days regarding operations’ timeframe per rack.

Thank you for your understanding.

18/09/18 14:20 UTC (16:20 Paris time)
  • The physical relocation of all servers has happened. 2 separated localized issues have appeared that can be followed here :
  • [DC5] Networking issues => https://status.online.net/incident/61
  • [DC5] Management interface unreachable on some servers https://status.online.net/incident/62 We are closing this status
18/09/18 12:40 UTC (14:40 Paris time)

B6 and B7 are now available again.

18/09/18 12:15 UTC (14:15 Paris time)

Following racks are up and pinging again : A4 - A5 - A6 - A7 - A8 - A9
Those servers have been unlocked from Maintenance mode.

18/09/18 12:05 UTC (14:05 Paris time)

Following racks are now up and pinging again : B5 - B8
Those servers have been unlocked from Maintenance mode.

18/09/18 11:20 UTC (13:20 Paris time)

Expecting the whole infrastructure to be completely migrated around 12:00 UTC (14:00 Paris time).
However, we are still experiencing and investigating network issues with switches.
Some of them are up but chained network isn't stable enough to restore services for now.
We will keep you updated.

18/09/18 10:00 UTC (12:00 Paris time)
  • 16 / 18 racks are in their final locations
  • Some issues reconnecting the switches
  • 1 rack is fully operational A11 - several others are starting progressively
18/09/18 09:25 UTC (11:25 Paris time)
  • B6 rack is now UP and pinging again. Services are being unlocked on the console.
  • B5 / B7 / B8 network equipment being restored right now.
18/09/18 08:50 UTC (10:50 Paris time)
  • Racks A10 A11 A13 A17 are being reconnected, network equipments are being restored.
  • A11 is fully up and pinging now, others are coming.
18/09/18 07:50 UTC (9:50 Paris time)
  • Network rack is up & running
  • 8 / 18 racks are in place and being connected progressively
18/09/18 07:00 UTC
  • Network rack is up & running
  • 4 / 18 racks are in place
18/09/18 06:10 UTC

The maintenance is on going. 50% complete (% rack moved out of their orginal location).

18/09/18 06:00 UTC

The maintenance is on going.

18/09/18 05:25 UTC

The maintenance has started on time at 04:00 UTC First racks are in place in their final location

14/09/18 07:04 UTC

The maintenance window schedule has been reduced from 2 days to one day. We plan to do the full move on the 18/09/18 (was before the 17th and 18th)

DC5 Network [DC5] Networking issues

DC5 - Network issues : We have several network issues following the DC5 server relocation from room DC5 1T to DC5 1 The teams are fully engaged to restore the service asap.

===================

18/09:18 15:50 UTC 17:50 Paris
  • Racks with partial network :
  • Racks with full network : A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11 A12 A13 A17 B5 B6 B7 B8 All the servers are now connected to the network correctly, if you still have issues accessing the server or power it up please follow the status : https://status.online.net/incident/62
18/09:18 14:50 UTC 16:50 Paris
  • Racks with partial network : A4 A13
  • Racks with full network : A1 A2 A3 A5 A6 A7 A8 A9 A10 A11 A12 A17 B5 B6 B7 B8
18/09:18 14:18 UTC 16:18 Paris
  • Racks with partial network : A1 A2 A3 A4 A5 A8 A12 A13 B7
  • Racks with full network : A6 A7 A9 A10 A11 A17 B5 B6 B8
18/09:18 14:00 UTC 16:00 Paris
  • Racks with partial network : A1 A2 A3 A4 A5 A8 A10 A12 A13 B7
  • Racks with full network : A6 A7 A9 A11 A17 B5 B6 B8
DC5 [DC5] Management interface unreachable on some servers

Following the maintenance (https://status.online.net/incident/29) we are finishing the configurations of the servers in the racks that have their network back online.
If your server is located in a rack that is listed as fixed here (https://status.online.net/incident/61) but is not reachable it means that the IPMI (management interface) is unreachable and our team is currently working on it to recover full access.

===================

19/09/18 10:00 UTC 12:00 Paris

All servers have been reconfigured. If you still experience an issue don't hesitate to contact our assistance team through ticketing so we can investigate further.

18/09/18 16:10 UTC 18:10 Paris

Servers are regaining access gradually. Operations are still ongoing to fix all the racks.

18/09/18 14:00 UTC 16:00 Paris

Our team is working on the racks to configure the servers (IPMI) inside. It might still take a few hours to be back on, we will update the status accordingly with new updates.

Monday 17th September 2018

DC5 Network [DC5] Services unavailability in room 1-T rack A1

Our teams identified a network outage in DC5 room 1-T rack A1 in the following blocks : A - B - C - D - E

We are fully focused on addressing the issue and will keep you updated through this status as soon as we have additional information.

Thank you for your patience.

Update 14:25 UTC : Issue has now been fixed, services are live again. Thank you for your patience.

Sunday 16th September 2018

No incidents reported

Saturday 15th September 2018

AMS1 Switch unavailability

The switch on this block is unavailable :

Datacenter : AMS1, Room : Hall 2 E-E, Rack : E20, Block : L, Position : 1

--

19/09/2018 14:45 UTC (16:45 Paris time)

A hardware piece requires attention and might need to be replaced, we will update the status when we have more information about it.

17/09/2018 08:30 UTC (10:30 Paris time)

We are still investigating and will keep you updated as soon as possible.

15/09/2018 17:30 UTC (19:30 Paris time)

Issue is related to the network, our team is still investigating to find the cause and resolve it as quickly as possible.

Friday 14th September 2018

AMS1 Network Electrical incident in AMS

We are currently expecting an electrical issue in the hall: 2 E-E, Rack : E20 Our engineers are already investigating on fix as fast as possible.

--

Update 12:27 UTC : Issue is now fixed, services are online again. We are sorry for any inconvenience caused.

DC5 Network issue in DC5

We are currently expecting a network issue in room : 1-T 1-T, Rack : A17 Our engineers are already working on fix and we will update you as soon as possible.

We are sorry for any inconvenience caused.

--

Update 12:20 UTC : Issue is now fixed, services are back online. Thank you for your patience.

--

Update 13:00 UTC : Some blocks are still unavailable on A17, our network team is restoring services one block after another.

DC5 DNS resolvers issue

Our team identified an issue regarding DNS resolvers from DC5. All servers from DC5 using our cache NS servers might experience issues while we are working to restore them.

We will keep you updated through this status.

--

Update 09-18-18 12:25 UTC

Issue has been fixed, there was a configuration issue following the migration.

Update 09-18-18 11:38 UTC

We are experience an outage regarding DC5 NS servers following our hardware migration. Our teams are currently investigating the issue in order to restore them as soon as possible.

Update 09-14-18 12:50 UTC

Issue has now been fixed, NS cache servers are available again on DC5. Thank you for your patience !

DC5 [DC5] Maintenance of the Low Voltage Master Board

A maintenance of the DC5 Low Voltage Master Board will happen today 14-sep-18 6:00amUTC => Estimated duration : 90mins

La magic room will be powered by generators during this time. There should be no impact for customers.