État des services

  • Etat Closes
  • Pourcentage achevé:
  • Type de tâche Incident
  • Catégorie Backbone
  • Sévérité Haute
  • Priorité Normale

FS#930 - [DC3] Linecard failure ASR

We have a 12x100g linecard with a bug that needs to be reloaded to recover,
Cisco is investigating the root cause of the bug, meanwhile we will reload the card to recover the situation.

a few traffic perturbation can be expected while the card reloads which should take a few minutes

Cette tâche ne dépend d'aucune autre tâche.

Close par  mikmak (mikmak)
Wednesday, 19 July 2017, 15:40 GMT
La raison de la clôture est:  Fixed
Thursday, 29 June 2017, 14:22 GMT
card is reloading
Thursday, 29 June 2017, 14:31 GMT
reload completed, bug still present :(
we are sending debug to Cisco and waiting for their feedback
Thursday, 29 June 2017, 17:47 GMT
After 2 hours of debug with Cisco, their current conclusion is that the issue propagated to another Linecard,
we will reload this card in the morning and check again
Friday, 30 June 2017, 06:13 GMT
we are reloading the linecard
Friday, 30 June 2017, 06:31 GMT
card reloaded
Friday, 30 June 2017, 08:52 GMT
with Cisco developers help we found a workaround for our bug, we are still investigating/confirming root cause and confirming action plan for a final fix. Cisco already has a bug identified which is probably the one we are hitting (patches not yet available on our version though)
Wednesday, 05 July 2017, 17:53 GMT
Cisco's conclusion is that we hit 2 different known bugs (but not fixed in our version) (actually some kind of variations of these bugs, which was bothering them a bit)
the workaround is working fine but we cannot consider the routers to be safe from this bugs occurring again,
to be on the safe side, we are preparing an minor version upgrade to solve the root issues,
we expect to reboot 1 ASR in the morning in the coming days, traffic impact should be low, as it will be rerouted on the 2nd ASR during the upgrade
Thursday, 06 July 2017, 05:32 GMT
we are reloading the first ASR
Thursday, 06 July 2017, 06:00 GMT
reload complete
Friday, 07 July 2017, 06:16 GMT
we are reloading the 2nd ASR
Friday, 07 July 2017, 06:48 GMT
reload complete