Storage Node Issue

  • Friday January 3rd, 2014
  • By:

On 3 January 2014, at around 12:30am SGT (Singapore time), we detected some technical issues on one of our storage nodes in our Singapore cloud infrastructure, causing I/O issue to servelets utilising the node. We performed an urgent maintenance to resolve the issue, and most of the affected servelets are back online by around 5 am SGT.

However, while still in self-healing mode, the storage node experienced problem again at around 4:45pm SGT, and an urgent reboot of the affected servelet node was done, resulting in I/O slowness on some servelets. The storage system was still in self-healing mode after the urgent maintenance, and it’s fully recovered by around 6 pm  SGT.

We have performed a full investigation and isolated the root cause of the problem to uneven distribution within the storage system, and we have performed the necessary actions to prevent similar problem from happening in the future.

We apologise for the inconvenience caused.