Friday, June 7, 2013

Modem Outage, May 31st - June 5th

This time the station went offline on Friday, May 31st at 9pm ChST, or local Saipan time.

I sent the following email to Gordon on June 3rd:
I wonder if you might be able to ask Docomo for another 'purge' like they did before?

The modem went offline on Friday the 31st at about 1100 UTC (700 Miami, 2100 Saipan, 100 Honolulu). It's starting to seem like it's most likely to go offline on Saipan-local Friday evening, which is of course the worst timing since the earliest you and Docomo are both in the office next is Saipan-local Tuesday morning (your Monday afternoon). I guess if this pattern holds then Docomo might find it useful in one day tracking down a root cause.
Gordon sent the following (edited) reply about 3 hours later:
I just got off the phone with [someone] from Docomo. He said that sometimes when the firmware is updated the settings (provisional) for the device get altered. He asked that I send him information on the device and the firmware version the device was upgraded to. In the meantime they will try to push the data through to get the system up and running again. Also, they will track the device and see if they can figure out what's going on.
This was at about 10am Saipan time on Tuesday, June 4th.  In fact this action by Docomo only brought the modem back online again for about 15 minutes, long enough for the station to catch us up on its one-hour and 6-minute data tables but not long enough to completely download the rest of its data.

I notified Gordon of this and he again contacted Docomo for another purge, which brought the station back online again on Wednesday, June 5th at 10am Saipan time.

There was another 3-h outage later that same day (from 7:30pm to 10:30pm Saipan time, Wednesday the 5th) but as far as I'm aware this outage did not require any intervention from Docomo to bring the station back online again.

In the days after this longer outage, Gordon shared the following explanation from a Docomo account executive:
I’ve been in contact with our engineering department to troubleshoot this issue. We’ve purged the service, hoping to force the device to reconnect again. Let me know if you were able to connect to the device again.

As for what may be causing this to disconnect. We suspect that high voice traffic in this area is what disconnects the device from the network. And our network prioritizes voice calls over data use. You mentioned that you noticed the device would disconnect on Fridays. This may be when the golf course nearby gets busy, and voice usage may increase. We will be adding another channel to our site that covers this area, with hopes that this would prevent the device from disconnecting. We plan to have this in place by next week Monday, our time. Which would be Sunday in Hawaii.

I apologize for the inconvenience. I’ll keep in touch with you while we work to resolve this. And let me know if anything else.
This Docomo update was dated Thursday, June 6th at 5:41pm.

Mike J+