June Modem Outages, both long and short

As we are finishing up our second month since the Saipan CREWS station was brought back online, some patterns are emerging with the cellular modem outages.  Two patterns in particular.

The first pattern is comprised of shorter-term outages that do not appear to require any manual intervention by PacIOOS or Docomo for resolution.  These include six outages of 1 - 3 hours described in this blog update and an additional 3-h outage mentioned at the end of this blog update.  It also includes the following outages that have not yet been mentioned in these blog updates (with all dates and times expressed in local Saipan time):

• 3 hours, Wed June 5th from 7:30pm to 10:30pm
• 3 hours, Mon June 10th from 5pm to 8pm
• 3 hours, Thurs June 13th from 5am to 8am
• 3 hours, Fri June 14th from 2pm to 5pm

Here there is a longer outage from June 15th - 17th that will be described below, but the intermittent outages continue with the following:

• 1.5 hours, Thurs June 20th from 2am to 3:30am
• 3 hours, Fri June 21st from 4am to 7am
• 3 hours, Sat June 22nd from 1am to 4am

And again this pattern is broken by a longer outage, from midnight on the 22nd/23rd until the present.

The two longer outages mentioned above both began at about midnight on a Saturday/Sunday.  The first began Sat/Sun June 15th/16th and lasted until Docomo could be contacted for another purge at noon on Tuesday the 18th.  The second outage began at midnight Sat/Jun June 22nd/23rd and as of the writing of this update the station remains offline (it is now about 6am on Thursday morning, June 27th, in Saipan).

It would also be possible to pick out a third pattern from the above accounts, that of certain period of perfect or near-perfect communications.  There was one such period from May 25th to May 30th and another from June 5th to June 10th.

The status at present is that Docomo has not yet been contacted by phone concerning the present outage, since Gordon has been busy with fieldwork.  My latest information suggests that Docomo may be contacted later today (Thursday morning in Saipan) to request another purge, so hopefully communications will resume shortly.

Mike J+

Troubleshooting the PacIOOS Turbidity Sensor

The SeaBird CTD contributed to the Saipan CREWS station operations by PacIOOS has two add-on sensors for measuring chlorophyll and turbidity.  Since deployment on April 30th, the turbidity readings have been zero.  Gordon Walker of PacIOOS/Hawai`i has been making plans with Steven Johnson of DEQ/Saipan to troubleshoot and fix this problem.

On June 13th, Gordon sent the following email to Steven with and attached photo:

I hope things are going well for you, David and the other folks at DEQ. Mike J mentioned that you will be heading out to the station this week for a monthly maintenance visit. The turbidity reading on the PacIOOS sensor is effectively 0.0000. Can you take a picture of the bulkhead connectors on the PacIOOS sensor (see attached)? Based on how the sensor is oriented, the bulkhead is currently on the "bottom", facing the ocean floor. It has the metal U guard protecting it. The picture should show each connector on the bulkhead as clearly as possible. Thanks.

Click on this photo to see a larger version.

In a followup message that same day Gordon explained that the FLNTUS may be connected to the wrong bulkhead connector (which would be obvious in the kind of photo requested above).  Later Gordon decided that a better diagnosis could be had by removing the PacIOOS CTD from the station and testing its cable:

It looks like the PacIOOS sensor will have to be retrieved on your trip to the station next week. We need to do a continuity check with a multimeter of the cable that connects the FLNTUS to the CTD, and test the bulkhead connector pins with a streaming external voltage check via the CTD. Can we set up a time for us to Skype once you have retrieved the sensor, installed the dummy plug and brought the sensor back to DEQ? Thanks.

Unfortunately these messages weren't received by DEQ in enough time before their June 19th visit to the station for them to follow through on the plans described.  However, plans are being made for further troubleshooting of the PacIOOS turbidity sensor during the monthly July maintenance visit.

[This update posted by Mike Jankulak.]

June CREWS Station Cleaning

The following update was provided by Steven Johnson, DEQ/Saipan, by email on June 22nd.  It describes a monthly CREWS maintenance visit that took place on Wednesday, June 19th, 2013.  The data record shows that the "groundtruth" CT was connected to the station shortly after 10am local time and removed at about 3:30pm local time, for a total connection time of about 5h18m.  Steven writes:

This past Wednesday DEQ was able to get the boat around Naftan Point and into Laolao Bay for our delayed station cleaning. The team that headed out to the pylon for the cleaning included myself, John Iguel, Ryan Okano and Jose Quan. We were also fortunate to have John Tomczuk from NOAA SEA Team join us. He was on Saipan assisting with the Jurisdictional Capacity Assessment being conducted by NOAA-CRCP and we decided to take him out with us. Tom was instrumental in supporting the CNMI and AOMLs decision in getting us the CREWS station. The least we could do was take him on some dives.

Most everything got accomplished at the station, save for the PacIOOS tasks (more on that later). We successfully attached the groundtruth CT to the station for a hour prior to the cleaning. During that time we completed one of MMTs long-term monitoring sites in the bay. Highlights from that dive included a handful of octopuses on the transect line!

When we returned to the station for the cleaning we took photos before and after and all instruments/cables seem to be in good shape. The chains of the station are starting to get a fair amount of Pocillopora sp. corals growing on it. Once they get bigger I'll be able to ID them for everyone ;) After cleaning the station we left the groundtruth sensor on for another hour before removing it and heading back to the dock.

Work on the PacIOOS sensors wasn't able to be completed because 1) I didn't see Gordon's email until the morning we went out (backed up email inbox from all of my May traveling) and 2) we could not find a multimeter to use to check the voltage of the sensor. This will be a high priority task for MMT during the first week of July. Gordon, please contact me if there are any updates or changes to what you'd like us to do.

[This update posted by Mike Jankulak.]

Modem Outage, May 31st - June 5th

This time the station went offline on Friday, May 31st at 9pm ChST, or local Saipan time.

I sent the following email to Gordon on June 3rd:

I wonder if you might be able to ask Docomo for another 'purge' like they did before?

The modem went offline on Friday the 31st at about 1100 UTC (700 Miami, 2100 Saipan, 100 Honolulu). It's starting to seem like it's most likely to go offline on Saipan-local Friday evening, which is of course the worst timing since the earliest you and Docomo are both in the office next is Saipan-local Tuesday morning (your Monday afternoon). I guess if this pattern holds then Docomo might find it useful in one day tracking down a root cause.

Gordon sent the following (edited) reply about 3 hours later:

I just got off the phone with [someone] from Docomo. He said that sometimes when the firmware is updated the settings (provisional) for the device get altered. He asked that I send him information on the device and the firmware version the device was upgraded to. In the meantime they will try to push the data through to get the system up and running again. Also, they will track the device and see if they can figure out what's going on.

This was at about 10am Saipan time on Tuesday, June 4th.  In fact this action by Docomo only brought the modem back online again for about 15 minutes, long enough for the station to catch us up on its one-hour and 6-minute data tables but not long enough to completely download the rest of its data.

I notified Gordon of this and he again contacted Docomo for another purge, which brought the station back online again on Wednesday, June 5th at 10am Saipan time.

There was another 3-h outage later that same day (from 7:30pm to 10:30pm Saipan time, Wednesday the 5th) but as far as I'm aware this outage did not require any intervention from Docomo to bring the station back online again.

In the days after this longer outage, Gordon shared the following explanation from a Docomo account executive:

I’ve been in contact with our engineering department to troubleshoot this issue. We’ve purged the service, hoping to force the device to reconnect again. Let me know if you were able to connect to the device again.

As for what may be causing this to disconnect. We suspect that high voice traffic in this area is what disconnects the device from the network. And our network prioritizes voice calls over data use. You mentioned that you noticed the device would disconnect on Fridays. This may be when the golf course nearby gets busy, and voice usage may increase. We will be adding another channel to our site that covers this area, with hopes that this would prevent the device from disconnecting. We plan to have this in place by next week Monday, our time. Which would be Sunday in Hawaii.

I apologize for the inconvenience. I’ll keep in touch with you while we work to resolve this. And let me know if anything else.

This Docomo update was dated Thursday, June 6th at 5:41pm.

Mike J+

Battery Update, June 4th

This was another limited-distribution email update about battery levels at the Saipan station.  Two and a half weeks previously we had made the call that we should buy replacement batteries for the station, since we were in a time when such purchases could easily be authorized and future prospects for these batteries were rather uncertain.

However, the wheels of purchasing move slowly and during this time the battery levels continued to strengthen until achieving levels that are entirely normal for a CREWS station running at full strength:

Click this graph to see a larger version.

Based on this new evidence we are postponing our purchase of replacement batteries, indefinitely.  These batteries appear to have survived everything we've thrown at them and are still going strong.

Mike J+