Tuesday, July 24, 2012

Station Power Failure

The Saipan CREWS station went offline as of 22:36:35 UTC on Thursday, July 19th, 2012.  In local Saipan time this is Friday morning, July 20th, at 8:36 am.  Miami time, it is Thursday night, July 19th, at 6:36 pm.

Gordon Walker (who is taking Ross Timmerman's position with PacIOOS) sent out this information on Monday, July 23rd:
We were able to connect to the modem at 9:30 am (locate Saipan time) and everything looked okay except for the voltage.  It is at 5.76 V.  It seems as though the power for the entire array is extremely low.  This may be the reason for the interruption.  We will continue to monitor the modem on our end.
Unlike previous outages, which were believed to be primarily caused by modem or cellular network problems, this latest outage is clearly due to a loss of power on the station.  Consider the graph of minimum (blue), average (red) and maximum (green) hourly datalogger voltages, plotted against Julian Day, for all of 2012, at right (click on the image for a larger version).

A CREWS station's normal operating voltage cycles between 12.5 V and 14 V, with peak voltages during the sunlight hours when the solar panels are supplying power and lowest voltages overnight when they are not.

A close examination of this station's power levels in 2012 shows three periods where the station did not appear to be reaching its usual daylight voltage peaks.  The first occurred from roughly May 19th to June 4th; the second from June 26th to July 5th; and the third began on July 12th and ended with the station's loss of power.

That third drop, where the station power level fell below 10 V before communications with the station entirely ceased, has happened before in CREWS history, though it is rare.  It happened once in Jamaica, in June/July 2008, when that station's "deep" light sensor's bulkhead connector was broken (we think this was caused by a period of strong currents and the sensor's cable being insufficiently tied down).  In that case the failed sensor was removed after two days, and the station resumed communications about 14 days later when its batteries were again sufficiently charged.  More or less the same thing happened in Puerto Rico, twice, once in April of 2010 and once in July of 2011.  Both of these latter outages were caused by flooded light sensors, at least one of which was caused by a puncture in the sensing surface of the instrument.

Before I continue, I'll just emphasize that what happened in Saipan on July 19th is nothing like the outages that this station has experienced before -- the first failure on October 2nd, or the more recent outages (April 12th - 19th, those few days in early May, and May 12th - 29th).  Those were quite clearly caused by problems with the cellular modem or the Docomo network.  They were communications outages, only.  After communications resumed, the data record showed that the station continued to operate (and store data locally) after we lost contact.  [In the October incident, the station continued to operate only for two more days, when a blown fuse up top took it out completely.]  In this case, to be clear, we believe all station functions to be non-operational, with the possible exception of the battery-powered PacIOOS CTD.

However, this current outage is somewhat different from the power losses we've seen before at Jamaica and Puerto Rico, for two reasons.  In the current case, there is a strange history of lower voltage levels for days or weeks at a time, which were then followed (the first two times) by a return to normal power levels.  This current problem, it seems, is somehow intermittent, which does not lend itself to explanation by something irreversible like a flooded instrument.

The second difference here is that those previous incidents left enough clues for us to determine with high probability what had failed.  Specifically, in the previous cases there was a measured voltage drop in one instrument only, which was then followed (in a matter of days, or hours) by the station's complete power loss.  This current Saipan power loss does not include any such hints in the power levels of its instruments.  This may be because one of those instrument (the underwater light sensor, as it happens) is already incommunicado.  Perhaps that light sensor has a loose wire in its communications but continues to take a full power/ground feed from the station.  If so, it could flood without warning and cause the failure of the entire station as happened in Jamaica and Puerto Rico.

However, those previous periods of voltage loss and recovery remain unexplained, and this suggests that a flooded instrument may not turn out to be the cause in this case.

Mike J+