diagnosis: a dissolved connector pin

Mike Shoemaker makes this report about the likely cause of the station's power problems:

These are the preliminary results of the examination of returned instruments from Saipan. The attached pictures are from the PAC IOOS CT main power Y-cable which provides power to the instrument and the attached water pump along with communication. The connector that attaches to the fish bite cable adapter has had a pin arced off. This pin is still embedded in the adapter. I speculate that this may have been caused by bringing the instrument up to the surface to disconnect from the station and connect to a laptop for download then being re connected without properly drying the connectors and lubing with silicon to form a proper seal. In addition to these problems the strain relief connector is missing from the main fish bite cable, this also could be the cause of the seawater intrusion as that the main fish bite cable megs out @ infinity but when the adapter is attached it megs out @ 1.5meg.

The BIC _DP cable megs out @ anywhere from 400meg to 600meg depending on which of the 4 conductors that you attach to and the CTD_DP cable megs out @ 200meg across all 4 conductors. I speculate that from the earlier e-mail from Mr. Benavente about the three prong spears used by local fisherman that the cables were punctured from these and that seawater intrusion has caused these high resistance short circuits.

My next step will be to see if Mr. Bishop has any of the Y-cables left from the Sea Bird CTD's that we transferred to his group to test the PAC IOOS CT and down load Gordon Walker's data along with testing the other returned underwater instruments. That is unless Gordon can expedite a shipment of a cable from his spares.

As Shoe explains, we now believe with a high degree of certainty that the station power problems were caused by a connector failure.  The ground pin from the PacIOOS Y-Cable is missing in the connector that attaches to the CREWS pigtail adapter.  Both this Y-Cable (a SeaBird part) and the pigtail adapter (custom-made by CREWS) must be replaced.

Next steps will include downloading the PacIOOS on-board memory and re-testing of the "brain" components.  The underwater sensors that have been retrieved, since they were deployed on the station for more than a year, will now be sent back to their manufacturers for evaluation and recalibration, and the station will next go live with its alternative set of sensors (most of which have been in storage in Saipan since August of 2011).

Clicking on any of the images below should load a full-sized version.

Mike J+

station equipment reaches Miami

Following the last blog update on September 5th (which mentioned how the station had been completely offline since August 24th), the station resumed its brief sunlit periods of activity for a few hours a day until September 19th.  The last coherent report from the PacIOOS CTD, however, occurred on September 10th.

I'm not sure of the details of timing and personnel, but I believe David Benavente and Steven Johnson were involved in an operation on September 27th, 2012, to retrieve all of the station underwater sensors and cables as well as the "brain" package.  These, along with the "groundtruth" CT, were packed up and shipped to AOML in Miami on September 28th.  The shipment arrived here on October 1st, 2012.

David Benavente had made the following observation about removing the instruments from the station:

So just thought you should know. While I was attempting to shut down the PACIOOS Seabird I removed the male/female plugs to connect it to the serial port. I noticed that one of the copper nodes on the male side had corroded off. Im not sure if its the ground wire, but I thought I should let you know. I decided to remove the batteries from the PACIOOS CTD so that it cannot continue to collect data.

Mike Shoemaker will be leading the effort to diagnose the power losses that have plagued the station in the past few months.

Mike J+

new plan: equipment recovery for analysis in Miami

As of today, September 5th, 2012, the decision has been made to recover all of the station equipment and ship it to Miami for analysis and repair.  An alternative plan to have Mike Shoemaker fly out to Saipan and work on site with Steven and David has been set aside due to budgeting and logistics difficulties.

After the last blog update, the station continued to come alive for a few hours a day until August 24th, at which time it went entirely offline and remains offline as of the writing of this update.

Mike J+

info from brief resumptions of communications

In the last few days, from August 10th - 13th, there have been four brief periods when communications with the Saipan station have resumed.  This post explains what we have learned from the existence and the contents of those communications.

First let's look at the timing and duration of these periods, speaking in local Saipan time:

• Friday, August 10th, 7:18 am - 7:42 am (5 records)
• Saturday, August 11th, 1:30 pm - 4:48 pm (34 records)
• Sunday, August 12th, 11:24 am - 11:30 am (2 records)
• Monday, August 13th, 9:42 am - 1:54 pm (33 records)

 The record counts I've listed are from the 6-minute data table.

First of all, note that every record in a data table is given a sequential record number.  This makes it possible to identify cases where records are (or are not) consecutive.  Because of this we can say with high probability that the station's datalogger was completely non-functional from the period from July 19th to August 9th (with times in UTC).  There is no chance that this could be yet another communications-only failure, with little impact otherwise to station operations.

The pattern of communications is somewhat consistent with the explanation that, following a severe power loss that to some extent drained the station's rechargeable batteries, the station may be recharging itself and may briefly resume operations during daylight hours.  However there is still no explanation for the initial power loss, or for the previous power drops (May 19th - June 4th, June 26th - July 5th) seen in the data record.

Perhaps worse, this pattern suggests that the intermittent voltage drop may not be over.  Normally if a short-circuit is repaired we would expect to see better and longer "ontimes" with each passing day.  Instead we start with a brief uptime in the early morning (followed by silence during that day's prime daylight hours).  We also see a stronger performance Saturday followed by a weaker performance Sunday.  Monday's communications are longer but they describe power levels that are getting lower throughout the morning and early afternoon.  And on Tuesday, which has already ended in Saipan, there was no resumption of communications at all.

Another problem is that many of the station's instruments are now malfunctioning.  This may be due to electrical problems (an artifact of running them at very low levels) or their calibration files and settings may have become corrupted, leading them to produce reports in a format that the datalogger program was not designed to parse.  A brief rundown of the state of station instruments follows:

The standalone air temperature sensor, barometer, anemometer and electronic compass all appear to be working normally.  These are all analog instruments and do not depend on serial communications in any way.

Both light sensors are damaged.  The surface light sensor continues to produce serial reports of some kind (as evidenced by that sensor's instrument "counts" in the logger) but these reports are apparently filled only with zeroes, day or night, including for temperature and voltage.  The underwater light sensor has been offline since last October and this has not changed.

The Deep CTD (Teledyne) seems to be at least partially operational.  Its conductivity and temperature readings seem reasonable, although its depth reading show an odd 30-cm shift in the past few days and may indicate a problem.

The PacIOOS CTD was producing data reports on Friday and Saturday but since that time its output has been in a format that was unrecognized by the datalogger programming.  It produced a full "status" report (in response to hourly prompting by the datalogger) on August 11th, 2012, at 4:02 UTC.  This report's contents were as follows:

• Year: 2012
• Month: 8
• Day: 11
• Hour: 4
• Minute: 2
• Second: 56
• Serial Num: 1606481
• Num Events: 15
• Volts Main: 7.3
• Volts Lith: 8
• Curr Main: 61.3
• Curr Pump: 283.4
• Curr Ext: 286.8
• Mem Bytes: 1082107
• Samples: 56953
• Sample Free: 3406107
• Sample Len: 19
• Headers: 4

The Vaisala WXT, like the surface light sensor, is apparently producing reports but as recorded by the datalogger these reports are all zeroes.

It is possible that there has been damage to the datalogger, memory unit, serial port units, radio or cellular modem, although so far there is no sign of such damage.

The main problem at this point is not the instrument failures but that of isolating the cause of these voltage drops.  It seems like there have been voltage drops since mid-May, that a severe voltage drop has rendered the station non-functional for three weeks, and that this may be an intermittent problem that is still ongoing.  If we knew or suspected the cause of the power issues then it would simply be a matter of replacing the station instruments with the spares in storage at CRM.  However our first focus must be on diagnosing the underlying cause of the power losses and then repairing it.

Mike J+

Maintenance log: Launched CRM vessel from shore

[From David Benavente's LAOLAO Bay ICON Station Maintenance log -- August 6, 2012:]

Launched CRM vessel from shore. Station was climbed by David B. Upon opening the brain housing all wires and connection plugs were first inspected. No sign of damage was found. Next, Brain hardware was visually inspected. Data logger lights, modem lights, and power supply lights were all illuminated. The inspection did not reveal any wiring problems to the brain.
Underwater cleaning and maintenance of station instruments was also conducted, during this visit.

Station Power Failure

The Saipan CREWS station went offline as of 22:36:35 UTC on Thursday, July 19th, 2012.  In local Saipan time this is Friday morning, July 20th, at 8:36 am.  Miami time, it is Thursday night, July 19th, at 6:36 pm.

Gordon Walker (who is taking Ross Timmerman's position with PacIOOS) sent out this information on Monday, July 23rd:

We were able to connect to the modem at 9:30 am (locate Saipan time) and everything looked okay except for the voltage.  It is at 5.76 V.  It seems as though the power for the entire array is extremely low.  This may be the reason for the interruption.  We will continue to monitor the modem on our end.

Unlike previous outages, which were believed to be primarily caused by modem or cellular network problems, this latest outage is clearly due to a loss of power on the station.  Consider the graph of minimum (blue), average (red) and maximum (green) hourly datalogger voltages, plotted against Julian Day, for all of 2012, at right (click on the image for a larger version).

A CREWS station's normal operating voltage cycles between 12.5 V and 14 V, with peak voltages during the sunlight hours when the solar panels are supplying power and lowest voltages overnight when they are not.

A close examination of this station's power levels in 2012 shows three periods where the station did not appear to be reaching its usual daylight voltage peaks.  The first occurred from roughly May 19th to June 4th; the second from June 26th to July 5th; and the third began on July 12th and ended with the station's loss of power.

That third drop, where the station power level fell below 10 V before communications with the station entirely ceased, has happened before in CREWS history, though it is rare.  It happened once in Jamaica, in June/July 2008, when that station's "deep" light sensor's bulkhead connector was broken (we think this was caused by a period of strong currents and the sensor's cable being insufficiently tied down).  In that case the failed sensor was removed after two days, and the station resumed communications about 14 days later when its batteries were again sufficiently charged.  More or less the same thing happened in Puerto Rico, twice, once in April of 2010 and once in July of 2011.  Both of these latter outages were caused by flooded light sensors, at least one of which was caused by a puncture in the sensing surface of the instrument.

Before I continue, I'll just emphasize that what happened in Saipan on July 19th is nothing like the outages that this station has experienced before -- the first failure on October 2nd, or the more recent outages (April 12th - 19th, those few days in early May, and May 12th - 29th).  Those were quite clearly caused by problems with the cellular modem or the Docomo network.  They were communications outages, only.  After communications resumed, the data record showed that the station continued to operate (and store data locally) after we lost contact.  [In the October incident, the station continued to operate only for two more days, when a blown fuse up top took it out completely.]  In this case, to be clear, we believe all station functions to be non-operational, with the possible exception of the battery-powered PacIOOS CTD.

However, this current outage is somewhat different from the power losses we've seen before at Jamaica and Puerto Rico, for two reasons.  In the current case, there is a strange history of lower voltage levels for days or weeks at a time, which were then followed (the first two times) by a return to normal power levels.  This current problem, it seems, is somehow intermittent, which does not lend itself to explanation by something irreversible like a flooded instrument.

The second difference here is that those previous incidents left enough clues for us to determine with high probability what had failed.  Specifically, in the previous cases there was a measured voltage drop in one instrument only, which was then followed (in a matter of days, or hours) by the station's complete power loss.  This current Saipan power loss does not include any such hints in the power levels of its instruments.  This may be because one of those instrument (the underwater light sensor, as it happens) is already incommunicado.  Perhaps that light sensor has a loose wire in its communications but continues to take a full power/ground feed from the station.  If so, it could flood without warning and cause the failure of the entire station as happened in Jamaica and Puerto Rico.

However, those previous periods of voltage loss and recovery remain unexplained, and this suggests that a flooded instrument may not turn out to be the cause in this case.

Mike J+

Maintenance log: clean SeaBird lens cleaner

[From David Benavente's LAOLAO Bay ICON Station Maintenance log -- July 5, 2012:]

Objective was to revisit the station and thoroughly clean SeaBird lens cleaner. David B had a scheduling conflict he joined the group later. On this visit Copper screens were replaced. They appeared to be punctured; the puncture marks looked as if they had come from a three prong spear, commonly used by fishermen.