A "smoking gun" ...

Mike Shoemaker and I examined the brain this morning for clues about what went wrong on Oct 4th. Almost immediately we found that one of the two fuses was blown.

How this could have happened: there is a "screw-down block" at the top of the brain on one side with "+12V" terminals arrayed on one side in red and "Ground" terminals on the other side in black. We think something touched this block in a way that completed the circuit and blew one of the two fuses. In my opinion the most likely cause was the windbird plug, which includes one unshielded wire that acts as the instrument's ground. This wire was originally wrapped in electrical tape but I had to expose everything on that that last visit (Saturday, August 27th) to rewire the plugs for the windbird because it wasn't working properly, and evidently I did not rewrap it before leaving. Then on Oct 4th, David must have shifted the tangle of wires enough to short-circuit that board in the moments before he powered off the station with the on/off switch. It could have been shorted by other things during that Oct 4th visit (a little seawater, a screwdriver, other metal tools) but the windbird-plug explanation is plausible.

The effect: with the whole drill-down block powerless, none of the electronics would have had power -- not the brain, not the modem, not the radio, and none of the instruments (including the PacIOOS CTD). When the power switch was turned back on again, the only working part would have been the batteries, the charger-controller (with a red light to indicate charging), and the solar panels. So we think that the station continued to charge its batteries every day until brain removal in January but nothing else was working during that time (except the PacIOOS CTD, running off its own battery power).

What comes next: Shoemaker has the brain in his lab and will be checking it over for (other) signs of trouble. We will be wrapping that windbird plug in tape again, and Shoemaker has a plan to cover that screw-down block with some kind of shield to avoid any repeat of this problem. When he's done I will update the programming (to the latest version and put this brain on our lab's roof with some surface instruments. This should confirm that the electronics are all operational and that the charger-controller can still charge a battery. I'll let that run for a few days and then we'll box up the brain and ship it back to Saipan again. We'll be sure to ship spare fuses back to Saipan as well.

Mike J+

Troubleshooting continues

As of Tuesday afternoon (Saipan time) on January 31st, 2012, the station's control unit or "brain" is on its way back to Miami for examination and possible repair by AOMLers.

This is part of a three-step approach to bringing the station back online:

1. Finding out why the cellular modem went offline (Oct 2nd).
2. Finding out why the station lost power (Oct 4th).
3. Finding out why the RF radios aren't working.

For (1) the cellular modem, it does not appear to be malfunctioning. It was reachable on land via wireless connection several times during troubleshooting operations jointly undertaken by David Benavente, Steven Johnson and Ross Timmerman last November. It merely seems to have gone spontaneously offline on October 2nd, and then later its station power supply seemed to have failed by November 29th. The only thing that stands out from its diagnostics is the unusually high number of "system resets" noted by Ross.

One possible avenue of investigation concerns an AT&T cellular modem that AOML is operating here in South Florida at a test station in Fort Lauderdale. On January 23rd I noticed that this modem's communications were undergoing frequent resets in a way reminiscent of the Docomo modem had done in Saipan before failing. I have relatively easy access to this modem if it should fail, and I've contacted Campbell Scientific support to try to interest them in looking at our software logs from this test station.

For (2) the station's loss of power, we will see if we learn anything from our examination of the brain. [It's entirely possible that we will find that everything appears to be normal, in which case we can only return the brain to Saipan and attempt reinstallation.] Earlier this week we successfully concluded another test of the electronics when David powered up both the datalogger and the cellular modem on the workbench, and I was able to connect to the logger from our systems here in Florida. I was able to download all of the 1-day, 60-minute, 6-minute and 1-minute data tables while this test was still running. So our analysis will initially focus on the wires and connections that supply power to all of these components.

Regarding (3) the RF radios, we have a success to report. David and Steven were able to get their RF radio connection working on land, after reviewing some configuration settings that I suggested might be causing problems. This is a very important step because it means they will have a way to connect to the station from a laptop in the boat, when it comes time to reinstall the brain. They will know immediately, after powering up the station, whether it is running and whether all of the instruments are properly connected. [There is still no way to tell from the boat whether the cellular modem is working properly. We might be able to brainstorm something if they have a wireless internet connection out there, or just coordinate their reinstallation so that they have someone to call on land who can try to connect to the modem.]

Mike Jankulak