saipan station offline again (apr 12)

[An excerpt from an email sent from Miami on Friday, April 13th, 2012.]

Bad news, the data feed from the CREWS station has stopped again.

This occurred (in the various time zones of relevance) at:

Thu Apr 12  5:10:35 (Honolulu / HAST / -1000)

Thu Apr 12 11:10:35 (Miami / EDT / -0400)

Thu Apr 12 15:10:35 (UTC / +0000)

Fri Apr 13  1:10:35 (Saipan / ChST / +1000)

So it was it the early hours of the morning, local time, Friday the 13th.  I see no signs of trouble in the data feed -- no voltage spikes or drops, no drop in barometric pressures or rise in winds, no instruments going suddenly or suspiciously offline.

So my best guess right now is that the modem went silent again.  Recall that we still haven't explained why that happened on October 2nd, although we do know why the station itself died on October 4th.

Mike J+

Maintenance log: Station cleaning with DEQ and CRM

[From David Benavente's LAOLAO Bay ICON Station Maintenance log -- April 12, 2012:]

Station cleaning with DEQ and CRM. There was a three-way objective to this trip. MMT had to accomplish a site survey, collect water quality samples and clean the ICON Station. The team broke into smaller groups. Ben C. assisted by Jose Q. were tasked with cleaning the ICON Station. When our surveys were complete I Inspected the station; Ben C. reported that he replaced the copper screens on the Deep CTD.

Station Update: Data Feeds Resuming

The Saipan CREWS station's "brain" (control unit) was reinstalled by David Benavente and colleagues on Monday, March 19th, 2012, as described elsewhere on this blog.

David & co. did a fantastic job, particularly considering that the brain-reconnection training we had hoped to give them last summer wasn't possible due to a combination of terrible seas and missing radio antennas. We have been relying on email (and my lengthy documents of instruction) for training, for everything from software installation and radio communications to installing and connecting the station's hardware.

Their March 19th visit had to be cut short and at the time they left, they reported that the Deep BIC (light sensor) and Deep CTD both seemed to be offline. In fact, there is a delayed startup for the CTD and it can take 6 or 12 minutes (or sometimes longer) for it to begin producing data. When I later examined the data feed, the Deep CTD was communicating normally.

The Deep BIC is offline. It is the only communications failure following the station's recovery. This could be caused by any number of wiring problems at the top of the pylon. I would not consider this to be a hugely urgent matter, but during the next visit David & co. may want to open up the top of the pylon and visually inspect the Deep BIC plugs -- are they connected properly, are they pushed together all the way, is there any sign of loose or broken wires on either side?

The other anomaly in the data stream is that the salinity numbers from the Deep CTD are not tracking properly with those from the Shallow/PacIOOS CTD. I would tend to believe what the Shallow/PacIOOS CTD is saying, particularly since it has recently been retrieved for memory download and battery replacement, and presumably was examined and cleaned at that time. The local team might want to examine the Deep CTD more closely during the next visit, give it a good cleaning and maybe replace its copper screens if they have started to dissolve.

Also, it would be a good idea to do a full station cleaning including the connection of the "groundtruth" CT sensor for the required three hours. This would give us another set of salinity numbers to confirm which of the two CTDs are inaccurate.

In addition, there is still the "extra" or "shallow" CT. We had originally intended to install this CT permanently in August but the instrument we'd planned to use was nonfunctional when retrieved from storage. The shipment that returned the "brain" to Saipan also included a replacement CT and this can be deployed at any time using the cables that were installed and connected last summer.

The data stream shows one short blip from the Shallow/PacIOOS CTD on about March 27th. I am assuming that this is when the Shallow/PacIOOS CTD was temporarily disconnected for memory download and battery replacement.

In the past few days, I have started up all of the Saipan data feeds again. The most recent 24/72 hours of data are updated hourly/daily on our web site:

http://www.coral.noaa.gov/static/data_llbp7_All_12.html

http://www.coral.noaa.gov/static/data_llbp7_All_72.html

The data feed to the National Data Buoy Center (NDBC) has been restarted. As of this writing, oceanic measurements are already populated on the NDBC site and a configuration error with the meteorological data has been corrected so those data should start loading shortly:

http://www.ndbc.noaa.gov/station_page.php?station=llbp7

The feed of CTD data from AOML to PacIOOS is under development and a version of that feed began running yesterday.

And finally, data are loading into CHAMP's "ecoforecast" page here:

http://ecoforecast.coral.noaa.gov/index/0/LLBP7/station-home

Congratulations to David B. and the rest of the team in Saipan on a job well done!

(signed)

Mike Jankulak, AOML, Miami

And its ON:

On the Morning of March 19^th Three MMT members (John Iguel, Ryan Okano & I) loaded up CRM’s small zodiac with The Laolao Bay Pylon Stations “brain”. With only a short timeframe to work, due to tidal constraints, we hurried out to the bay, pushed our small boat over the reef and headed for the station.

Once at the station everything seemed to go according to plan. I geared up and climbed to the top of the station. After months of reading through extraction and installation instructions provided by Mike J, I had clearly developed a system for keeping track of the wiring. After about ten minutes of set-up I signaled Ryan to send up the Brain. Once the Brain was at the top of the station all my focus was set on its installation, which to my surprise went by very quickly. Upon switching the brain “On” I was excited to see that all its components were lighting up and blinking normally.

Back on the boat, with my laptop out, we checked to see if the brain was working properly. Although everything seemed to be working more diagnostics are still very much needed. I felt a great sense of pride upon reinstalling the Brain, and I am so grateful for all the help from NOAA AOML, PACIOOS and the MMT. Special thanks to Mike J., and Mike S. from NOAA AOML.

A "smoking gun" ...

Mike Shoemaker and I examined the brain this morning for clues about what went wrong on Oct 4th. Almost immediately we found that one of the two fuses was blown.

How this could have happened: there is a "screw-down block" at the top of the brain on one side with "+12V" terminals arrayed on one side in red and "Ground" terminals on the other side in black. We think something touched this block in a way that completed the circuit and blew one of the two fuses. In my opinion the most likely cause was the windbird plug, which includes one unshielded wire that acts as the instrument's ground. This wire was originally wrapped in electrical tape but I had to expose everything on that that last visit (Saturday, August 27th) to rewire the plugs for the windbird because it wasn't working properly, and evidently I did not rewrap it before leaving. Then on Oct 4th, David must have shifted the tangle of wires enough to short-circuit that board in the moments before he powered off the station with the on/off switch. It could have been shorted by other things during that Oct 4th visit (a little seawater, a screwdriver, other metal tools) but the windbird-plug explanation is plausible.

The effect: with the whole drill-down block powerless, none of the electronics would have had power -- not the brain, not the modem, not the radio, and none of the instruments (including the PacIOOS CTD). When the power switch was turned back on again, the only working part would have been the batteries, the charger-controller (with a red light to indicate charging), and the solar panels. So we think that the station continued to charge its batteries every day until brain removal in January but nothing else was working during that time (except the PacIOOS CTD, running off its own battery power).

What comes next: Shoemaker has the brain in his lab and will be checking it over for (other) signs of trouble. We will be wrapping that windbird plug in tape again, and Shoemaker has a plan to cover that screw-down block with some kind of shield to avoid any repeat of this problem. When he's done I will update the programming (to the latest version and put this brain on our lab's roof with some surface instruments. This should confirm that the electronics are all operational and that the charger-controller can still charge a battery. I'll let that run for a few days and then we'll box up the brain and ship it back to Saipan again. We'll be sure to ship spare fuses back to Saipan as well.

Mike J+

Troubleshooting continues

As of Tuesday afternoon (Saipan time) on January 31st, 2012, the station's control unit or "brain" is on its way back to Miami for examination and possible repair by AOMLers.

This is part of a three-step approach to bringing the station back online:

1. Finding out why the cellular modem went offline (Oct 2nd).
2. Finding out why the station lost power (Oct 4th).
3. Finding out why the RF radios aren't working.

For (1) the cellular modem, it does not appear to be malfunctioning. It was reachable on land via wireless connection several times during troubleshooting operations jointly undertaken by David Benavente, Steven Johnson and Ross Timmerman last November. It merely seems to have gone spontaneously offline on October 2nd, and then later its station power supply seemed to have failed by November 29th. The only thing that stands out from its diagnostics is the unusually high number of "system resets" noted by Ross.

One possible avenue of investigation concerns an AT&T cellular modem that AOML is operating here in South Florida at a test station in Fort Lauderdale. On January 23rd I noticed that this modem's communications were undergoing frequent resets in a way reminiscent of the Docomo modem had done in Saipan before failing. I have relatively easy access to this modem if it should fail, and I've contacted Campbell Scientific support to try to interest them in looking at our software logs from this test station.

For (2) the station's loss of power, we will see if we learn anything from our examination of the brain. [It's entirely possible that we will find that everything appears to be normal, in which case we can only return the brain to Saipan and attempt reinstallation.] Earlier this week we successfully concluded another test of the electronics when David powered up both the datalogger and the cellular modem on the workbench, and I was able to connect to the logger from our systems here in Florida. I was able to download all of the 1-day, 60-minute, 6-minute and 1-minute data tables while this test was still running. So our analysis will initially focus on the wires and connections that supply power to all of these components.

Regarding (3) the RF radios, we have a success to report. David and Steven were able to get their RF radio connection working on land, after reviewing some configuration settings that I suggested might be causing problems. This is a very important step because it means they will have a way to connect to the station from a laptop in the boat, when it comes time to reinstall the brain. They will know immediately, after powering up the station, whether it is running and whether all of the instruments are properly connected. [There is still no way to tell from the boat whether the cellular modem is working properly. We might be able to brainstorm something if they have a wireless internet connection out there, or just coordinate their reinstallation so that they have someone to call on land who can try to connect to the modem.]

Mike Jankulak

brain retrieval and data file analysis

Again the background: the Saipan CREWS station lost communications (October 2nd), an attempt was made to power-cycle the cellular modem (October 4th or 5th) followed by the retrieval of that modem (November 14th or 15th) for testing and evaluation, a land-based test of the modem near the station (November 21st) and the reinstallation of the cellular modem at the station (November 28th or 29th). This was followed by an unsuccessful attempt to connect to the station by radio (December 15th).

The only diagnostic step left was to retrieve the station's control unit (or "brain"). This unfortunately would have the effect of shutting down all station operations except for the PacIOOS-supplied CTD, which has its own battery backup. However at this point we still had no assurance whatsoever that the station had continue to operate in any capacity after its initial loss of communications on October 2nd.

On December 19th (though mindful of the disruptive nature of the holiday season) I sent out detailed instructions to David Benavente (Coastal Resources Management, Saipan) and Steven Johnson (Division of Environmental Quality, Saipan) on how to safely disconnect all power and instruments at the station and remove the brain. On January 3rd I followed up with instructions on how to recover the station's locally-stored data files once the brain had been retrieved.

On Thursday, January 12th at 8:30am (Saipan time) Steven reported some good news:

David and I were able to retrieve the brain out of the station yesterday. We will start doing some preliminary trouble shooting today. We will keep you posted on our progress.

This was followed later on at 3:39pm with a message from David saying:

So I connected to the control unit and downloaded data for the first TAB0/ TAB1 and TAB2. I uploaded them on to the AOML ftp site. I didn't have to enter a username when uploading the files so I wasn't sure whether they had gotten through. Just let me know if they didn't and I'll try again.

Oh another thing that I noticed while retrieving the brain was that there was a bit a moisture inside the grounding plug. As I pulled the two ends apart a drop of liquid fell onto my hand. Not sure what the implications of that are but it seemed odd because everything else was dry. Well thats it for now, I'll continue to download the other files and send them over. Let me know if the files downloaded correctly.

This represented the first new data report from the station since it lost communications on October 4th. I will try to be very clear about what I have learned:

1. The station has been completely offline since early October, 2011. This of course is a crushing disappointment for all of us.
2. The biggest surprise is that the data stream ends on October 4th, not on October 2nd when communications were lost. There are no indications of unusual circumstances in the data record at the time when the station initially lost communications. This initial loss of communications took place at UTC Sunday, October 2nd 19:10 (in Saipan time this is at 5:10am on the morning of Monday, October 3rd).
3. This means that the initial loss of communications is still unexplained. My guess would be either a gradually-worsening loose power connection or some kind of progressive failure of the cellular modem. Our best evidence about this failure remains Ross Timmerman's discovery of the unusually-large number of system resets by the modem (described in this previous blog posting).
4. The station's data record ends at UTC Tuesday, October 4th 4:12 (in Saipan time this would have been at about 2:12pm on Tuesday, October 4th). Again, there is no indication whatsoever in the data record of any problems leading up to the moment of total systems failure. The station appears to have been operating perfectly for those last two days except for its loss of communications. I have gone instrument by instrument and diagnostic by diagnostic over every retrieved data point and the station appears to have been operating perfectly up to the very last minute.
5. Thus far I have only seen the station's 1-day, 60-minute and 6-minute data tables. There may be some further details to be gleaned from the other three data tables (1-minute, 30-second and 5-second). But based on what I've seen so far, there are not likely to be any significant revelations in these other data tables.
6. There is no possibility of data corruption (i.e., it is not possible that the station continued to operate normally after October 4th but we merely failed to recover its complete data record) because the datalogger numbers its records and none are missing.

One question remains to be investigated: when exactly did was the modem power-cycled? To my mind the most likely explanation is that the cellular modem failed for reasons unknown and then some further problem was accidentally introduced when the stationtop was opened up during that October 4th/5th power-cycling visit. Given that there is no sign of station or instrument distress up to the moment of failure, the most likely explanation is human intervention.

This, sadly enough, is the risk that we take every time that we open up the CREWS station top and work with its innards. Normally our "insurance policy" is to connect to the station via radio from the boat every time after closing up the station top. But unfortunately we did not have the time, weather, equipment and software that we needed to configure and test the radios during installation in August. This left the on-site maintenance team without their most important tool.

The takeaway from all this is that we probably have a wire (or many wires) pulled loose somewhere. We know that the datalogger probably lost power on October 4th and has never regained it. We know that the cellular modem did not appear to have power when it was reinstalled on November 28th or 29th. We also have reason to think that some of the other instruments, possibly the SIO4 serial ports or the radio, have had at least intermittent power since October 4th, given David's report of seeing lights blinking on November 28th or 29th.

So the next thing to try would be a visual examination of the "brain" unit for signs of loose wires. This may be followed either by attempts to reinstall the brain in the station or perhaps by shipping the entire "brain" package back to Miami for evaluation. I will update this blog again when we have decided on our next step.