Station back online -- Docomo had network problems

[From an email by Mike Jankulak on Thursday, April 19, 2012:]

The cellular modem was back online this morning when I arrived at work.  Right now our data feeds are still down because the server files are out of sync with what's stored on the logger (this happened last month too), so I'm working on that.

[From an email by Ross Timmerman on Thursday, April 19, 2012:]

Just had a look at AceManager and all the modem settings appear okay. I don't know why the modem went down earlier, but it was definitely network related. I asked Docomo whether they had a network outage, their response below (sent last night):

"Your services were not disconnected.  Although, we may have resolved the problem.  Please try connecting to the device again, and advise if successful or not."

saipan station offline again (apr 12)

[An excerpt from an email sent from Miami on Friday, April 13th, 2012.]

Bad news, the data feed from the CREWS station has stopped again.

This occurred (in the various time zones of relevance) at:

Thu Apr 12  5:10:35 (Honolulu / HAST / -1000)

Thu Apr 12 11:10:35 (Miami / EDT / -0400)

Thu Apr 12 15:10:35 (UTC / +0000)

Fri Apr 13  1:10:35 (Saipan / ChST / +1000)

So it was it the early hours of the morning, local time, Friday the 13th.  I see no signs of trouble in the data feed -- no voltage spikes or drops, no drop in barometric pressures or rise in winds, no instruments going suddenly or suspiciously offline.

So my best guess right now is that the modem went silent again.  Recall that we still haven't explained why that happened on October 2nd, although we do know why the station itself died on October 4th.

Mike J+

Maintenance log: Station cleaning with DEQ and CRM

[From David Benavente's LAOLAO Bay ICON Station Maintenance log -- April 12, 2012:]

Station cleaning with DEQ and CRM. There was a three-way objective to this trip. MMT had to accomplish a site survey, collect water quality samples and clean the ICON Station. The team broke into smaller groups. Ben C. assisted by Jose Q. were tasked with cleaning the ICON Station. When our surveys were complete I Inspected the station; Ben C. reported that he replaced the copper screens on the Deep CTD.

Station Update: Data Feeds Resuming

The Saipan CREWS station's "brain" (control unit) was reinstalled by David Benavente and colleagues on Monday, March 19th, 2012, as described elsewhere on this blog.

David & co. did a fantastic job, particularly considering that the brain-reconnection training we had hoped to give them last summer wasn't possible due to a combination of terrible seas and missing radio antennas. We have been relying on email (and my lengthy documents of instruction) for training, for everything from software installation and radio communications to installing and connecting the station's hardware.

Their March 19th visit had to be cut short and at the time they left, they reported that the Deep BIC (light sensor) and Deep CTD both seemed to be offline. In fact, there is a delayed startup for the CTD and it can take 6 or 12 minutes (or sometimes longer) for it to begin producing data. When I later examined the data feed, the Deep CTD was communicating normally.

The Deep BIC is offline. It is the only communications failure following the station's recovery. This could be caused by any number of wiring problems at the top of the pylon. I would not consider this to be a hugely urgent matter, but during the next visit David & co. may want to open up the top of the pylon and visually inspect the Deep BIC plugs -- are they connected properly, are they pushed together all the way, is there any sign of loose or broken wires on either side?

The other anomaly in the data stream is that the salinity numbers from the Deep CTD are not tracking properly with those from the Shallow/PacIOOS CTD. I would tend to believe what the Shallow/PacIOOS CTD is saying, particularly since it has recently been retrieved for memory download and battery replacement, and presumably was examined and cleaned at that time. The local team might want to examine the Deep CTD more closely during the next visit, give it a good cleaning and maybe replace its copper screens if they have started to dissolve.

Also, it would be a good idea to do a full station cleaning including the connection of the "groundtruth" CT sensor for the required three hours. This would give us another set of salinity numbers to confirm which of the two CTDs are inaccurate.

In addition, there is still the "extra" or "shallow" CT. We had originally intended to install this CT permanently in August but the instrument we'd planned to use was nonfunctional when retrieved from storage. The shipment that returned the "brain" to Saipan also included a replacement CT and this can be deployed at any time using the cables that were installed and connected last summer.

The data stream shows one short blip from the Shallow/PacIOOS CTD on about March 27th. I am assuming that this is when the Shallow/PacIOOS CTD was temporarily disconnected for memory download and battery replacement.

In the past few days, I have started up all of the Saipan data feeds again. The most recent 24/72 hours of data are updated hourly/daily on our web site:

http://www.coral.noaa.gov/static/data_llbp7_All_12.html

http://www.coral.noaa.gov/static/data_llbp7_All_72.html

The data feed to the National Data Buoy Center (NDBC) has been restarted. As of this writing, oceanic measurements are already populated on the NDBC site and a configuration error with the meteorological data has been corrected so those data should start loading shortly:

http://www.ndbc.noaa.gov/station_page.php?station=llbp7

The feed of CTD data from AOML to PacIOOS is under development and a version of that feed began running yesterday.

And finally, data are loading into CHAMP's "ecoforecast" page here:

http://ecoforecast.coral.noaa.gov/index/0/LLBP7/station-home

Congratulations to David B. and the rest of the team in Saipan on a job well done!

(signed)

Mike Jankulak, AOML, Miami

And its ON:

On the Morning of March 19^th Three MMT members (John Iguel, Ryan Okano & I) loaded up CRM’s small zodiac with The Laolao Bay Pylon Stations “brain”. With only a short timeframe to work, due to tidal constraints, we hurried out to the bay, pushed our small boat over the reef and headed for the station.

Once at the station everything seemed to go according to plan. I geared up and climbed to the top of the station. After months of reading through extraction and installation instructions provided by Mike J, I had clearly developed a system for keeping track of the wiring. After about ten minutes of set-up I signaled Ryan to send up the Brain. Once the Brain was at the top of the station all my focus was set on its installation, which to my surprise went by very quickly. Upon switching the brain “On” I was excited to see that all its components were lighting up and blinking normally.

Back on the boat, with my laptop out, we checked to see if the brain was working properly. Although everything seemed to be working more diagnostics are still very much needed. I felt a great sense of pride upon reinstalling the Brain, and I am so grateful for all the help from NOAA AOML, PACIOOS and the MMT. Special thanks to Mike J., and Mike S. from NOAA AOML.

A "smoking gun" ...

Mike Shoemaker and I examined the brain this morning for clues about what went wrong on Oct 4th. Almost immediately we found that one of the two fuses was blown.

How this could have happened: there is a "screw-down block" at the top of the brain on one side with "+12V" terminals arrayed on one side in red and "Ground" terminals on the other side in black. We think something touched this block in a way that completed the circuit and blew one of the two fuses. In my opinion the most likely cause was the windbird plug, which includes one unshielded wire that acts as the instrument's ground. This wire was originally wrapped in electrical tape but I had to expose everything on that that last visit (Saturday, August 27th) to rewire the plugs for the windbird because it wasn't working properly, and evidently I did not rewrap it before leaving. Then on Oct 4th, David must have shifted the tangle of wires enough to short-circuit that board in the moments before he powered off the station with the on/off switch. It could have been shorted by other things during that Oct 4th visit (a little seawater, a screwdriver, other metal tools) but the windbird-plug explanation is plausible.

The effect: with the whole drill-down block powerless, none of the electronics would have had power -- not the brain, not the modem, not the radio, and none of the instruments (including the PacIOOS CTD). When the power switch was turned back on again, the only working part would have been the batteries, the charger-controller (with a red light to indicate charging), and the solar panels. So we think that the station continued to charge its batteries every day until brain removal in January but nothing else was working during that time (except the PacIOOS CTD, running off its own battery power).

What comes next: Shoemaker has the brain in his lab and will be checking it over for (other) signs of trouble. We will be wrapping that windbird plug in tape again, and Shoemaker has a plan to cover that screw-down block with some kind of shield to avoid any repeat of this problem. When he's done I will update the programming (to the latest version and put this brain on our lab's roof with some surface instruments. This should confirm that the electronics are all operational and that the charger-controller can still charge a battery. I'll let that run for a few days and then we'll box up the brain and ship it back to Saipan again. We'll be sure to ship spare fuses back to Saipan as well.

Mike J+

Troubleshooting continues

As of Tuesday afternoon (Saipan time) on January 31st, 2012, the station's control unit or "brain" is on its way back to Miami for examination and possible repair by AOMLers.

This is part of a three-step approach to bringing the station back online:

1. Finding out why the cellular modem went offline (Oct 2nd).
2. Finding out why the station lost power (Oct 4th).
3. Finding out why the RF radios aren't working.

For (1) the cellular modem, it does not appear to be malfunctioning. It was reachable on land via wireless connection several times during troubleshooting operations jointly undertaken by David Benavente, Steven Johnson and Ross Timmerman last November. It merely seems to have gone spontaneously offline on October 2nd, and then later its station power supply seemed to have failed by November 29th. The only thing that stands out from its diagnostics is the unusually high number of "system resets" noted by Ross.

One possible avenue of investigation concerns an AT&T cellular modem that AOML is operating here in South Florida at a test station in Fort Lauderdale. On January 23rd I noticed that this modem's communications were undergoing frequent resets in a way reminiscent of the Docomo modem had done in Saipan before failing. I have relatively easy access to this modem if it should fail, and I've contacted Campbell Scientific support to try to interest them in looking at our software logs from this test station.

For (2) the station's loss of power, we will see if we learn anything from our examination of the brain. [It's entirely possible that we will find that everything appears to be normal, in which case we can only return the brain to Saipan and attempt reinstallation.] Earlier this week we successfully concluded another test of the electronics when David powered up both the datalogger and the cellular modem on the workbench, and I was able to connect to the logger from our systems here in Florida. I was able to download all of the 1-day, 60-minute, 6-minute and 1-minute data tables while this test was still running. So our analysis will initially focus on the wires and connections that supply power to all of these components.

Regarding (3) the RF radios, we have a success to report. David and Steven were able to get their RF radio connection working on land, after reviewing some configuration settings that I suggested might be causing problems. This is a very important step because it means they will have a way to connect to the station from a laptop in the boat, when it comes time to reinstall the brain. They will know immediately, after powering up the station, whether it is running and whether all of the instruments are properly connected. [There is still no way to tell from the boat whether the cellular modem is working properly. We might be able to brainstorm something if they have a wireless internet connection out there, or just coordinate their reinstallation so that they have someone to call on land who can try to connect to the modem.]

Mike Jankulak