Datalogger files erased, feeds resume

Following the April 12th - 19th Docomo outage, some of the datalogger's data tables stopped syncing properly.  I made a few attempts to work around the problem by connecting (via the cellular modem) to the station with the LoggerNet software, but eventually I decided to cut our losses and reformat the logger's memory card to resume full data collection as soon as possible.

All available data files were downloaded from the logger on April 24th, then the memory card was formatted.  After that, all data functions (storage and retrieval) returned to normal.

The data table most affected by this problem was the "6 minute" table.  Those data are available from March 20th (reinstallation) to April 12th (Docomo outage), and from April 24th (reformatting) onwards.

With the 6-minute data table operating normally again, our data feeds to PacIOOS and NDBC have resumed.

Mike J+

Station back online -- Docomo had network problems

[From an email by Mike Jankulak on Thursday, April 19, 2012:]

The cellular modem was back online this morning when I arrived at work.  Right now our data feeds are still down because the server files are out of sync with what's stored on the logger (this happened last month too), so I'm working on that.

[From an email by Ross Timmerman on Thursday, April 19, 2012:]

Just had a look at AceManager and all the modem settings appear okay. I don't know why the modem went down earlier, but it was definitely network related. I asked Docomo whether they had a network outage, their response below (sent last night):

"Your services were not disconnected.  Although, we may have resolved the problem.  Please try connecting to the device again, and advise if successful or not."

saipan station offline again (apr 12)

[An excerpt from an email sent from Miami on Friday, April 13th, 2012.]

Bad news, the data feed from the CREWS station has stopped again.

This occurred (in the various time zones of relevance) at:

Thu Apr 12  5:10:35 (Honolulu / HAST / -1000)

Thu Apr 12 11:10:35 (Miami / EDT / -0400)

Thu Apr 12 15:10:35 (UTC / +0000)

Fri Apr 13  1:10:35 (Saipan / ChST / +1000)

So it was it the early hours of the morning, local time, Friday the 13th.  I see no signs of trouble in the data feed -- no voltage spikes or drops, no drop in barometric pressures or rise in winds, no instruments going suddenly or suspiciously offline.

So my best guess right now is that the modem went silent again.  Recall that we still haven't explained why that happened on October 2nd, although we do know why the station itself died on October 4th.

Mike J+

Maintenance log: Station cleaning with DEQ and CRM

[From David Benavente's LAOLAO Bay ICON Station Maintenance log -- April 12, 2012:]

Station cleaning with DEQ and CRM. There was a three-way objective to this trip. MMT had to accomplish a site survey, collect water quality samples and clean the ICON Station. The team broke into smaller groups. Ben C. assisted by Jose Q. were tasked with cleaning the ICON Station. When our surveys were complete I Inspected the station; Ben C. reported that he replaced the copper screens on the Deep CTD.

Station Update: Data Feeds Resuming

The Saipan CREWS station's "brain" (control unit) was reinstalled by David Benavente and colleagues on Monday, March 19th, 2012, as described elsewhere on this blog.

David & co. did a fantastic job, particularly considering that the brain-reconnection training we had hoped to give them last summer wasn't possible due to a combination of terrible seas and missing radio antennas. We have been relying on email (and my lengthy documents of instruction) for training, for everything from software installation and radio communications to installing and connecting the station's hardware.

Their March 19th visit had to be cut short and at the time they left, they reported that the Deep BIC (light sensor) and Deep CTD both seemed to be offline. In fact, there is a delayed startup for the CTD and it can take 6 or 12 minutes (or sometimes longer) for it to begin producing data. When I later examined the data feed, the Deep CTD was communicating normally.

The Deep BIC is offline. It is the only communications failure following the station's recovery. This could be caused by any number of wiring problems at the top of the pylon. I would not consider this to be a hugely urgent matter, but during the next visit David & co. may want to open up the top of the pylon and visually inspect the Deep BIC plugs -- are they connected properly, are they pushed together all the way, is there any sign of loose or broken wires on either side?

The other anomaly in the data stream is that the salinity numbers from the Deep CTD are not tracking properly with those from the Shallow/PacIOOS CTD. I would tend to believe what the Shallow/PacIOOS CTD is saying, particularly since it has recently been retrieved for memory download and battery replacement, and presumably was examined and cleaned at that time. The local team might want to examine the Deep CTD more closely during the next visit, give it a good cleaning and maybe replace its copper screens if they have started to dissolve.

Also, it would be a good idea to do a full station cleaning including the connection of the "groundtruth" CT sensor for the required three hours. This would give us another set of salinity numbers to confirm which of the two CTDs are inaccurate.

In addition, there is still the "extra" or "shallow" CT. We had originally intended to install this CT permanently in August but the instrument we'd planned to use was nonfunctional when retrieved from storage. The shipment that returned the "brain" to Saipan also included a replacement CT and this can be deployed at any time using the cables that were installed and connected last summer.

The data stream shows one short blip from the Shallow/PacIOOS CTD on about March 27th. I am assuming that this is when the Shallow/PacIOOS CTD was temporarily disconnected for memory download and battery replacement.

In the past few days, I have started up all of the Saipan data feeds again. The most recent 24/72 hours of data are updated hourly/daily on our web site:

http://www.coral.noaa.gov/static/data_llbp7_All_12.html

http://www.coral.noaa.gov/static/data_llbp7_All_72.html

The data feed to the National Data Buoy Center (NDBC) has been restarted. As of this writing, oceanic measurements are already populated on the NDBC site and a configuration error with the meteorological data has been corrected so those data should start loading shortly:

http://www.ndbc.noaa.gov/station_page.php?station=llbp7

The feed of CTD data from AOML to PacIOOS is under development and a version of that feed began running yesterday.

And finally, data are loading into CHAMP's "ecoforecast" page here:

http://ecoforecast.coral.noaa.gov/index/0/LLBP7/station-home

Congratulations to David B. and the rest of the team in Saipan on a job well done!

(signed)

Mike Jankulak, AOML, Miami

And its ON:

On the Morning of March 19^th Three MMT members (John Iguel, Ryan Okano & I) loaded up CRM’s small zodiac with The Laolao Bay Pylon Stations “brain”. With only a short timeframe to work, due to tidal constraints, we hurried out to the bay, pushed our small boat over the reef and headed for the station.

Once at the station everything seemed to go according to plan. I geared up and climbed to the top of the station. After months of reading through extraction and installation instructions provided by Mike J, I had clearly developed a system for keeping track of the wiring. After about ten minutes of set-up I signaled Ryan to send up the Brain. Once the Brain was at the top of the station all my focus was set on its installation, which to my surprise went by very quickly. Upon switching the brain “On” I was excited to see that all its components were lighting up and blinking normally.

Back on the boat, with my laptop out, we checked to see if the brain was working properly. Although everything seemed to be working more diagnostics are still very much needed. I felt a great sense of pride upon reinstalling the Brain, and I am so grateful for all the help from NOAA AOML, PACIOOS and the MMT. Special thanks to Mike J., and Mike S. from NOAA AOML.