Intermittent Modem Outages, May 22nd - 31st

This update covers a nine-day period during which the cellular modem was largely online and functioning correctly.  But during this time there were repeated, brief outages.  These outages might be occurring in a pattern that could be repeated in the future, or might be illuminating to Docomo at some point down the line.

The dates and times described here are all in ChST, local Saipan time.  Times may be rounded to the nearest hour.  These, then, were the outages during this time:

• 3 hours, Wed May 22nd at 10pm until Thu May 23nd at 1am
• 1.5 hours, Thu May 23rd from 11:30am to 1pm
• 3 hours, Fri May 24th from 2am to 5am
• 3 hours, Sat May 25th from 2am to 5am
• 2 hours, Sun May 26th from 10am to noon
• 3 hours, Fri May 31st from midnight to 3am

Following this last outage the modem went offline for another prolonged period of time and required another "purge" from Docomo before coming back online (to be described in a future blog update).

Mike J+

Battery Update, May 22nd

The graph was created on May 22nd, when we regained contact with the station after a week-long modem outage and could have another look at its battery levels.  My own emailed update from this date reads as follows:

I'm attaching a graph of battery voltages since go-live April 30th. Right when I'd given up on these batteries ever climbing above 12V, they seem to be improving. So we remain uncertain what the long-term prospects for these batteries are. I would say there is certainly no immediate danger of out-and-out battery failure, though.

Click this graph to see a larger version.

[This update posted by Mike Jankulak.]

Modem Outage, May 15th - 22nd

This modem outage began at about 10am Saipan time, Wednesday morning, May 15th, 2013.  This was only about 16 hours after coming back online after the last outage, which had lasted about 4 days.

The outage lasted a week, with communications resuming on Wednesday, May 22nd at about 10am Saipan time.

After communications resumed, I received the following (somewhat edited) update from Gordon Walker, who is our PacIOOS contact and manages the Docomo account that provides cellular service to this station in Saipan:

I was able to speak with [a Docomo representative] from the Guam office. Calls to Docomo, Saipan that are not answered by a certain number of rings are routed to Guam. [He] was able to get their tech folks to "purge" the line. They asked if we could do a hard reset. After explaining our situation a few times, he finally understood that getting to the modem in Lao Lao bay is a difficult task.

So it seems in this case that Docomo was prodded into action that directly resulted in the resumption of communications with the station.  This is potentially bad news for Gordon and Docomo, since it means they must be alert to future outages and take manual correction action, but good news for the station and the DEQ/CRM folks, since it means that communications can sometimes be restored without the need for going out to the station and climbing it to power-cycle the modem.

Mike J+

Modem Outage, May 10th - 14th

The station went offline at about 19:20 ChST on May 10th, 2013 (local time, Friday evening, in Saipan).  It was noted by Mike J+ in Miami during Miami's Friday afternoon, i.e. after about 8 hours offline.  Word was sent to Gordon and Joe in Honolulu but Gordon was on travel and not available for immediate followup with Docomo.

Communications resumed at about 2pm Saipan time on Tuesday afternoon, May 14th, 2013.  So it was offline for just a few hours shy of 4 days.  As far as we know, there was no contact with Docomo in this case that brought the station back online, it just spontaneously came back online.

Note that with the benefit of hindsight (I am retro-posting these blog updates so we have them for the record) we know that the station only stayed online for about 16 hours and then went offline again.

Mike J+

Station Update -- Modem, Batteries

[From an email sent by Mike Jankulak on May 6th]

The CREWS station at Lao Lao Bay has been online for about seven days now, so I thought I'd send a brief update on these two issues.

Regarding the modem -- I am not aware of any further modem failures since the last observed outage on Wednesday the 1st. Note that brief outages may in fact go unnoticed, since our systems automatically "catch up" our data files with all new data since the last successful connection. Extended outages (on the order of one day or more) however, would trigger alerts that would be logged, and that has not happened in the last five days.

Regarding the batteries -- please see attached graph (below) of min/max/avg station voltages per 6-minute period, plotted against day-of-year. The first night's power outage has not been repeated since, and the good news is that the station's nightly lowest-power levels have steadily increased over the past week. This is promising sign, and I believe the station could continue to operate indefinitely at these power levels.

It is true that these are not "normal" power levels for a CREWS station. Ideally I would like to see power levels range diurnally from about 12.5V to 14.5V, as (for example) we observed for this station from March to May, 2012. Right now the power levels range from about 11V to 12V. However, a look back at the station's early days in August and September of 2011 reveals a similar pattern of initial-deployment battery levels, where during its first week of life it rarely exceeded 12.5V, and in fact required a bit more than three weeks before assuming a normal 12.5V - 14.5V diurnal pattern.

So I would remain cautiously optimistic about the performance of both the modem and the batteries. Certainly there is no cause at present for immediate intervention, and I would recommend that we continue our watch-and-watch approach for at least another two weeks before making any decisions about modem troubleshooting or battery replacement.

Click this image for a larger version of the graph.

Mike J+

CREWS and MApCO2 Operations in Saipan

[Please note that clicking on any of the photos in this post will load a full-sized version of same.]

At the end of April a great inter-agency collaboration took place in Saipan, in the Pacific U.S. territory of CNMI (the Commonwealth of the Northern Mariana Islands), not far from Guam.  CNMI's Division of Environmental Quality (DEQ) hosted visitors from PacIOOS (Pacific Islands Ocean Observing System), who were also affiliated with the University of Hawai`i, and Miami's AOML (Atlantic Oceanographic and Meteorological Laboratory), who were also affiliated with the University of Miami.

There were two separate projects being addressed:  the Coral Reef Early Warning System (CREWS) station was being brought back online, and site surveys were being conducted for a potential deployment of a MApCO2 (Moored Autonomous pCO2) buoy.  Both of these programs are sponsored in whole or in part by NOAA's Coral Reef Conservation Program (CRCP).

The CREWS station was originally installed in August of 2011, but has suffered from intermittent communications outages and power failures, and its equipment was removed in September of 2012 for inspection, diagnosis and repair.  A failed plug for one of the underwater instruments was identified as the cause of the power failures, and plans were made to reinstall the station with entirely new equipment.  CREWS stations, which are also found in St. Croix, Puerto Rico and Belize, are a key component of the Integrated Coral Observing Program (ICON) which is run by AOML's Coral Health and Monitoring Program (CHAMP) team.

On the MApCO2 side, CRCP is in the process of implementing their National Coral Reef Monitoring Plan (NCRMP).  NCRMP calls for sustained monitoring of biological, climate, and socio-economic metrics at all US coral reefs.  The end goal is that every four years, based on NCRMP data collection, the Coral Reef Conservation Program will provide a report card to congress on the state of US coral reefs.  The NCRMP climate monitoring plan calls for establishment of three sentinel (or class III) sites in the Atlantic and Pacific basins where high-resolution monitoring of climate change variables (mainly temperature and CO2) will be made.  Based on the successful installation of the ICON/CREWS pylon in 2011 and the ongoing relationship between DEQ, AOML and others, Saipan was identified as a potential candidate for establishment of one of these sentinel climate monitoring sites.

Work proceeded in tandem on both projects over a period of about two weeks.  First, the entire team visited Lao Lao Bay for the recovery of all remaining equipment left over from the initial deployment, including the station batteries.  All CREWS work during these operations was carried out exclusively by CNMI personnel from DEQ and also Coastal Resources Management (CRM), with guidance as needed from AOML/CREWS people.  This is a first for CREWS, and sets up the Saipan station as the most completely autonomous operation in the program.  The CNMI staff are now capable of conducting their own operations on the CREWS station, down to the installation and recovery of every type of equipment, without the need for assistance by Miami personnel (and the associated travel costs).

CRM's David Benavente spent four long hours atop the Saipan CREWS station during reinstallation operations on April 30th, 2013.

DEQ's Steven Johnson finished up the reinstallation work (seen here getting some assistance from PacIOOS' Joe Gilmore) after having, a few days earlier, removed all equipment left over from the station's original deployment.

While the CREWS people were aloft on the station, the first of several MApCO2 site surveys was taking place in Lao Lao Bay.  In the following days, while the CREWS were being configured and tested on land, MApCO2 site surveys continued at such sites as Boy Scout Reef, Managaha Bay, and Sugar Dock.

AOML's Derek Manzello took photos at Lao Lao Bay while CREWS work was underway.

AOML's Ian Enochs at Lao Lao Bay.

One of the photos taken at Lao Lao Bay by Derek Manzello.

At the start of the second week the team from PacIOOS/UH arrived from Honolulu.  PacIOOS provides a Conductivity-Temperature-Depth (CTD) instrument outfitted with Chlorophyll and Turbidity add-on sensors, and they also provide the digital cellular modem (both the device and the service plan) used for station communications.  At this point the final system tests between CREWS equipment, modem and PacIOOS CTD were conducted on land.

The last step of the process was reinstallation of the CREWS station, which required many long hours atop the station by CRM and DEQ personnel, while AOML and PacIOOS divers deployed the underwater sensors and secured their cables.  Station operations were verified via a radio-connected laptop on the boat before tying everything down and leaving the site.

AOML's Mike Jankulak and PacIOOS' Gordon Walker are seen here installing the CREWS station's "deep" CTD.

Results on the CREWS side were mostly favorable.  All instruments are connected and working normally for the first time since October of 2011, with the inclusion for the first time of a backup Conductivity-Temperature (CT) instrument next to the shallow CTD.  There remains some question about the status of the station's rechargeable batteries, which were drained by a short-circuit and left uncharged on site for seven months.  But battery levels, while not optimal, have proved strong enough to keep the station running nonstop since the first day after redeployment.  A separate question remains about the quality of cellular service at the site and the need for frequent intervention by the service provider to resolve communications outages.  These are being addressed by the PacIOOS team.

The MApCO2 team had come to Saipan hoping to find that the optimal buoy site would be close to the CREWS station in Lao Lao Bay.  Somewhat to their surprise, however, Sugar Dock has emerged as potentially the most favorable buoy deployment site.  This site is most accessible year-round and is relatively free of any issues that may interfere with Ocean Acidification (OA) monitoring such as groundwater, runoff, or sedimentation.  No final decision has yet been made about MApCO2 buoy placement and deployment work is still at least a year away, but the team leaves Saipan having met face-to-face with many of the key CNMI people and having learned a great deal about the data collection efforts that have been ongoing at these sites for many years.

Underwater sights at Obyan Beach, another site visited in Saipan.

Another photo taken at Obyan Beach in Saipan.

The AOML team, consisting of Derek Manzello, Ian Enochs and Mike Jankulak, wishes to thank all of their collaborators in these two projects, including DEQ's Fran Castro, Steven Johnson, Ryan Okano and John Iguel, and CRM's David Benavente (who made a special trip from Guam), as well as PacIOOS' Gordon Walker and Joe Gilmore, who traveled from Honolulu.

The "deep" sensors on the CREWS station:  the CTD in front, and the light sensor on its extended platform behind.

The "shallow" sensors on the CREWS station:  the PacIOOS CTD (with add-on chlorophyll and turbidity sensors) on the right, and an additional CT on the left.

A view of the completed Saipan CREWS station in Lao Lao Bay.

[Update written by Mike Jankulak.  Photos by Derek Manzello, Joe Gilmore, Gordon Walker and Mike Jankulak.]