attempting a radio connection with the station

As of November 29th, 2011 we had exhausted all of our troubleshooting efforts with the cellular modem on the CREWS station, and the station remained completely offline. The next step was to attempt a radio connection with the station. This, if successful, would prove that the station (though offline) was operational, and would allow us to access its locally-stored data records that described the time since communications were lost. It would do all this without any negative impact on the station's ability to collect and store its data.

Unfortunately, logistic errors during station installation rendered us unable to configure and test the radios that would now be required for this connection. We had the radios during installation in August but not the antennas required to make them work, and we were given the wrong version of the LoggerNet software (required to communicate with the datalogger via the radios) to install.

I summed up the situation in an email message dated Tuesday, November 29th, 2011, at 2:54pm Miami time:

Well the $64000 question is: what is wrong with the station?

Until yesterday it seemed like the likeliest explanation was that the modem wasn't working properly. But now we know that (1) the modem works okay on land, very close to the station and (2) now there doesn't appear to be any power coming to the modem when it is plugged in to the station. [I think that's new, right? The modem's lights were okay when it was last observed connected to the station?]

Dave describes seeing some red lights when looking at the control package, or "brain." The brain does not have a power indicator, but several of its components have lights. My best guess is that Dave is seeing the status light from the RF radio. This light normally blinks red when it's waiting for a connection but it may shine solid red or green at other times, like during startup, or if it is communicating.

I'm a little reluctant to draw conclusions based on Dave's report without knowing exactly which light he was looking at. I generally don't pay much attention to the brain lights and sometimes in strong sunlight I can't really tell if they're lit. But normally the lights aren't important because I can connect by laptop and RF radio from the boat and verify that everything is working.

Which is what I think we should try next. Give me a day or two and I'll see if I can take some shortcuts with the LoggerNet configuration -- I will install it afresh here and see if I can cut and past some key files, rather than try to lead you step by step through the software setup. I'll also check if we can ship you the power cords you'll need to run the radio, either on land or from the boat (your choice).

If I had to guess, I'd say maybe some wires on the brain have been pulled loose since October 2nd. If so, the best scenario would be if only the modem's power wires have come loose, and the station is otherwise fully operational. Given Dave's report of on-again, off-again lights, though, we can't be sure of this. Also it doesn't explain what happened on October 2nd in the first place.

But once we have a better idea of what's going on now, we can decide what to do next. This will probably involve disconnecting everything up top and removing the brain package completely. Then there may be some limited troubleshooting we can try remotely, or we may have to ship it back to AOML for examination.

Following this exchange I shipped radio/logger power cables to David on December 1st (a shipment that was delivered on December 6th). On December 8th I wrote up a four-page document (available here: <ftp://ftp.aoml.noaa.gov/ocd/pub/jankulak/saipan/Saipan-LoggerNet.pdf>) with detailed instructions on how to install the software and connect to the station using the radios.

However, this attempt was ultimately unsuccessful, as reported by David on Thursday, December 15th at 11:11am Saipan time:

I tried to connect to the station using the RF radios today but was unsuccessful. Log error message box said that it was an "unreachable destination". That was the same result when I tested it out at the office.

This signaled the end of our efforts to connect by radio. The next step would be to physically remove the entire control package (the "brain") in order to access its stored data.

cellular modem: testing and reinstallation

Following the loss of communications (October 2nd), an attempt to power-cycle the modem in place (October 4th or 5th) and the recovery of the cellular modem (November 14th or 15th), the next step was to check the modem's diagnostics on land near the CREWS station and then ultimately to reinstall the modem in the station.

Ross Timmerman (PacIOOS, University of Hawai`i describes the land-based testing (which I believe to have taken place on the morning of Monday, November 21st, Saipan time) in his email dated November 21st sent at 9:35am Honolulu time:

Yesterday's modem test at the golf course was successful. Thanks for your help Steven and David. I was able to connect remotely and check the settings. Everything appears in order. The signal strength was decent (-57) and the network status was good. The number of system resets is also okay; no major increase.

I'm a little concerned we didn't find the "smoking gun", but there isn't much else to test. Perhaps the Docomo network cut out and left the modem with a hanging or stale connection. In these instances I have always been able to connect with AceManager to reboot. However I'm not sure what role Loggernet might play in allowing AceManager to connect in this situation. Anyways, unless we can't devise another test and you folks agree, I would like to reconnect the modem to the station.

Steven Johnson (Division of Environmental Quality, Saipan) describes problems encountered during modem reinstallation in a message dated Tuesday, November 29th at 11:57am Saipan time:

Dave and a few of the MMT guys went out to reinstall the modem and they ran into a few snags. First, when Dave ran up to the station and reconnected the modem, but no power was making its way to the modem. No light – no nothing. Next, the station itself powered down for a moment. The sequence of the lights were as follows: Solid red light –> blinking red light –> no light –> solid red light.

When pressed for more details, Steven wrote the following later that same day at 3:02pm Saipan time:

Hopefully Dave will chime in so you can hear it straight from the horses mouth, but I’ll do my best to clarify:

- The lights we are talking about (red lights changing colors) is in reference to the station itself, NOT the modem. The modem itself showed no signs of power when Dave reconnected it to the station. I believe he left the modem plugged into the station.
- From what Dave told me, the station lights were the ones that were running through the sequence I described in the first email. What lights these were exactly, I am not sure as it was Dave on the stick, not I.

David Benavente was able to provide further clarification in a message sent 4:27pm, Saipan time:

When I climbed up the station I checked to see if there was station power. There was, I saw a solid red light, I observed the light for a couple of seconds. Then I began to hook the modem up, first I hooked up the serial port and then I plugged in the power source. I flipped it over to see if there was power but I didn't see any of the green "Modem" lights flashing. So I left the modem alone for about six minutes observing whether or not there was any change. The modem lights did not light up at all. So I once again tried to remove the power cable and replugged it, but still nothing. At that point I checked to see if there was station power but the red light was not illuminated. I then once again removed and replaced the modem power cable and waited for another two minutes still no "modem" lights illuminated.

After that I checked to see if there was station power and i saw the red light was illuminated again.

I believe this reinstallation visit probably took place sometime on Monday (the 28th) or possibly Tuesday morning (the 29th). Reinstalling the modem did not lead to any improvement of communications with the station -- it has remained completely offline since its original loss of communications on October 2nd.

At this point we had a shift in focus from modem troubleshooting to station troubleshooting. These next steps would involve (1) connecting to the station by radio transceiver, and (2) retrieving the entire control unit (or "brain") for data download and inspection.

cellular modem: analysis

The inoperative cellular modem was retrieved from the CREWS station some time between Monday, November 14th (7:44am Saipan time) and Tuesday, November 15th (3:13pm Saipan time). Steven Johnson (Division of Environmental Quality, Saipan) writes in this excerpt:

Dave, myself, and the MMT went to the station and retrieved the bad modem and did a cleaning.

From the email trail I believe this to have been the first visit to the CREWS station since the modem was power-cycled on October 4th or 5th.

Ross Timmerman (PacIOOS / University of Hawai`i) describes the post-retrieval evaluation of the cellular modem in a message dated Tuesday, November 15th (4:32pm Honolulu time):

I was able to connect to the modem, both with AceManager and telnet. The modem data looks good, except for the number of system resets, presently at 3953. A system reset is counted whenever the power is cycled. I'm not sure if a network outage will cause a system reset (I'm checking). Normally the number is less than 50. I've seen high number of resets when a solar battery failed to provide sufficient power. The voltage should be between 10-28V.

Given this, there are a few potential explanations: station power is too low, modem power cable was loose, faulty modem, or the modem just needed to be reset. David confirmed the station was powered when the modem was recovered, though the voltage wasn't checked. I don't think this is the cause since the voltage was looking good prior to 10/3.

Regarding the question of station power levels, I did up a graph to confirm that the reported voltages (prior to losing communications) were normal for a CREWS station. They were. I described the attached graph in an email dated Wednesday, November 16th (11:37am Miami time) in this way:

I worked up a quick graph of the 6-minute voltage reports from August 23rd (day 235) to October 2nd (day 275) and it shows a normal pattern of startup charging until about September 12th, and then diurnal variations after that of about 12.5V to 14.0V (with peaks higher or lower depending on levels of sunlight each day).

Later that same day Michael Shoemaker (AOML, Miami) provided this background on the type of batteries that are installed at the CREWS station:

The batteries in the station are AGM Deep Cycle super low internal resistance mil spec. As can be seen by the graph after being in storage for most of a year on the island they started at 11.5vdc then charged to an average 12.65vdc at their lowest for night power usage. Each battery is 48Ah with a total of 96Ah total to run the station.

Ross Timmerman then suggests that David and Steven try taking the modem to Lao Lao golf course to confirm that there is cellular coverage in the immediate neighborhood of the station and that the signal strength is sufficient. In an email dated Wednesday, November 16th at 10:02am Honolulu time, he says:

The modem was left powered overnight and in checking this morning, no additional system resets were logged. I don't have records of how many resets there were prior to deployment, so it may not be an issue. The modem appears to be working fine now. Given this and the fact that the station power is also good, the next logical step is to try powering the modem from Laolao golf course to check the network and signal strength. If this proves successful, then we can plan on re-installing the modem back in the station. I think it best to rule out network or signal issues on land before scheduling the boat.

station visit: power-cycling the cellular modem

This blog entry concerns the first visit by our Saipan colleagues to the CREWS station after its loss of communications. It is reconstructed from email descriptions. I am not certain when this visit took place except that it must have been some time on Tuesday October 4th or Wednesday October 5th.

From an excerpt from an email from David Benavente (Coastal Resources Management, Saipan), dated Thursday, October 6th, 2011 (7:38am Saipan time):

So there is still power at the station and we did what you suggested. I climbed up the stick, first I checked the power supply and saw that the light was still blinking. I unplugged then replaced the power [cord]. But I wasn't properly able to check everything else, the swells were rocking pretty hard. We still weren't able to get a connection though. The plan is to try and head out there today and if the weather cooperates I'll try cycling the modem again.

Some more details were requested by Jim Hendee (AOML, Miami) in a message dated Thursday, October 6th, 2011 (11:30am Miami time):

I take the station is still in an upright position and not leaning? Glad it has been making it through the swells. Does anybody have an idea on just how bad the swells and winds were there in the Bay?

That led to this update from David on Friday, October 7th, 2011 (11:33am Saipan time):

6-9 ft maybe a little higher, The stick seems to be holding up well. Our field day got scrapped the other day. We do plan to head out there again next week.

(signed)
Mike J+

Saipan CREWS station offline

[This is an edited version of an announcement I sent out on Monday, October 3rd.]

As of this writing the Saipan CREWS station is offline.

The last successful data download from the station took place at 19:10 UTC on October 2nd. To put this into perspective, this was 9am Sunday in Honolulu, 3pm Sunday in Miami, 7pm Sunday Universal, and 5 am Monday in Saipan. As I write this message it was just over 25 hours ago.

When I realized, the first thing I tried was to connect to the modem. I tried connecting with the Airlink software and by telnet, from AOML servers and from machines outside our firewall. I cannot reach the modem.

[This was followed by a request to the Saipan folks to verify that the station was still physically intact (it was) and for our PacIOOS collaborator in Honolulu to verify that he was similarly unable to connect to the modem (he was) and to begin the process of troubleshooting the cellular modem failure (a process to be described in forthcoming blog posts).]

NDBC Data Feed Completed

Last week the ICON team completed the coding and testing of a feed of data from the newly-installed Saipan Coral Reef Early Warning System (CREWS) station to the National Data Buoy Center (NDBC). The National Weather Service (NWS) has assigned the designation LLBP7 to this new CREWS station. Unlike all previous CREWS stations, which feed six-minute wind data but only hourly measurements of all other parameters, the Saipan CREWS station feeds NDBC with 6-minute data for winds, air temperatures and pressures, sea temperatures and salinities, and chlorophyll and turbidity data. This level of detail is made possible through the use of a digital cellular modem at this station, which is a communications technology that allows direct, constant and realtime access to the station data and programming, unlike other CREWS stations which transmit hourly by GOES (satellite) or via local radio connections. The Saipan data will now be included in the NWS operational stream and will be made available globally for research and operations.

The NDBC page for the Saipan CREWS station can be found here.

Mike Jankulak

[The photo (credit: Ross Timmerman) shows the installation of the station's control unit (or "brain") last month, which was the final step in bringing the CREWS station online. Clicking on the image will load the original, larger version.]

"groundtruth" sensor removed, CTD reinstalled correctly

As confirmed by email from David Benavente (Coastal Resources Management, Saipan), dated Thursday, September 15th, 2011 (4:16pm Saipan time):

Today they went out to the CREWS station and retrieved the "groundtruth" CT sensor. This is the instrument that normally is only attached for 3 hours at a time during station cleanings. It was connected during station installation (August 23rd) and left installed after the final visit by the installation team (August 27th) so that we could confirm that its wiring and programming were working properly in the field.

As confirmed by a later email message (received October 31st), they also inspected the "deep" CTD during this September 15th visit. They confirmed that it had been installed backwards (with its pressure sensor pointing inwards towards the pylon) and they were able to reinstall it correctly.

Mike J+