cellular modem: testing and reinstallation

Following the loss of communications (October 2nd), an attempt to power-cycle the modem in place (October 4th or 5th) and the recovery of the cellular modem (November 14th or 15th), the next step was to check the modem's diagnostics on land near the CREWS station and then ultimately to reinstall the modem in the station.

Ross Timmerman (PacIOOS, University of Hawai`i describes the land-based testing (which I believe to have taken place on the morning of Monday, November 21st, Saipan time) in his email dated November 21st sent at 9:35am Honolulu time:

Yesterday's modem test at the golf course was successful. Thanks for your help Steven and David. I was able to connect remotely and check the settings. Everything appears in order. The signal strength was decent (-57) and the network status was good. The number of system resets is also okay; no major increase.

I'm a little concerned we didn't find the "smoking gun", but there isn't much else to test. Perhaps the Docomo network cut out and left the modem with a hanging or stale connection. In these instances I have always been able to connect with AceManager to reboot. However I'm not sure what role Loggernet might play in allowing AceManager to connect in this situation. Anyways, unless we can't devise another test and you folks agree, I would like to reconnect the modem to the station.

Steven Johnson (Division of Environmental Quality, Saipan) describes problems encountered during modem reinstallation in a message dated Tuesday, November 29th at 11:57am Saipan time:

Dave and a few of the MMT guys went out to reinstall the modem and they ran into a few snags. First, when Dave ran up to the station and reconnected the modem, but no power was making its way to the modem. No light – no nothing. Next, the station itself powered down for a moment. The sequence of the lights were as follows: Solid red light –> blinking red light –> no light –> solid red light.

When pressed for more details, Steven wrote the following later that same day at 3:02pm Saipan time:

Hopefully Dave will chime in so you can hear it straight from the horses mouth, but I’ll do my best to clarify:

- The lights we are talking about (red lights changing colors) is in reference to the station itself, NOT the modem. The modem itself showed no signs of power when Dave reconnected it to the station. I believe he left the modem plugged into the station.
- From what Dave told me, the station lights were the ones that were running through the sequence I described in the first email. What lights these were exactly, I am not sure as it was Dave on the stick, not I.

David Benavente was able to provide further clarification in a message sent 4:27pm, Saipan time:

When I climbed up the station I checked to see if there was station power. There was, I saw a solid red light, I observed the light for a couple of seconds. Then I began to hook the modem up, first I hooked up the serial port and then I plugged in the power source. I flipped it over to see if there was power but I didn't see any of the green "Modem" lights flashing. So I left the modem alone for about six minutes observing whether or not there was any change. The modem lights did not light up at all. So I once again tried to remove the power cable and replugged it, but still nothing. At that point I checked to see if there was station power but the red light was not illuminated. I then once again removed and replaced the modem power cable and waited for another two minutes still no "modem" lights illuminated.

After that I checked to see if there was station power and i saw the red light was illuminated again.

I believe this reinstallation visit probably took place sometime on Monday (the 28th) or possibly Tuesday morning (the 29th). Reinstalling the modem did not lead to any improvement of communications with the station -- it has remained completely offline since its original loss of communications on October 2nd.

At this point we had a shift in focus from modem troubleshooting to station troubleshooting. These next steps would involve (1) connecting to the station by radio transceiver, and (2) retrieving the entire control unit (or "brain") for data download and inspection.

cellular modem: analysis

The inoperative cellular modem was retrieved from the CREWS station some time between Monday, November 14th (7:44am Saipan time) and Tuesday, November 15th (3:13pm Saipan time). Steven Johnson (Division of Environmental Quality, Saipan) writes in this excerpt:

Dave, myself, and the MMT went to the station and retrieved the bad modem and did a cleaning.

From the email trail I believe this to have been the first visit to the CREWS station since the modem was power-cycled on October 4th or 5th.

Ross Timmerman (PacIOOS / University of Hawai`i) describes the post-retrieval evaluation of the cellular modem in a message dated Tuesday, November 15th (4:32pm Honolulu time):

I was able to connect to the modem, both with AceManager and telnet. The modem data looks good, except for the number of system resets, presently at 3953. A system reset is counted whenever the power is cycled. I'm not sure if a network outage will cause a system reset (I'm checking). Normally the number is less than 50. I've seen high number of resets when a solar battery failed to provide sufficient power. The voltage should be between 10-28V.

Given this, there are a few potential explanations: station power is too low, modem power cable was loose, faulty modem, or the modem just needed to be reset. David confirmed the station was powered when the modem was recovered, though the voltage wasn't checked. I don't think this is the cause since the voltage was looking good prior to 10/3.

Regarding the question of station power levels, I did up a graph to confirm that the reported voltages (prior to losing communications) were normal for a CREWS station. They were. I described the attached graph in an email dated Wednesday, November 16th (11:37am Miami time) in this way:

I worked up a quick graph of the 6-minute voltage reports from August 23rd (day 235) to October 2nd (day 275) and it shows a normal pattern of startup charging until about September 12th, and then diurnal variations after that of about 12.5V to 14.0V (with peaks higher or lower depending on levels of sunlight each day).

Later that same day Michael Shoemaker (AOML, Miami) provided this background on the type of batteries that are installed at the CREWS station:

The batteries in the station are AGM Deep Cycle super low internal resistance mil spec. As can be seen by the graph after being in storage for most of a year on the island they started at 11.5vdc then charged to an average 12.65vdc at their lowest for night power usage. Each battery is 48Ah with a total of 96Ah total to run the station.

Ross Timmerman then suggests that David and Steven try taking the modem to Lao Lao golf course to confirm that there is cellular coverage in the immediate neighborhood of the station and that the signal strength is sufficient. In an email dated Wednesday, November 16th at 10:02am Honolulu time, he says:

The modem was left powered overnight and in checking this morning, no additional system resets were logged. I don't have records of how many resets there were prior to deployment, so it may not be an issue. The modem appears to be working fine now. Given this and the fact that the station power is also good, the next logical step is to try powering the modem from Laolao golf course to check the network and signal strength. If this proves successful, then we can plan on re-installing the modem back in the station. I think it best to rule out network or signal issues on land before scheduling the boat.