Friday, December 16, 2011

attempting a radio connection with the station

As of November 29th, 2011 we had exhausted all of our troubleshooting efforts with the cellular modem on the CREWS station, and the station remained completely offline. The next step was to attempt a radio connection with the station. This, if successful, would prove that the station (though offline) was operational, and would allow us to access its locally-stored data records that described the time since communications were lost. It would do all this without any negative impact on the station's ability to collect and store its data.

Unfortunately, logistic errors during station installation rendered us unable to configure and test the radios that would now be required for this connection. We had the radios during installation in August but not the antennas required to make them work, and we were given the wrong version of the LoggerNet software (required to communicate with the datalogger via the radios) to install.

I summed up the situation in an email message dated Tuesday, November 29th, 2011, at 2:54pm Miami time:

Well the $64000 question is: what is wrong with the station?

Until yesterday it seemed like the likeliest explanation was that the modem wasn't working properly. But now we know that (1) the modem works okay on land, very close to the station and (2) now there doesn't appear to be any power coming to the modem when it is plugged in to the station. [I think that's new, right? The modem's lights were okay when it was last observed connected to the station?]

Dave describes seeing some red lights when looking at the control package, or "brain." The brain does not have a power indicator, but several of its components have lights. My best guess is that Dave is seeing the status light from the RF radio. This light normally blinks red when it's waiting for a connection but it may shine solid red or green at other times, like during startup, or if it is communicating.

I'm a little reluctant to draw conclusions based on Dave's report without knowing exactly which light he was looking at. I generally don't pay much attention to the brain lights and sometimes in strong sunlight I can't really tell if they're lit. But normally the lights aren't important because I can connect by laptop and RF radio from the boat and verify that everything is working.

Which is what I think we should try next. Give me a day or two and I'll see if I can take some shortcuts with the LoggerNet configuration -- I will install it afresh here and see if I can cut and past some key files, rather than try to lead you step by step through the software setup. I'll also check if we can ship you the power cords you'll need to run the radio, either on land or from the boat (your choice).

If I had to guess, I'd say maybe some wires on the brain have been pulled loose since October 2nd. If so, the best scenario would be if only the modem's power wires have come loose, and the station is otherwise fully operational. Given Dave's report of on-again, off-again lights, though, we can't be sure of this. Also it doesn't explain what happened on October 2nd in the first place.

But once we have a better idea of what's going on now, we can decide what to do next. This will probably involve disconnecting everything up top and removing the brain package completely. Then there may be some limited troubleshooting we can try remotely, or we may have to ship it back to AOML for examination.


Following this exchange I shipped radio/logger power cables to David on December 1st (a shipment that was delivered on December 6th). On December 8th I wrote up a four-page document (available here: <ftp://ftp.aoml.noaa.gov/ocd/pub/jankulak/saipan/Saipan-LoggerNet.pdf>) with detailed instructions on how to install the software and connect to the station using the radios.

However, this attempt was ultimately unsuccessful, as reported by David on Thursday, December 15th at 11:11am Saipan time:

I tried to connect to the station using the RF radios today but was unsuccessful. Log error message box said that it was an "unreachable destination". That was the same result when I tested it out at the office.


This signaled the end of our efforts to connect by radio. The next step would be to physically remove the entire control package (the "brain") in order to access its stored data.

Wednesday, November 30, 2011

cellular modem: testing and reinstallation

Following the loss of communications (October 2nd), an attempt to power-cycle the modem in place (October 4th or 5th) and the recovery of the cellular modem (November 14th or 15th), the next step was to check the modem's diagnostics on land near the CREWS station and then ultimately to reinstall the modem in the station.

Ross Timmerman (PacIOOS, University of Hawai`i describes the land-based testing (which I believe to have taken place on the morning of Monday, November 21st, Saipan time) in his email dated November 21st sent at 9:35am Honolulu time:

Yesterday's modem test at the golf course was successful. Thanks for your help Steven and David. I was able to connect remotely and check the settings. Everything appears in order. The signal strength was decent (-57) and the network status was good. The number of system resets is also okay; no major increase.

I'm a little concerned we didn't find the "smoking gun", but there isn't much else to test. Perhaps the Docomo network cut out and left the modem with a hanging or stale connection. In these instances I have always been able to connect with AceManager to reboot. However I'm not sure what role Loggernet might play in allowing AceManager to connect in this situation. Anyways, unless we can't devise another test and you folks agree, I would like to reconnect the modem to the station.


Steven Johnson (Division of Environmental Quality, Saipan) describes problems encountered during modem reinstallation in a message dated Tuesday, November 29th at 11:57am Saipan time:

Dave and a few of the MMT guys went out to reinstall the modem and they ran into a few snags. First, when Dave ran up to the station and reconnected the modem, but no power was making its way to the modem. No light – no nothing. Next, the station itself powered down for a moment. The sequence of the lights were as follows: Solid red light –> blinking red light –> no light –> solid red light.


When pressed for more details, Steven wrote the following later that same day at 3:02pm Saipan time:

Hopefully Dave will chime in so you can hear it straight from the horses mouth, but I’ll do my best to clarify:

- The lights we are talking about (red lights changing colors) is in reference to the station itself, NOT the modem. The modem itself showed no signs of power when Dave reconnected it to the station. I believe he left the modem plugged into the station.
- From what Dave told me, the station lights were the ones that were running through the sequence I described in the first email. What lights these were exactly, I am not sure as it was Dave on the stick, not I.


David Benavente was able to provide further clarification in a message sent 4:27pm, Saipan time:

When I climbed up the station I checked to see if there was station power. There was, I saw a solid red light, I observed the light for a couple of seconds. Then I began to hook the modem up, first I hooked up the serial port and then I plugged in the power source. I flipped it over to see if there was power but I didn't see any of the green "Modem" lights flashing. So I left the modem alone for about six minutes observing whether or not there was any change. The modem lights did not light up at all. So I once again tried to remove the power cable and replugged it, but still nothing. At that point I checked to see if there was station power but the red light was not illuminated. I then once again removed and replaced the modem power cable and waited for another two minutes still no "modem" lights illuminated.

After that I checked to see if there was station power and i saw the red light was illuminated again.


I believe this reinstallation visit probably took place sometime on Monday (the 28th) or possibly Tuesday morning (the 29th). Reinstalling the modem did not lead to any improvement of communications with the station -- it has remained completely offline since its original loss of communications on October 2nd.

At this point we had a shift in focus from modem troubleshooting to station troubleshooting. These next steps would involve (1) connecting to the station by radio transceiver, and (2) retrieving the entire control unit (or "brain") for data download and inspection.

Thursday, November 17, 2011

cellular modem: analysis

The inoperative cellular modem was retrieved from the CREWS station some time between Monday, November 14th (7:44am Saipan time) and Tuesday, November 15th (3:13pm Saipan time). Steven Johnson (Division of Environmental Quality, Saipan) writes in this excerpt:

Dave, myself, and the MMT went to the station and retrieved the bad modem and did a cleaning.


From the email trail I believe this to have been the first visit to the CREWS station since the modem was power-cycled on October 4th or 5th.

Ross Timmerman (PacIOOS / University of Hawai`i) describes the post-retrieval evaluation of the cellular modem in a message dated Tuesday, November 15th (4:32pm Honolulu time):

I was able to connect to the modem, both with AceManager and telnet. The modem data looks good, except for the number of system resets, presently at 3953. A system reset is counted whenever the power is cycled. I'm not sure if a network outage will cause a system reset (I'm checking). Normally the number is less than 50. I've seen high number of resets when a solar battery failed to provide sufficient power. The voltage should be between 10-28V.

Given this, there are a few potential explanations: station power is too low, modem power cable was loose, faulty modem, or the modem just needed to be reset. David confirmed the station was powered when the modem was recovered, though the voltage wasn't checked. I don't think this is the cause since the voltage was looking good prior to 10/3.


Regarding the question of station power levels, I did up a graph to confirm that the reported voltages (prior to losing communications) were normal for a CREWS station. They were. I described the attached graph in an email dated Wednesday, November 16th (11:37am Miami time) in this way:

I worked up a quick graph of the 6-minute voltage reports from August 23rd (day 235) to October 2nd (day 275) and it shows a normal pattern of startup charging until about September 12th, and then diurnal variations after that of about 12.5V to 14.0V (with peaks higher or lower depending on levels of sunlight each day).


Later that same day Michael Shoemaker (AOML, Miami) provided this background on the type of batteries that are installed at the CREWS station:

The batteries in the station are AGM Deep Cycle super low internal resistance mil spec. As can be seen by the graph after being in storage for most of a year on the island they started at 11.5vdc then charged to an average 12.65vdc at their lowest for night power usage. Each battery is 48Ah with a total of 96Ah total to run the station.


Ross Timmerman then suggests that David and Steven try taking the modem to Lao Lao golf course to confirm that there is cellular coverage in the immediate neighborhood of the station and that the signal strength is sufficient. In an email dated Wednesday, November 16th at 10:02am Honolulu time, he says:

The modem was left powered overnight and in checking this morning, no additional system resets were logged. I don't have records of how many resets there were prior to deployment, so it may not be an issue. The modem appears to be working fine now. Given this and the fact that the station power is also good, the next logical step is to try powering the modem from Laolao golf course to check the network and signal strength. If this proves successful, then we can plan on re-installing the modem back in the station. I think it best to rule out network or signal issues on land before scheduling the boat.

Saturday, October 8, 2011

station visit: power-cycling the cellular modem

This blog entry concerns the first visit by our Saipan colleagues to the CREWS station after its loss of communications. It is reconstructed from email descriptions. I am not certain when this visit took place except that it must have been some time on Tuesday October 4th or Wednesday October 5th.

From an excerpt from an email from David Benavente (Coastal Resources Management, Saipan), dated Thursday, October 6th, 2011 (7:38am Saipan time):

So there is still power at the station and we did what you suggested. I climbed up the stick, first I checked the power supply and saw that the light was still blinking. I unplugged then replaced the power [cord]. But I wasn't properly able to check everything else, the swells were rocking pretty hard. We still weren't able to get a connection though. The plan is to try and head out there today and if the weather cooperates I'll try cycling the modem again.


Some more details were requested by Jim Hendee (AOML, Miami) in a message dated Thursday, October 6th, 2011 (11:30am Miami time):

I take the station is still in an upright position and not leaning? Glad it has been making it through the swells. Does anybody have an idea on just how bad the swells and winds were there in the Bay?


That led to this update from David on Friday, October 7th, 2011 (11:33am Saipan time):

6-9 ft maybe a little higher, The stick seems to be holding up well. Our field day got scrapped the other day. We do plan to head out there again next week.


(signed)
Mike J+

Tuesday, October 4, 2011

Saipan CREWS station offline

[This is an edited version of an announcement I sent out on Monday, October 3rd.]

As of this writing the Saipan CREWS station is offline.

The last successful data download from the station took place at 19:10 UTC on October 2nd. To put this into perspective, this was 9am Sunday in Honolulu, 3pm Sunday in Miami, 7pm Sunday Universal, and 5 am Monday in Saipan. As I write this message it was just over 25 hours ago.

When I realized, the first thing I tried was to connect to the modem. I tried connecting with the Airlink software and by telnet, from AOML servers and from machines outside our firewall. I cannot reach the modem.

[This was followed by a request to the Saipan folks to verify that the station was still physically intact (it was) and for our PacIOOS collaborator in Honolulu to verify that he was similarly unable to connect to the modem (he was) and to begin the process of troubleshooting the cellular modem failure (a process to be described in forthcoming blog posts).]

Tuesday, September 20, 2011

NDBC Data Feed Completed


Last week the ICON team completed the coding and testing of a feed of data from the newly-installed Saipan Coral Reef Early Warning System (CREWS) station to the National Data Buoy Center (NDBC). The National Weather Service (NWS) has assigned the designation LLBP7 to this new CREWS station. Unlike all previous CREWS stations, which feed six-minute wind data but only hourly measurements of all other parameters, the Saipan CREWS station feeds NDBC with 6-minute data for winds, air temperatures and pressures, sea temperatures and salinities, and chlorophyll and turbidity data. This level of detail is made possible through the use of a digital cellular modem at this station, which is a communications technology that allows direct, constant and realtime access to the station data and programming, unlike other CREWS stations which transmit hourly by GOES (satellite) or via local radio connections. The Saipan data will now be included in the NWS operational stream and will be made available globally for research and operations.

The NDBC page for the Saipan CREWS station can be found here.

Mike Jankulak

[The photo (credit: Ross Timmerman) shows the installation of the station's control unit (or "brain") last month, which was the final step in bringing the CREWS station online. Clicking on the image will load the original, larger version.]

Friday, September 16, 2011

"groundtruth" sensor removed, CTD reinstalled correctly

As confirmed by email from David Benavente (Coastal Resources Management, Saipan), dated Thursday, September 15th, 2011 (4:16pm Saipan time):

Today they went out to the CREWS station and retrieved the "groundtruth" CT sensor. This is the instrument that normally is only attached for 3 hours at a time during station cleanings. It was connected during station installation (August 23rd) and left installed after the final visit by the installation team (August 27th) so that we could confirm that its wiring and programming were working properly in the field.

As confirmed by a later email message (received October 31st), they also inspected the "deep" CTD during this September 15th visit. They confirmed that it had been installed backwards (with its pressure sensor pointing inwards towards the pylon) and they were able to reinstall it correctly.

Mike J+

Sunday, August 28, 2011

Electronics installation, three of three

[Reminder: you can click on any of these pictures to see a larger version thereof.]

Our third and final day at the pylon was on Saturday, August 27th.

Our originally scheduled departure day was Thursday, August 25th, and since the underwater instruments were all working properly it was decided that most of the installation team (Derek, Rachel, Ross) would leave on schedule. I extended my stay by four days to wait for weather and seas to ease up and hopefully give me one more day at the pylon to troubleshoot the misbehaving anemometer.

We couldn't do this last operation by rowboat, since I needed a working surface where I could (1) retrieve the "brain" and the anemometer, (2) power the brain off the boat battery and connect the anemometer, and (3) use my laptop to connect directly to the brain and determine what was wrong with the wind sensor. Some way had to be found to bring a larger boat around the island to Lao Lao Bay.

Unfortunately, Wednesday and Thursday saw some of the worst weather of our stay, and TS Talas formed on the Thursday from all of the stormclouds that had been plaguing us. I consulted with Kraig Church of Seafix, Inc. and he eventually settled on Saturday as the last best chance to get a boat to Lao Lao Bay. By then the departing TS Talas had pulled away most of the stormclouds but it left seas on the usually-calm western side of the island a lot rougher in its wake.

Happily the Seafix folks were able to get a good-sized boat to Lao Lao Bay as planned and Saturday morning's operation came off as planned. We also met up with David Benavente (CRM) and Steven Johnson (DEQ) who shadowed me all week for training purposes.

As planned, I retrieved the ailing anemometer and the station's "brain" to the boat for diagnosis. It turns out that one of the wires had broken loose from the connectors we use atop the pylon and had to be replaced. Our operation was drawn out longer than expected because of confusion I experienced when my anemometer wiring diagram did not match the wiring I'd previously connected to the datalogger (in March of 2010, so memories were no longer fresh). It turns out that there are two different versions of this manual, one by the instrument manufacturer (RM Young) and one by the datalogger manufacturer (Campbell Scientific), and the latter manual which I'd brought with me had the wrong wiring information for our sensor. By trial and error I was able to determine the correct wiring of the sensor and out of an abundance of caution I installed the project's "spare" anemometer in place of the instrument that had originally been installed and found to be non-working.

Once we finished with the troubleshooting, all equipment was reinstalled and this time CRM's David Benavente donned the protective gear, climbed the station, and turned it on! Again we packed up the station and left not knowing whether everything was working but this time the news from land was good: all systems operational! A picture of the completed station follows:



We are deeply grateful to our hosts on the island: from Coastal Resources Management (CRM) David Benavente, Rodney Camacho, Guy Macaranas and others; from the Department of Environmental Quality (DEQ, Steven Johnson; from Seafix Inc, Mark Blackburn, Kraig Church, Jesse (our Saturday boat operator) and others. Speaking for the installation team of Derek Manzello (UMiami/CIMAS), Rachel Kotkowski (NOAA Corps), Ross Timmerman (UHawaii/PacIOOS) and myself (UM/CIMAS), we very much appreciate the warm welcome we were given out in the field and on land, this operation would not have been possible without the enthusiastic support of CRM, DEQ and Seafix.

(signed)
Mike Jankulak

photo credit: Steven Johnson

Wednesday, August 24, 2011

Electronics installation, two of three

Our second day in the field was on Tuesday, August 23rd. Logistical complications abounded!

The boat trips around Saipan's southern point the previous Friday had been pretty rough, but as we got further into our stay the seas worsened considerably. A large area of stormclouds had lingered over the entire area since we arrived, part of a tropical system that later developed into TS Talas (on August 25th). We tried to return to the pylon on Monday, August 22nd but the seas prevented the boat with all of our instruments from rounding the southern point and reaching Lao Lao Bay, so we aborted that day's trip.

For Tuesday (the 23rd) conditions were no better, and with only two days left to work before returning we had to consider other options. We brought the installation contractors (Seafix Inc., led by Mark Blackburn and coordinated by Kraig Church) back into the conversation. We discussed the possibility of chartering a larger boat that could safely navigate the higher seas but in the end we accepted Kraig's offer of the use of his cliff stairs (to bring our equipment down to Lao Lao Beach) and rowboats (to row out across the relatively calm waters of the bay to the pylon). I rowed our equipment out on the rowboat with a second boat towed behind and two CRM/DEQ folks in the water to help steer. The divers drove further up the road to enter the water below the golf course. This would be our first-ever rowboat-supported station installation (see photo at right).

All in all things went reasonably well. Steven Johnson (DEQ) stayed in the boats to hand up tools and equipment while I worked up top, and CRM folks (including Rodney and David) stayed in the water to help maneuver the two boats closer to the station when necessary. We started by running the five underwater cables so that the divers could begin installation of the underwater instruments. These included the "deep" light sensor or BIC, the "deep" Teledyne CTD, and the "shallow" Seabird/PacIOOS CTD with add-on chlorophyll and turbidity sensors, all of which wer permanently installed. Pictures of the BIC, "deep" CTD and "shallow" PacIOOS CTD (in this order) here follow:





A fourth cable was for the "groundtruth" CT which was left connected throughout the installation and after but will shortly be retrieved by CRM/DEQ. A fifth cable was for a second "extra" CT planned for deployment at the "shallow" level near the PacIOOS CTD. Unfortunately this second CT was dead when we pulled it out for testing after its 16-month rest in storage, but this sensor's cable was installed and connected and the programming is running on the station, so it can be added on whenever a repaired/replaced sensor comes available.

The pylon-top instruments included the Vaisala "Weather Transmitter" or WXT, a surface light sensor (BIC), an anemometer with electronic compass, a barometer, and an air temperature sensor. The last step was to install the main electronics board (what we sometimes call the "brain"), connect everything up, and turn on the power switch. We were missing the required radio transceiver antennas throughout this operation (they have now arrived in Saipan and have been added to the CREWS storage box for use during future operations), but without a larger boat (and boat battery) we could not have communicated with the station anyhow. So we left the station without knowing whether it was working properly.

From land that afternoon and evening, the results were found to be mostly positive: the station was indeed communicating by digital cellular modem, which marks a milestone of the first time a CREWS station has been deployed using this communications method. All installed instruments were operating correctly except one, the analog anemometer, whose wind speeds were all reporting near zero and whose directions were only intermittently non-zero.

One more day at the pylon would be required to diagnose and fix whatever was ailing the anemometer.

(signed)
Mike Jankulak

all photos by Derek Manzello

Saturday, August 20, 2011

Electronics installation, one of three

On Friday, August 19th the electronics installation team made our first visit to the pylon. The underwater team was made up of Derek Manzello (UMiami/CIMAS), Rachel Kotkowski (NOAA Corps), and Ross Timmerman (UHawaii/PacIOOS). At the pylon-top was yours truly, Mike Jankulak (UMiami/CIMAS). We joined in the boat by Coastal Resoures Management (CRM) and Department of Environmental Quality (DEQ) folks, including David Benavente, Rodney Camacho and Guy Macaranas (from CRM) and Steven Johnson (from DEQ).

The main electronics were still required on land for a full systems test to take place the following day (Saturday), but we wanted to get out to the station at least once before the weekend, look around, and begin our work. On this visit we installed the station batteries and navigational light as well as the extended platform mount for the underwater light sensor (see photo).

(signed)
Mike Jankulak

photo credit: Derek Manzello

Friday, July 22, 2011

Stick is Up!




Seafix, Inc. of Garapan, Saipan has finished installing the CREWS station pylon at Lao Lao Bay. The Seafix crew consisted of Jesse Mailuw, Jerome Wong, Ning-Ning Li, Dale Carhill, and was led by Seafix General Manager Mark Blackburn. Support by boat and access to the Bay was by Kraig Church, also of Seafix, Inc. On site technical consultants were John Halas (NOAA, Office of National Marine Sanctuaries, Florida Keys National Marine Sanctuary), and Jim Hendee (NOAA, Ocean and Atmospheric Research, Atlantic Oceanographic and Meteorological Laboratory; Miami, FL)

Consulting


John Halas of NOAA/ONMS takes a breather.

Oversight


Jim Hendee of NOAA/AOML dries off before another snorkeling plunge.

Securing Shackles








The Seafix team attends to various shackles around the station by wiring them closed to prevent them from unscrewing through action of seas and tides.

Lashing the Eyes to Bottom Plate





All four bottom eyes are lashed to the pins of the bottom plate, and are tightened only so much as to keep the cup on the bottom of the pylon snug against the 2" stainless steel ball on the plate. Thus, the ball articulates within the cup, much as a knee joint. The lashings cannot be too snug, or the hounds on the pylon may bend or break the pins on the bottom of the pylon under heavy seas and defeat the purpose of the pylon being able to move (hence the term, a "dynamic pylon").

Thursday, July 21, 2011

Consulting and Oversight



Jim Hendee of NOAA/AOML observes as the Seafix crew prepares to attend to removing temporary nylon lines from bottom plate to lower hounds, and replacing with blue Spectra line.

Attaching Wraps of Spectra to Bottom Pin




The come-a-long is utilized again to tension line while the final wraps of Spectra line are made through the bottom eye-pins and the thimbles of the extension Spectra lines.

Extensions


A stainless steel shackle is utilized between Spectra line extensions.

Tightening the Lines






A new method calls for a come-a-long for tightening the lines to straighten the pylon, rather than lift bags. This makes for much less cumbersome positioning.

Lashings



The blue Spectra line, attached to the extensions that are in turn attached to the chains, are lashed to a U-pin on the southern side of the pylon, or a standard pin on the north, west and east sides of the pylons.

Floating the Chains



Float bags (in orange) are used to lift the chains from the bottom into place. This part of the operation will be changed in the future.

Spectra Lines


Spectra lines (in blue) are used to steady pylon as chains are attached, while Ning Ning takes movies of the operation.

Wednesday, July 20, 2011

Chains and Shackles








Mark Blackburn and the Seafix crew meet with John Halas as they begin to position chains on shackles at hounds near the waterline.


Friday, July 1, 2011

Baby Steps in the Right Direction

Though we still are waiting for a contract to be issued for the ICON station installation, we continue to slowly make progress. Most recently the modem provided by PacIOOS that will provide wireless communication to and from the station was delivered to Docomo Pacific to make sure the unit will work with the wireless carrier's system.

Monday, March 14, 2011

Business as Usual

After a flurry of activity, its been quiet in the field, though the bureaucrats, accountants, and engineers have been busy. The technical types have been integrating the new PacIOOS parts into the system. An engineering study was needed to make sure that the station would not succumb to the ever more occasional typhoons that pass by during "winter" time in the Marianas. Still despite the delays, we remain more or less on schedule and still anticipate having the station on line by this summer. Considering that this station has been nearly eight years of planning-- what is a few more months...