Based on the the little trouble-shooting I had done to try to fix this problem it seemed that the problem was rooted in the type of sensor I was using: semi-passive and analog. The device is cheap and very easy to use but the output is weak and lacks the ability to consistently drive the relatively long cable runs (~50 feet) from sensor to processor. Most of the sensors seem to work most of the time but I was having enough trouble that it seemed time to make a change.
My plan was simple: put an amplifier out by each sensor to provide enough drive to be immune from the unidentified craziness that seemed to come and go. I found a candidate part and when it arrived this past Saturday I prototyped a new little sensor board Saturday evening and built and installed the rest on Sunday. As you can see from the results below, things seem to be working much better.
Besides an increased reliability, the amplified sensors allow me to greatly increase the performance of the system overall. The number of samples I average per log entry went from 35 to 1000 (take that central limit thereom!) and the time between log entries went from 3.5 minutes to 1 minute. I could have easily pushed to system more in both direction but for now this seems more than adequate.
Before installing the sensors I tried to calibrate them with little success. I placed a bad of crushed ice and water over the sensors with the hopes that all the sensors would reach the freezing point and I could compare their outputs. After ten minutes of this it was obvious to me that this wasn't getting the sensors cold enough and so I settled for the next best thing. Once the sensors reached room temperature again I simply compared their outputs and added an offset in the processing code to each one, bringing them all into rough agreement. All the sensors now read the same temperature but there is no guarantee that this temperature is the correct one. All six sensors are now consistent but not necessarily accurate in absolute terms.
I'm very happy with the results so far. The system is much less noisy and the higher update rate is nice; its like my temperature system is brand new again.
Nerd Details
The TMP36 is a nice little sensor but lacks drive; the output is high impedance and so it seems susceptible to static build-up and noise over these long cable runs I have. Additionally, to measure multiple sensors on the Arduino, the high impedance forces long delays between read and multiple reads per sensors to get things accurate. This limited the rate as which I could collect data and even then things didn't work so well, as can be seen above.
The op-amp I used is the MCP6002 which is a low-power, rail-to-rail op-amp. I did a quick characterization of the op-amp as a voltage follower and it followed the input 1:1 from 0.1 to 5V, more than enough range for the expected output of the TMP36 (and much better than the other op-amps I tried, see below). Doing a little algebra revealed I could probably even amplify the signal a bit to push my nominal values a little bit out of the weeds (77'F is 0.75V). Instead of configuring the op-amp as a follower I set it up as a non-inverting amplifier with a total gain of 2.82. To prevent oscillation in driving my long twisted pairs and the capacitance they tend to create I used this table to make an informed guess and added a 470 ohm resistor in series with the output. Finally, to make installing the sensor much easier, I used a three pin terminal block as the connector.
The changes to the Arduino code were trivial. I got rid of the extra analogRead(), reduced the delay between reads down to 10ms (I tested at 5ms and it seemed to work just as well), changed the number of reads from 35 to 1000, changed a couple data types to accomodate the much larger values I'd be generating in the running total, and added a unique offset that is applied to each final averaged sensor value based on my quick and dirty "calibration" I had done at my desk. Oh, and I divided the final converted temperature by the 2.82 gain factor the op-amp introduced.
