There's an old truism that says "if you can't measure something then you can't fix it." This applies especially well to an off-grid renewable energy installation not only because there are so many components that have to be working in order for the whole system to work, but because the fail-state for many key components may not be immediately obvious UNLESS you're keeping a record of their performance over time. Add to this that off-grid electricity is generally much more expensive than utility-grid electricity and having a way to track and visualize power production and power use becomes an important part of an efficient functional off-grid system. Please note that document will describe (including technical details) how I got my system working, this isn't a How-To per se. It's more of a rambling diary of how I got here.

My intent is to gather various data points from the system - What's the AC load in amps and watts? What's the output of the solar array in amps and watts? What's the state-of-charge for the battery bank in volts? Etc. I want to have a quick way to see what the system is doing now, and I want to have a way to put this into historical perspective. Basically this means gather the data, report on Now, and provide graphs comparing Now to Then.

The system that I inherited came with a nearly ten-year-old Trace inverter, a SW4024. While this inverter DOES have a serial port, it is not immediately practicle to use it to gather data due to a proprietary data protocol. This limitation helped push me toward an incremental change to OutBack based hardware. The OutBack hardware is tied together by two rediculously expensive pieces of hardware called a Hub and a Mate. The Mate is required to perform any kind of configuration to an OutBack inverter, but when a Mate is combined with the Hub it can minimally configure other OutBack hardware AND gather data from them which it will then export via a 9-pin serial interface.

Curiously OutBack has taken a very hands-off stance with regards to providing useful information from the exported data. Despite ample evidence in their forums that there is a market for a simple device that consumes their exported data and produces a small set of historical graphs, years have gone by without OutBack filling this niche. Fortunately other entrepreneurs have stepped in to fill this niche. Options include (but are not limited to) GreenMonitor, WattPlot, WinVerter, and WEBmate. These solutions all have pros and cons to them, but they all also accomplish the basic task of informing you of what your system is doing now, and providing a historical record of how your system has performed. If you want or need something that Just Works I highly encourage you to explore these products.

I'm a fan of hacking things together whenever possible. I've been very hands-on in upgrading and modifying my off-grid system (which is to say that I've ordered the parts I want and, when they're delivered, I'm the one that then installs them). I work in the software industry. I have some basic background in electronics. There's no reason I can't pull all this together to start making pretty graphs.

Step 1: What do I need?

Some of the stuff here is specific to my solution; some of it is generally applicable. Adjust based on your situation.

An always-on computer that will acquire, process, and log the Mate data

The Mate doesn't provide aggregate data (with one exception - the MX/Flex chargers do report kWH aggregate information), so in order to really get any kind of historical data that is at all meaningful you need something to be gathering data updates as often as the Mate is willing to report it. Since the device will be on 24/7 it's also important for the power consumption to be as low as possible.

I considered using a retired junk-laptop (I happen to have one or two with blown HDs, or blown displays, that could be brought back to life), but generally speaking these systems draw at least 20 watts, usually more. There are things I could do to trim the power - replace the HDD with a SSD drive. A small drive would certainly be enough for my needs, but SSD drive are also expensive. I might be able to convince the laptop to boot and run off a thumb-drive. I can disconnect the display. I can set aggressive power-saving settings in the BIOS and the OS. Ultimately I decided to first determine if there were smaller embedded systems that might be better suited for the job.

There are many small embedded PC motherboard computers available that could be purposed for this task. With a power consumption somewhere in the 14-20W range and a typical off-the-shelf price higher than $200, these systems also seemed inappropriate for my needs. I was also somewhat unhappy with the extra features these devices offered - HDMI output, optical sound ports, etc. While great for a machine that will be assembled into a Home Theater PC, these were not needed features that increase power consumption and hadware costs with no benefit to this task.

Ultimately I decided to use an old Linksys NSLU2 NAS device with Debian Linux installed onto it. The NSLU2 is a 256MHz ARM based system with minimal RAM (29M) and no built-in storage that consumes ~4W of power under saturated CPU conditions. Getting Debian Linux installed onto the NSLU2 is outside the scope of this paper. Suffice to say that you'll spend a few hours making that part of the system come together. Ultimately you can use any computing device you feel meets your needs as long as you can get Perl to run on it (to log the Mate data) and have a suite of shell utilities typicall found on *NIX based systems (Linux and MacOS will generally already have all the tools you need. Cygwin will provide them for you on Windows.) Incidentally this NSLU2 was already doing duty as a MRTG data gatherer and reporter. It is still performing this task.

Because the NSLU2 is a fairly primitive special purpose piece of hardware it does NOT come with a serial port (or at least not an easily accessible one). To make it talk with the Mate you need a USB to Serial adapter cable. I am using a TRENDnet TU-S9 cable. According to the Linux kernel this cable uses the pl2303 driver. In order to make the cable connect to the Mate serial port I had to remove the female screw-down nipples on the cable itself since they interfered witht he female screw-down nipples on the Mate. The removal involved vice-grips and twisting/tugging.

Network

My power shack is located ~100 feet (~30.5 meters) from my house. In order for the data logged and processed to be useful to me it needs to be visible from the house (or, ideally, from anywhere on the Internet). My first plan was to use IP-over-AC adapters, but I found that the satellite adapter was unable to make a solid connection to the house. I don't know if this is due to too many breakers between the units (they seem sensitive to breakers in the circuit, so much so that the main "hub" will break the whole network between it and the satellites if it is plugged into a power-strip rather than a wall-outlet) or the length of wire between them (~100 feet + house wiring). Since these devices give nearly no feedback about why they are failing I quickly gave up on troubleshooting this and switched to a Buffalo 802.11g-to-ethernet adapter with a Yagi antenna. This device acts as a client to my house Wifi network and has a 4-port hub on the back into which I can plug non-wifi enabled devices. The normal antenna included with the device are too week to push through 3+ walls, a few trees, and ~100 feet of air, but a nice Yagi antenna is able to punch through the mess and give a solid IP connection (initally the antenna was not well aimed and the link was providing ~5% packet loss. Once the antenna was trimmed the packet loss dropped to 0.2%). The Buffalo consumes between 3W and 4W depending on traffic load.

A server

Many (most?) of the solutions I mention above generate their own graphs and run their own embedded web server that allow you to see the graphs. This is a great solution for something that Just Works, but I dislike it for my use. Viewing the data when I'm away from home would mean having to punch holes in my firewall, it would mean running a server locally, etc. Since I already own and run an external colocated server box that is much more powerful than the NSLU2 there's no reason for me to force the NSLU2 to draw pictures. Instead I'll simply have the NSLU2 push data updates every five minutes to my server and will produce the pretty grraphs there via a combionation of two free and open source projects - RRDtool and a front end for it, Cacti.

Step 2: Gathering data

The Mate has a few specific requirements in order to get the data out of it properly. You can't just plug into the serial port ande start seeing data. You have to get the speed right, the stop-bits, the RTS and DTS to active or no, etc. This is all pretty easily done in a Perl script using the Device::SerialPort module from CPAN.

Here's how I'm logging the data. Note that I'm running this script as root, and my logs are in root's home directory. This is bad sysadmin security practice. This SHOULD be being run as a user with only enough permission on the system to read data from the TTY and write to the log file and temp files. Also note that this script assumes the USB-to-Serial cable is assigned /dev/ttyUSB0. You may need to change this to reflect your system.

When run from a command line this script will happily loop and log forever. The Right Thing to do would be to have init.d launch it at startup.

#!/usr/bin/perl
#
# Requirements: Device::SerialPort 0.12 (from cpan)
#
# Description:  This perl script is for logging of data from a serial
# port, to a specified logfile.  The logfile can then be parsed with
# other programs for reporting purposes.
#
# Version: 0.1
# Author: Bruce S. Garlock
# Date:   2002-09-11
#
# Version: 0.2, DzM
# Date:    2010-05-31
# Adapted to log data from Outback Mate
#
#
#
#

use Device::SerialPort 0.12;

$LOGDIR = "/root"; # path to data file
$LOGFILE = "mate.log"; # file name to output to
$PORT = "/dev/ttyUSB0"; # port to watch


#
# Serial Settings
#

$ob = Device::SerialPort->new ($PORT) || die "Can't Open $PORT: $!";
$ob->baudrate(19200)                  || die "failed setting baudrate";
$ob->parity("none")                   || die "failed setting parity";
$ob->databits(8)                      || die "failed setting databits";
$ob->stopbits (1)                     || die "fail setting stopbits";
$ob->handshake("none")                || die "failed setting handshake";
$ob->dtr_active (1)                   || die "fail setting dtr_active";
$ob->rts_active (0)                   || die "fail setting rts_active";
$ob->write_settings                   || die "no settings";


#
# open the logfile, and Port
#

open(LOG,">>${LOGDIR}/${LOGFILE}") 
  || die "can't open smdr file $LOGDIR/$LOGFILE for append: $SUB $!\n";

open(DEV, "<$PORT") 
  || die "Cannot open $PORT: $_";

select(LOG), $| = 1;      # set nonbufferd mode


#
# Loop forver, logging data to the log file
#


# An outer infinite loop is required because the Mate prints an
# End Of Record (EOR) after each data dumpt. The EOR makes Perl
# believe it has received all the data that will be presented and
# stop logging.
while (1) {
        # inner loop
        while($_ = <DEV>){ # print input device to file
                print LOG $_;
        }
        # end inner loop

        # Doing an infilite loop with a while(true) as above causes
        # Perl to re-run the loop as quickly as the CPU can hundle
        # it. This wastes a lot of CPU cycles and power. Let's sleep
        # for a half second between loops.
        select(undef,undef,undef,0.5)
}

undef $ob;

So now we have a log file filling up with data that looks like this:

1,03,00,00,001,122,00,02,000,00,266,008,000,034
C,00,47,17,085,035,00,00,000,02,269,000,000,078
D,00,42,16,077,029,00,00,000,02,269,000,000,077
1,03,00,00,001,122,00,02,000,00,268,008,000,036
C,00,47,17,085,035,00,00,000,02,269,000,000,078
D,00,42,16,077,029,00,00,000,02,269,000,000,077

We now have to digest this into something that makes sense to plotting and graphing software. And since that software is running elsewhere it also makes sense to do minimal processing locally, then publish summaries to the remote system.

Before we can do anything we have to understand what is being told to us. The format is defined in the Mate Serial Communications Guide. At this stage what's important to us is to know that the firest field is both an address and a device identifyer. A numeral (1-10) is an inverter with the number representing which port of the Hub the inverter is plugged into, a UCAlpha (A-J) is a charger with the letter representing which port of the Hub the device is connected to, and a lcAlpha (a-j) is a Flexnet DC monitor.

All the fields are described in manual (linked above). We'll be making use of many of them to produce out pretty graphs.

Here's how I digest the data prior to export. This script is run by cron every 5 minutes. It digests the recent data in the log to create 1, 5, and 15 minute averages of the various bits of data. It exports a digested form of the data that Cacti will like, then uses scp (secure copy) to push the digested data to the remote server. A sample of the exported data (the exported data is all one line, and is broken into multiple lines here for legibility):

battLevel1:27.2 battLevel5:26.9 battLevel15:26.8 inv1loadAmp1:1.0 inv1loadAmp5:1.0
inv1loadAmp15:2.1 inv1acOutVolt1:121.4 inv1acOutVolt5:121.2 inv1acOutVolt15:120.8
inv1acChgAmp1:0.0 inv1acChgAmp5:0.0 inv1acChgAmp15:0.0 inv1watts1:121.4 inv1watts5:121.2
inv1watts15:253.68 chg1chgAmp1:48.2 chg1chgAmp5:43.2 chg1chgAmp15:41.4 chg1pvAmp1:17.2
chg1pvAmp5:15.3 chg1pvAmp15:14.6 chg1pvVolt1:85.1 chg1pvVolt5:85.0 chg1pvVolt15:85.4
chg1kWH:3.7 chg1wattsIn1:1463.72 chg1wattsIn5:1300.5 chg1wattsIn15:1246.84 chg1chgAmp1:48.2
chg1chgAmp5:43.2 chg1chgAmp15:41.4 chg1pvAmp1:17.2 chg1pvAmp5:15.3 chg1pvAmp15:14.6
chg1pvVolt1:85.1 chg1pvVolt5:85.0 chg1pvVolt15:85.4 chg1kWH:3.7 chg1wattsIn1:1463.72
chg1wattsIn5:1300.5 chg1wattsIn15:1246.84 comkWH:7.4

This script will determine the number of devices plugged into the Hub and spew this data for each of them (i.e. it will automatically notice when you add a second inverter or a third charger), but it makes no effort to digest the data from a Flexnet DC monitor (if someone wants to send me a Flexnet DC monitor then I'll happily add its output).

The script:

#!/bin/bash 

# export_report.sh
# 
# Accepts log files from Outback Mate and exports summary data that can
# be easily consumed by Cacti and RRDTool.
#
# v0.3 6/24/2010
#      Add Twitter support
#
# v0.2 6/1/2010
#      Minor optimizations
#      Use a (hopefully) more reliable mechanism to generate summary
#      for all devices attached to Mate
#
# v0.1 5/30/2010
#      First public release
#
# Created by DzM on 5/21/10.
# Copyright 2010 DzM. All rights reserved.


# Function to quickly create number averages
# Using awk because it can work with decimals and appears (on an NSLU2) to run
# faster than bc.
function average {
awk '
 {
  # this code is executed once for each line
    #lines++;
  # increase the total size, which is field #1
    total+=$1;
  }
END {
  # now output the total
        printf "%0.1f", total/NR;
  }'
}

# Where's our data at?
LOG=/root/mate.log
TEMP=/tmp/mate_

# Figure out how many devices we have (borrowed from tallgirl
# from the Outback forums)
tail -20 ${LOG} | tr '\r' '\n' | grep -v '^$' | awk -F , '{print $1}' > ${TEMP}0

# Count the unique devices.
COUNT=$(sort ${TEMP}0 | uniq | wc -l)

# Create a snapshot of log data for 1 min, 5 min, and 15 min
tail -$(expr ${COUNT} \* 60) ${LOG} > ${TEMP}1
tail -$(expr ${COUNT} \* 300) ${LOG} > ${TEMP}5
tail -$(expr ${COUNT} \* 900) ${LOG} > ${TEMP}15

# And truncate the log file to the last 30 min of data. I have
# no reason to need to archive the raw data.
tail -$(expr ${COUNT} \* 1800) ${LOG} > ${TEMP}30; cat ${TEMP}30 > ${LOG}

# Now, go get the number of devices we counted earlier. Put
# the inverters first, and in order.
sort ${TEMP}0 | uniq > ${TEMP}A
mv ${TEMP}A ${TEMP}0

# Calculate 1, 5, and 15 min average battery levels.
# Since the Mate outputs the battery level as a whole number (e.g. 24.9v
# is represented as 249), also use this as an opportunity to convert
# to decimal notation.
# All the devices have battery meters that measure slightly differently.
# Average them all together.
battLevel1=$(cat ${TEMP}1 | awk -F , '{print $11*.1}' | average)
battLevel5=$(cat ${TEMP}5 | awk -F , '{print $11*.1}' | average)
battLevel15=$(cat ${TEMP}15 | awk -F , '{print $11*.1}' | average)

# Start building our report results
RESULT="battLevel1:$battLevel1 battLevel5:$battLevel5 battLevel15:$battLevel15"

while read LINE ; do
    eval set $(echo $LINE | sed -e 's/,/ /g')
    case $1 in
    [0-9]) # We're an inverter if we're in this loop

        # Calculate 1, 5, and 15 min average load
        # It boggles my mind that the FX reports load only in Amps, and only
        # as whole numbers. This means that the granularity of load reporting
        # is, at best, ~120 watts (Amps * Volts). Worse, the rounding rule
        # applied by the FX appears to be to round down (e.g. 0.9A is 
        # reported as 0A, 1.9A is reported as 1A, etc.)
        # Also, when charging from a generator Load ($2) goes to zero and
        # Buy ($4) becomes Charger ($3) plus whatever the actual AC load is.
        # In order to continue to know what the AC Out load is we have to sum together
        # Load ($2) and Buy ($4) less Charger ($3).
        # Eventually I'll be using an external loop meter to calculate this
        # metric more precisely.
        loadAmp1=$(egrep "^${1}" ${TEMP}1 | awk -F , '{print $2 + ( $4 - $3 ) }' | average)
        loadAmp5=$(egrep "^${1}" ${TEMP}5 | awk -F , '{print $2 + ( $4 - $3 ) }' | average)
        loadAmp15=$(egrep "^${1}" ${TEMP}15 | awk -F , '{print $2 + ( $4 - $3 ) }' | average)

        RESULT="${RESULT} $(echo inv${1}loadAmp1:${loadAmp1} inv${1}loadAmp5:${loadAmp5} inv${1}loadAmp15:${loadAmp15})"

        # Calculate 1, 5 and 15 min average AC voltage 
        # There is a surprising amount of variation in the output voltage. Let's
        # keep track of it.
        acOutVolt1=$(egrep "^$1" ${TEMP}1 | awk -F , '{print $6}' | average)
        acOutVolt5=$(egrep "^$1" ${TEMP}5 | awk -F , '{print $6}' | average)
        acOutVolt15=$(egrep "^$1" ${TEMP}15 | awk -F , '{print $6}' | average)

        RESULT="${RESULT} $(echo inv${1}acOutVolt1:${acOutVolt1} inv${1}acOutVolt5:${acOutVolt5} inv${1}acOutVolt15:${acOutVolt15})" 


        # Calculate 1, 5 and 15 min average AC charge current 
        # Let's keep track of how much amperage we're taking in from the Generator.
        acChgAmp1=$(egrep "^$1" ${TEMP}1 | awk -F , '{print $3}' | average)
        acChgAmp5=$(egrep "^$1" ${TEMP}5 | awk -F , '{print $3}' | average)
        acChgAmp15=$(egrep "^$1" ${TEMP}15 | awk -F , '{print $3}' | average)
        RESULT="${RESULT} $(echo inv${1}acChgAmp1:${acChgAmp1} inv${1}acChgAmp5:${acChgAmp5} inv${1}acChgAmp15:${acChgAmp15})"


        # Quickly calculate AC load Watt averages
        # An attempt to keep track of actual watts used. This is a rough estimate
        # at best (see loadAmps comments above for why).
        watts1=$(echo "${acOutVolt1} ${loadAmp1}" | awk '{print $1*$2}')
        watts5=$(echo "${acOutVolt5} ${loadAmp5}" | awk '{print $1*$2}')
        watts15=$(echo "${acOutVolt15} ${loadAmp15}" | awk '{print $1*$2}')
        RESULT="${RESULT} $(echo inv${1}watts1:${watts1} inv${1}watts5:${watts5} inv${1}watts15:${watts15})"
        ;;

    [A-K])# We're a charger if we're in this loop

        # Pretty print the charger identification
        NAME=$(echo ${1} | tr '[B-J]' '[0-9]')

        # Calculate 1, 5, and 15 min charge amps
        # Note that the MX chargers only report entire amps, not tenths. The
        # FlexMAX chargers actually provide tenths. Lame for those of us with 
        # MX chargers.
        chgAmp1=$(egrep "^$1" ${TEMP}1 | awk -F , 'BEGIN{ OFS = "." } {print $3, $7}' | average)
        chgAmp5=$(egrep "^$1" ${TEMP}5 | awk -F , 'BEGIN{ OFS = "." } {print $3, $7}' | average)
        chgAmp15=$(egrep "^$1" ${TEMP}15 | awk -F , 'BEGIN{ OFS = "." } {print $3, $7}' | average)
        RESULT="${RESULT} $(echo chg${NAME}chgAmp1:${chgAmp1} chg${NAME}chgAmp5:${chgAmp5} chg${NAME}chgAmp15:${chgAmp15})"

        # Calculate 1, 5, and 15 min PV amps 
        # Note that the chargers only report entire amps, not tenths. Lame.
        pvAmp1=$(egrep "^$1" ${TEMP}1 | awk -F , '{print $4}' | average)
        pvAmp5=$(egrep "^$1" ${TEMP}5 | awk -F , '{print $4}' | average)
        pvAmp15=$(egrep "^$1" ${TEMP}15 | awk -F , '{print $4}' | average)
        RESULT="${RESULT} $(echo chg${NAME}pvAmp1:${pvAmp1} chg${NAME}pvAmp5:${pvAmp5} chg${NAME}pvAmp15:${pvAmp15})"


        # Calculate 1, 5, and 15 min PV voltage 
        # Note that the chargers only report entire volts, not tenths. Lame.
        pvVolt1=$(egrep "^$1" ${TEMP}1 | awk -F , '{print $5}' | average)
        pvVolt5=$(egrep "^$1" ${TEMP}5 | awk -F , '{print $5}' | average)
        pvVolt15=$(egrep "^$1" ${TEMP}15 | awk -F , '{print $5}' | average)
        RESULT="${RESULT} $(echo chg${NAME}pvVolt1:${pvVolt1} chg${NAME}pvVolt5:${pvVolt5} chg${NAME}pvVolt15:${pvVolt15})"


        # Daily kWH. Reset when the chargers wakup from night sleep, OR every 24 hours
        # in locations with no night
        # As with battery voltage, reported as an integer (e.g. 10.5 is 105).
        # Also combine the kWH info from all chargers.
        kWH=$(tail -${COUNT} ${TEMP}1 | egrep "$1" | awk -F , '{print $6*.1}')
        comkWH=$(echo "${comkWH} ${kWH}" | awk '{print $1+$2}')
        RESULT="${RESULT} $(echo chg${NAME}kWH:${kWH})"


        # Quickly calculate PV Watt averages
        # An attempt to keep track of actual watts gathered. This is a rough estimate
        # at best (see pvAmps and pvVolts comments above for why).
        wattsIn1=$(echo "${pvAmp1} ${pvVolt1}" | awk '{print $1*$2}')
        wattsIn5=$(echo "${pvAmp5} ${pvVolt5}" | awk '{print $1*$2}')
        wattsIn15=$(echo "${pvAmp15} ${pvVolt15}" | awk '{print $1*$2}')

        RESULT="${RESULT} $(echo chg${NAME}wattsIn1:${wattsIn1} chg${NAME}wattsIn5:${wattsIn5} chg${NAME}wattsIn15:${wattsIn15})"
        ;;
        [a-j]) LOGGER!!! ;;
    esac
done < ${TEMP}0

# Print data in a form that is easily digested by Cacti
rm -rf stats
echo "${RESULT} comkWH:${comkWH}" > stats

# Push the report to the server for consumption by Cacti
scp stats cactiuser@sample.com:

# Tweet things
min=`date +%M`
mod=`expr ${min} % 10`

if [ ${min} -eq 0 ] ; then
curl -u twitpower@jmccabe.com:QweQwe -d status="Tully Creek is generating ${totalWattsIn} watts and using ${totalWattsOut}. Batteries are at ${battLevel1} http://tinyurl.com/tullypower" http://twitter.com/statuses/update.xml
fi

# Clean up our temporary files
rm -rf ${TEMP}[0-9]*

On the Cacti server a "script input" is required to consume the pushed data (this seems like something worthy of ranting about). In order to satisfy this requirement the server runs this script when prompted by the Cacti poller:

#!/bin/sh

cat /home/cactiuser/stats

As with hackign the NSLU2, making Cacti make pretty pictures is outside the scope of this document. A sample of the graphs I have Cacti producing can be seen here.