Productive Rage

22 August 2020

Monitoring my garden's limited sunlight time period with an Arduino (and some tupperware)

My house has a lovely little garden out front. The house and garden itself are elevated one storey above the street (and so my basement is really more of a bizarre ground floor because it has natural light windows but is full of dust and my life-accumulated rubbish is in one room of it while my covid-times "trying to stay fit, not fat" home gym is in the other) and there was no fence around it when I moved in. Meaning that that the interesting characters that amble past (suffice to say that I went for a nicer house in a slightly on-the-cusp between classy and rougher neighbourhoods as opposed to a less nice house in a posh place) could see in and converse between sips on their 9am double-strength lager. Once fenced off, kitted out with a cute little table and chairs that my friendly neighbours found at a tip and with some lovely raised flower beds installed, it is a delight in Summer.. only problem is that my house faces the wrong way and so only gets direct sunlight at certain hours of the day. And this time period varies greatly depending upon the time of year - in March, it might not get the light until almost 5pm whilst in July and August it's getting warm and light and beautiful (well, on the days that English weather allows) more in time for a late lunch.

The problem is that, even after four years here, I still don't really have any idea when it's going to be sunny there for a given time of year and I want to be able to plan opportunities around it - late evening drinks outside with friends, lunch time warm weather meals for myself, just any general chance top up my vitamin D!

I guess that one way to sort this out would be to just keep an eye out on sunny days and take the opportunity whenever it strikes. A more organised plan would be to start a little diary and mark down every fortnight or so through the year when the sun hits the garden and when it leaves.

But I work in technology, damnnit, and so I expect to be able to solve this using that electronics and magic! (Cue comments about everything looking like a nail when you're holding a hammer).

To be really honest, maybe I'm describing this situation back to front. My friend gave me an Arduino UNO r3 because he had a kit spare from the coding club that he runs for kids locally and I'd been looking for a use for it.. and this seemed like it!

What I needed

Being a total Arduino noob (and, since my Electronics GCSE was over 20 years ago now, I'm basically a total hardware noob.. you should have seen the trouble that I had trying to build a custom PC a few years ago; I swear it was easier when I was 14!) I wanted something nice and simple to begin with.

So I had the starter kit, which included the Arduino board and some jumper cables, a prototyping breadboard and some common components (including, essentially, a photoresistor) and so I figured that all I'd then need is a way to record the light levels periodically, a power source and some sort of container for when it rains (again; England).

I considered having some sort of fancy wifi server in it that would record the values somehow and let me either poll it from somewhere else or have it push those results to the cloud somewhere but eventually decided to go for what seemed like a simpler, more robust and (presumably) more power efficient mechanism of storing the light values throughout the day - using an SD card. Because I'd got the kit for free (on the agreement that I would try to do something useful with it), I was looking for something cheap to write to an SD card that I'd had lying around since.. well, I guess since whenever SD cards were useful. Could it have been a digital camera? The very concept seems absurd these days, with the quality of camera that even phones from three or four generations ago have.

I came across something called a "Deek Robot SD/RTC datalogging shield" that would not only write to an SD card but would also keep time due to a small battery mounted on it.

These are cheap (mine was less than £5 delivered, new from eBay) but documentation is somewhat.. spotty. There is a lot of documentation for the "Adafruit Assembled Data Logging shield" but they cost more like £13+ and I was looking for the cheap option. Considering how much time I spent trying to make it work and find good information, it probably would have made more sense to buy a better supported shield than a knock-off from somewhere.. but I did get it working eventually, so I'll share all the code throughout this post!

Note: I found a warning that when using this particular shield, "If you have a UNO with a USB type B connector this shield may NOT WORK because the male pins are NOT LONG ENOUGH" on a forum page - my UNO r3 does have the USB B connector but I've not had this problem.. though if you do encounter this problem then maybe some sort of pin extenders or raisers would fix it.

Step 1: Writing to the SD card

After reading around, I settled on a library called SdFat that should handle the disk access for me. I downloaded it from the Github repo and followed the "Importing a .zip Library" instructions on the Installing Additional Arduino Libraries page.

This allowed me to stack the data logging shield on top of the UNO, put an SD card into the shield, connect the UNO to my PC via a USB lead and upload the following code -

#include <SdFat.h> // https://github.com/greiman/SdFat

// chipSelect = 10 according to "Item description" section of
// https://www.play-zone.ch/de/dk-data-logging-shield-v1-0.html
#define SD_CHIP_SELECT 10

void setup() {
  Serial.begin(9600);

  // See "Note 1" further down about SPI_HALF_SPEED
  SdFat sd;
  if (!sd.begin(SD_CHIP_SELECT, SPI_HALF_SPEED)) {
    Serial.println("ERROR: sd.begin() failed");
  }
  else {
    SdFile file;
    if (!file.open("TestData.txt", O_WRITE | O_APPEND | O_CREAT)) {
      Serial.println("ERROR: file.open() failed - unable to write");
    }
    else {
      file.println("Hi!");
      file.close();
      Serial.println("Successfully wrote to file!");
    }
  }
}

void loop() { }

The Arduino IDE has an option to view the serial output (the messages written to "Serial.println") by going to Tools / Serial Monitor. Ensure that the baud rate shown near the bottom right of the window is set to 9600 to match the setting in the code above.

This happily showed

Successfully wrote to file!

in the Serial Monitor's output and when I yanked the card out and put it into my laptop to see if it had worked, it did indeed have a file on it called "TestData.txt" with a single line saying "Hi!" - an excellent start!

Note 1: In the "sd.begin" call, I specify SPI_HALF_SPEED primarily because that's what most of the examples that I've found use - there is an option SPI_FULL_SPEED but I read in an Arduino forum thread that: "You should be able to use SPI_FULL_SPEED instead, but if that produces communication errors you can use SD_SCK_HZ(4 * MHZ) instead of SPI_HALF_SPEED" and I'm not sure what might be the limiting factor with said communication errors; whether it's the card or the shield or something else and I'm only going to be writing small amounts of data at relatively infrequent intervals and so I thought that I would err on the safe side and stick with SPI_HALF_SPEED.

Note 2: In a lot of code samples, in the "setup" method you will see code after the "Serial.begin(..)" call that looks like this:

while (!Serial) {
  // wait for serial port to connect - needed for native USB
}

^ This is only needed for particular variants of the Arduino - the "Leonardo", I believe - and is not required for the UNO and so I haven't included it in my code.

Gotcha One: Initially, I had formatted my SD card (branded as "Elgetec", who I can't remember ever hearing of other than on this card) on my Windows laptop - doing a full format, to make absolutely sure that it was as ready for action as possible. However, not only did that full format take a long time, I found that when I left my Arduino shield writing files over a period of a few hours then it would often get reported as being corrupted when I tried to read it. I've found that if the SdFormatter.ino (from the examples folder of the SdFat GitHub repo) is used then these corruption problems have stopped occurring (and the formatting is much faster!).

Gotcha Two: While I was fiddling around with writing to the SD card, particularly when connected to a battery instead of the USB port (where I could use the Serial Monitor to see what was happening), I tried setting the LED_BUILTIN to be on while writing and then go off again when the file was closed. This didn't work. And it can't work, though it took me a lot of reading to find out why. It turns out that the SPI (the Serial Peripheral Library) connection from the Arduino to the Deek Robot shield will use IO pins 10, 11, 12 and 13 for its own communications. 13 happens to be the output used to set the LED_BUILTIN state and so you lose access to setting that built-in LED while this shield is connected. Specifically, "pin 13 is the SPI clock. Pin 13 is also the built-in led and hence you have a conflict".

Step 2: Keeping time

Since I want to record light levels throughout the day, it's important to know at what time the recording is being made. The shield that I'm using also includes an "RTC" (a real-time clock) and so I needed to work out how to set that once and then read from it each time I took a light level reading.

The UNO board itself can do some basic form of time keeping, such as telling you how long it's been since the board started / was last reset (via the millis() function) but there are a few limitations with this. You can bake into the compiled code the time at which it was compiled and you could then use that, in combination with "millis()", to work out the current time but you will hit problems if power is temporarily lost or if the board is reset (because "millis()" will start from zero again and timing will start again from that baked-in "compiled at" time).

Gotcha Three: I didn't realise when I was first fiddling with this that any time you connected the USB lead, it would reset the board and the program (the "sketch", in Arduino-speak) would start all over again. (This will only make a difference if you're using an external power source because otherwise the program would stop whenever you disconnected the USB lead and there would be nothing running to reset when plugging the USB lead back in! I'll be talking about external power supplies further down).

So the next step was using the clock on the shield that I had bought, instead of relying on the clock on the Arduino board itself. To do this, I'd inserted a CR1220 battery and then tested with the following code:

#include <Wire.h>
#include <RTClib.h> // https://github.com/adafruit/RTClib

RTC_DS1307 rtc;

void setup() {
  // The clock won't work with this (thanks https://arduino.stackexchange.com/a/44305!)
  Wire.begin();

  bool rtcWasAlreadyConfigured;
  if (rtc.isrunning()) {
    rtcWasAlreadyConfigured = true;
  }
  else {
    rtc.adjust(DateTime(__DATE__, __TIME__));
    rtcWasAlreadyConfigured = false;
  }

  Serial.begin(9600);

  if (rtcWasAlreadyConfigured) {
    Serial.println("setup: RTC is already running");
  }
  else {
    Serial.println("setup: RTC was not running, so it was set to the time of compilation");
  }
}

void loop() {
  DateTime now = rtc.now();
  Serial.print("Year: ");
  Serial.print(now.year());
  Serial.print(" Month: ");
  Serial.print(now.month());
  Serial.print(" Day: ");
  Serial.print(now.day());
  Serial.print(" Hour: ");
  Serial.print(now.hour());
  Serial.print(" Minutes: ");
  Serial.print(now.minute());
  Serial.print(" Seconds: ");
  Serial.print(now.second());
  Serial.println();

  delay(1000);
}

The first time you run this, you'll see the first line say:

setup: RTC was not running, so it was set to the time of compilation

.. and then you'll see the date and time shown every second.

If you remove the USB cable and then re-insert it then you'll see the message:

setup: RTC is already running

.. and then the date and time will continue to show every second and it will be the correct date and time (it won't have reset each time that the USB cable is connected and the "setup" function is run again).

Gotcha Four: When disconnecting and reconnecting the USB lead, sometimes (if not always) I need to close the Serial Monitor and then re-open it otherwise it won't update and it will say that the COM port is busy if I try to upload a sketch to the board.

Gotcha Five: I've seen a lot of examples use "RTC_Millis" instead of "RTC_DS1307" in timing code samples. This is not what we want! That is a timer that is simulated by the board and it just uses the "millis()" function to track time which, as I explained earlier, is no good for persisting time across resets. We want to use "RTC_DS1307" because that uses the RTC on the shield, which will maintain the time between power cycles due to the battery on the board.

Gotcha Six: If you don't include "Wire.h" and call "Wire.begin();" at the start of setup then the RTC won't work properly and you will always get the same weird date displayed when you read it:

Year: 2165 Month: 165 Day: 165 Hour: 165 Minutes: 165 Seconds: 85

Step 3: An external power source

So far, the board has only been powered up when connected to the USB lead but this is not the only option. There are a few approaches that you can take; a regulated 5V input, the barrel-shaped power jack and the option of applying power to the vin and gnd pins on the board.

The power jack makes most sense when you are connecting to some sort of wall wart but I want a "disconnected" power supply for outside. I did a bunch of reading on this and some people are just connecting a simple 9V battery to the vin/gnd pins but apparently that's not very efficient - the amount of power stored in a standard MN1604 9V battery (the common kind that you might use in a smoke alarm) is comparatively low and the vin/gnd pins will be happy with something in the 6V-12V range and there is said to be more loss in regulating 9V to the internal 5V than there would be from a 6V supply.

So I settled on a rechargable 6V sealed lead acid battery, which I believe is often used in big torches or in remote control cars. I got one for £8 delivered from ebay that is stated to have 4.5Ah (which is a measure, essentially, of how much energy it stores) - for reference, a 9V battery will commonly have about 0.5Ah and so will run out much more quickly. Whatever battery you select, there are ways to eke out more life from them, which I'll cover shortly.

It's completely safe to connect the battery to the vin/gnd ports at the same time as the USB lead is inserted, so you don't have to worry about only providing power by the battery or the USB lead and you can safely connect and disconnect the USB lead while the battery is connected as often as you like.

Step 4: Capturing light levels

The starter kit that I've got conveniently included an LDR (a "Light Dependent Resistor" aka a "photo-resistor") and so I just had to work out how to connect that. I knew that the Arduino has a range of digital input/output pins and that it has some analog input pins but I had to remind myself of some basic electronics to put it all together.

What you can't do is just put 5V into one pin of the LDR and connect the other end of the LDR straight into an analog pin. I'm going to try to make a stab at a simple explanation here and then refer you to someone who can explain it better!

The analog pin will read a voltage value from between 0 and 5V and allow this to be read in code as a numeric value between 0 and 1023 (inclusive). When we talk about the 5V output pin, this only makes sense in the context of the ground of the board - so the concept of a 5V output with no gnd pin connection makes no sense, there is nothing for that 5V to be measured relative to. So what we need to do is use the varying resistance of the LDR and somehow translate that into a varying voltage to provide to an analog pin (I chose A0 in my build).

The way to do this is with a "voltage divider", which is essentially a circuit that looks a bit like this:

gnd <--> resistor <--> connection-to-analog-input <--> LDR <--> 5V

If the resistance of the LDR happens to precisely match that resistance of the fixed resistor then precisely 2.5V will be delivered to the analog input. But if the LDR resistance is higher or lower than the fixed resistor's value then a higher or lower voltage will be delivered to analog pin.

There is a tutorial on learn.adafruit.com that does a much better job of explaining it! It also suggests what fixed resistor values that you might use for different environments (eg. are you more interested in granular light level readings at low light levels but don't mind saturation at high levels or are you more interested in more granular readings at high levels and less granular at lower?) - at the moment, I'm still experimenting with a few different fixed resistor values to see which ones work for my particular climate.

The shield that I'm using solder pads for mounting components onto but I wasn't brave enough for that, so I've been using the pass-through pins and connecting them to the bread board that came with my starter kit.

When it's not connected to a power supply, it looks a bit like this:

The code to read the light level value looks like this (while running this code, try slowly moving your hand closer and further from covering the sensor to see the value change when it's read each second) -

void setup() {
  Serial.begin(9600);
}

void loop() {
  Serial.print("Light level reading: ");
  Serial.print(analogRead(0));
  Serial.println();

  delay(1000);
}

In an effort to start putting all of this together into a more robust package, I picked up a pack of self-adhesive felt pads from the supermarket and stuck them to appropriate points under the breadboard -

.. and then I secured it all together with an elastic band:

Step 5: Sleeping when not busy

In my ideal dream world, I would be able to leave my light level monitoring box outside for a few months. As I explained earlier, due to the direction that my garden faces, the hours at which the sun hits it fully varies by several hours depending upon the time of year. However, NO battery is going to last forever and even with this 4.5Ah battery that is at a 6V output (which is only a small jump down to regulate to 5V), the time that it can keep things running is limited.

Note: Recharging via a solar panel sounds interesting but it's definitely a future iteration possibility at this point!

There are, however, some things that can be done to eke out the duration of the battery by reducing the power usage of the board. There are ways to put the board into a "power down" state where it will do less - its timers will stop and its CPU can have a rest. There are tutorials out there about how to put it into this mode and have it only wake up on an "interrupt", which can be an external circuit setting an input pin (maybe somehow using the RTC on the shield I'm using) or using something called the "Watchdog Timer" that stays running on the Arduino even when it's in power down mode.

I read a lot of posts and tutorials on this and I really struggled to get it to work. Until, finally, I came across this one: Arduino Sleep Modes and How to use them to Save the Power. It explains in a clear table the difference between the different power-reduced modes (idle, power-save, power-down, etc..) and it recommends a library called "Low-Power" that takes all of the hard work out of it.

Whereas other tutorials talked about calling "sleep_enable()" and "set_sleep_mode(..)" functions and then using "attachInterrupt(..)" and adding some magic method to then undo all of those things, this library allows you to write a one-liner as follows:

LowPower.powerDown(SLEEP_8S, ADC_OFF, BOD_OFF);

This will cause the board to go into its most power-saving mode for eight seconds (which is the longest that's possible when relying upon its internal Watchdog Timer to wake it up).

No muss, no fuss.

I haven't measured yet how long that my complete device can sit outside in its waterproof box on a single charge of a battery but I'm confident that it's definitely measured in days, not hours - and that was before introducing this "LowPower.powerDown(..)" call.

Since I only want a reading every 30-60s, I call "LowPower.powerDown(..)" in a loop so that there are several 8s power down delays. While I haven't confirmed this yet, I would be astonished if it didn't last at least a week out there on one charge. And if I have to bring it in some nights (when it's dark and I don't care about light measurements) to charge it, then that's fine by me (though I'd like to be as infrequently as possible).

Gotcha Seven: When entering power-down mode, if you are connected to the USB port in order to use the Serial Monitor to watch what's going on, ensure that you call "Serial.flush()" before entering power-down, otherwise the message might get buffered up and not fully sent through the serial connection before the board takes a nap.

Step 6: Preparing for the outdoors (in the British weather!)

I always associate the brand "Tupperware" as being a very British thing - it's what we get packed lunches put into and what we get takeaway curries in. At least, I think that it is - maybe it's like "hoover", where everyone uses the phrase "hoover" when they mean their generic vacuum cleaner. Regardless the origin, this seemed like the simplest way to make my device waterproof. The containers are not completely transparent but they shouldn't make a significant impact on the light levels being recorded by the photo-resistor because they're also far from opaque. And these containers are sealable, waterproof and come in all shapes and sizes!

I took my elastic-band-wrapped "stack" of Arduino-plus-shield-plus-breadboard and connected it to the battery -

.. and put in a plastic box. By turning the battery so that it was length-side-up, it was quite a snug fit and meant that the battery wouldn't slide around inside the box. There wasn't a lot of space for the stack to move around and so it seemed like quite a secure arrangement:

Step 7: The final code

So far, each code sample has demonstrated aspects of what I want to do but now it's time to bring it all fully together.

In trying to write the following code, I was reminded how much I've taken for granted in C# (and other higher level languages) with their string handling! I tried a little C and C++ maaaaany years ago and so writing Arduino code was a bit of a throwback for me - at first, I was trying to make a char array for a filename and I set the length of the array to be the number of characters that were required for the filename.. silly me, I had forgotten that C strings need to be null-terminated and so you need an extra zero character at the end in order for things to work properly! Failing to do so would not result in a compile or run time error, it would just mean that the files weren't written properly. Oh, how I've been spoilt! But, on the other hand, it also feels kinda good being this close to the "bare metal" :)

The following sketch will record the light level about twice a minute to a file on the SD card where the filename is based upon the current date (as maintained by the RTC module and its CR1220 battery) -

#include <Wire.h>
#include <SdFat.h>    // https://github.com/greiman/SdFat
#include <RTClib.h>   // https://github.com/adafruit/RTClib
#include <LowPower.h> // https://github.com/rocketscream/Low-Power

// chipSelect = 10 according to "Item description" section of
// https://www.play-zone.ch/de/dk-data-logging-shield-v1-0.html
#define SD_CHIP_SELECT 10

RTC_DS1307 rtc;

void setup() {
  // The clock won't work with this (thanks https://arduino.stackexchange.com/a/44305!)
  Wire.begin();

  bool rtcWasAlreadyConfigured;
  if (rtc.isrunning()) {
    rtcWasAlreadyConfigured = true;
  }
  else {
    rtc.adjust(DateTime(__DATE__, __TIME__));
    rtcWasAlreadyConfigured = false;
  }

  Serial.begin(9600);

  if (rtcWasAlreadyConfigured) {
    Serial.println("setup: RTC is already running");
  }
  else {
    Serial.println("setup: RTC was not running, so it was set to the time of compilation");
  }
}

void loop() {
  // Character arrays need to be long enough to store the number of "real" characters plus the
  // null terminator
  char filename[13]; // yyyyMMdd.txt = 12 chars + 1 null
  char timestamp[9]; // 00:00:00     =  8 chars + 1 null
  DateTime now = rtc.now();
  snprintf(filename, sizeof(filename), "%04u%02u%02u.txt", now.year(), now.month(), now.day());
  snprintf(timestamp, sizeof(timestamp), "%02u:%02u:%02u", now.hour(), now.minute(), now.second());

  int sensorValue = analogRead(0);

  Serial.print(filename);
  Serial.print(" ");
  Serial.print(timestamp);
  Serial.print(" ");
  Serial.println(sensorValue);

  SdFat sd;
  if (!sd.begin(SD_CHIP_SELECT, SPI_HALF_SPEED)) {
    Serial.println("ERROR: sd.begin() failed");
  }
  else {
    SdFile file;
    if (!file.open(filename, O_WRITE | O_APPEND | O_CREAT)) {
      Serial.println("ERROR: file.open() failed - unable to write");
    }
    else {
      file.print(timestamp);
      file.print(" Sensor value: ");
      file.println(sensorValue);
      file.close();
    }
  }

  Serial.flush(); // Ensure we finish sending serial messages before going to sleep

  // 4x 8s is close enough to a reading every 30s, which gives me plenty of data
  // - Using this instead of "delay" should mean that the battery will power the device for longer
  for (int i = 0; i < 3; i++) {
    LowPower.powerDown(SLEEP_8S, ADC_OFF, BOD_OFF);
  }
}

At the moment, I'm bringing the box inside each night and then disconnecting the battery, pulling out the card and looking at the values recorded in the file to see if it's clear when the sun was fully hitting the table that I had placed the box on.

I've only started doing this in the last couple of days and each day has been rather grey and so there haven't been any sunny periods so that I can confirm that the readings clearly distinguish between "regular daylight" and "sun directly on the table". Once I get some sun again, I'll be able to get a better idea - and if I can't distinguish well enough then I'll adjust the pull-down resistor that splits the voltage with the LDR and keep experimenting!

When I'm happy with the configuration, then I'll start experimenting with leaving the box outside for longer to see how long this battery can last in conjunction with the "LowPower.powerDown(..)" calls. One obvious optimisation for my use case would be to continue keeping it in power-down mode between the hours of 10pm and 8am - partly because I know that it will definitely be dark after 10pm and partly because I am not a morning person and so would not want to be outside before 8am, even if it was streaming with light (which it wouldn't be due to when my yard actually gets direct sunlight).

Gotcha Eight: The RTC has no awareness of daylight savings time and so I'll need to take this into account when the clocks change in the UK. I'll worry about this another day!

Step 8: Draw some graphs (one day)

As you can tell from the above, I'm still very much in the early phases of gathering data. But, at some point, I'm going to have to use this data to predict when the yard will get sun for future dates - once I've got a few months of data for different times of year, hopefully I'll be able to do so!

I foresee a little bit of data-reading and Excel-graph-drawing in my future! There's just something about seeing results on a graph that make everything feel so much more real. As much as I'd like to be able to stare at 1000s of numbers and read them like the Matrix, seeing trends and curves plotted out just feels so much more satisfying and definitive. Maybe there will be a follow-up post with the results, though I feel that they would be much more personal and less useful to the general populace than even my standard level of esoteric and niche blog posts! Maybe there are some graphs in my Twitter stream's future!

On the other hand.. if I learn any more power-saving techniques or have any follow-up information about how long these rechargeable torch-or-remote-control batteries last then maybe that will be grounds for a follow-up!

In the meantime, I hope you've enjoyed this little journey - and if you've tried to do anything similar with these cheap Deek Robot boards, then maybe the code samples here have been of use to you. I hope so! (Because, goodness knows, feeling like a beginner again and getting onto those new forums has been quite an experience!)

Posted at 21:34