Automatic Deployment of Let’s Encrypt Certs

Many of you already use Let’s Encrypt certificates in various capacities to provide secure connectivity to applications and devices. Most of the time, these apps and devices automatically reach out, get certs issued, installed and everything just works. That’s cases like traefik, or certbot with apache/nginx, etc.

Then there are those “other” use cases you’ve got. Like say, a custom certificate for a Plex server, or maybe even something more exotic like a certificate for an HP printer. How do you take care of those in an automated, “hands-off” sort of way? How do you make it work so that you’re not having to set reminders for yourself to get in there and swap out certs manually every 3 months? Because you know what’s going to happen right? That reminder’s going to go off, you’re going snooze it for a couple of days, then you’ll tick that checkbox, saying, “yeah, I’ll do it after I get back from lunch” and then something happens and it never gets done. Next thing you know, the cert expires, and it becomes a pain in the rear at the worst possible moment.

That’s where deploy-hooks come into play. If you’ve got a script that can install the certificate, you can call that script right after the cert has been issued by specifying the --deploy-hook flag on the certbot renew command. Let’s look at an example of how we might add this to an existing certbot certificate that’s already setup for automatic renewal. Remember, automatic renewal and automatic installation are different things.

First, we’ll do a dry-run, then we’ll force the renewal. It’s really that easy. Check it:

sudo certbot renew --cert-name --deploy-hook /usr/local/sbin/ --dry-run
sudo certbot renew --cert-name --deploy-hook /usr/local/sbin/ --dry-run

Once this process is completed, the automatic renewal configuration for will include the deploy-hook /usr/local/sbin/ But, what does that really mean? Upon successful renewal, that script will execute, at which point, you’re (presumably) using the script to install the newly refreshed certificate. In this case, the script is unique to that particular certificate. It’s possible to have deploy-hooks that are executed fro EVERY cert as well, by dropping them in the /etc/letsencrypt/renewal-hooks/deploy directory.

For some examples, check out the ones I’m using. Especially interesting (to me at least) is the HP Printer script. That one took a bit of hackery to get working. I had to run the dev tools, and record the browser session a couple of times to get all the variable names straight, and so forth, but once I had it down, it was a snap. Now when the Let’s Encrypt cert updates, within a few seconds, I’ve got the latest cert installed and running on the printer!

What certs will you automate the installation of?

The Dryer Update…

[Any Amazon Links below are Non-Affiliate Links that just go to Amazon Smile]

So, if you think back a bit, you may recall that I was using a Pi 4 for my IoT project that monitored the dryer, shooting out Telegram group messages to the whole family when the dryer was done with the laundry.

Times being what they are, it’s pretty difficult to come by a new Raspberry Pi these days, as I’m sure many of you know. I needed the power of the Pi 4 for something else, at least on a temporary basis. Meanwhile, back at the ranch, a couple of months prior, I’d received a ping from the Micro Center about 45 minutes away informing me that they had a handful of Pi Zero 2 W’s on hand. Those little suckers are super hard to find, so I snapped up my max of 2, along with the GPU I’d been dying to lay hands on for the longest time. For those who care, I finally got an EVGA 3080. Pandemics and supply-chain constraint conditions suck, by the way, in case you were wondering my position on that issue.

So, having my Pi Zero 2 W in the drawer ready to roll, I unscrewed the box from the way that housed the Pi 4, fitted the sensor I had directly onto the Pi Zero 2 W, and scaled down from a 2-project-box solution down to 1 box. Sadly, it sucked. But, it wasn’t the hardware’s fault. In reality it was totally a self-inflicted condition.

I modified (slightly) the pins on the old 801s sensor I had, fitted it onto that new Pi Zero 2W (since it didn’t have any GPIO pin headers soldered on), and sort of Rube-Goldberged it together using 3M VHB tape inside the project box. Total hack job. I thought about using a bunch of hot glue, but then I thought better of it. Why not solder? Honestly? I suck at soldering. One of these days I’ll get around to getting good at it. But that’s not today.

It was wildly unstable. The sensor kept on moving, losing contact with the side of the GPIO holes, it was awful. I all but gave up. I had a brief flirtation with the Aqara Smart Hub and one of their Zigbee Vibration sensors, and believe me, when I say brief, I mean like 12 hours. It just wasn’t fit for the job.

My grand plan with that was to mimic what I was doing over on the washer – write some Python code and run it in a container to query an API somewhere in the cloud every X seconds to see if the thing was vibrating or not, then based on that, work out the state of the dryer to determine if the dryer had started or stopped and then act accordingly. But alas, since step 2 in this plan was a klunker, steps 3 through infinity? Yeah, those never happened.

So, back to the drawing board. I found that I couldn’t easily lay hands on a new 801s again, and the project for the Pi4 was now finished, so I had that back. I did find a new vibe sensor – the SW-420. 3 pins instead of 4, but it’s still a digital output that works fine with the Pi, and my existing code worked as-is, so who cares, right? Yeah, I classed the thing up quite a bit more this time too. This time, instead of shoving the Pi inside a project box that’s mounted on the wall running from the SD card, I opted to run in one of those snazzy Argon One M.2 SSD cases booting Ubuntu 22.04 from an M.2 SSD in the basement of the case. I’ve got that sitting on a lovely little shelf mounted just above and behind the dryer, with my 3 GPIO leads running out of the top of the case, directly into the small project box that’s attached to the front of the dryer, inside which is the sensor, which is stuck to the inside of the box using 3M VHB tape. The box itself is stuck to the dryer using VHB tape as well.

In the end, all’s well that ends well. I’ve had to do a good bit more tuning on the SW-420 sensor. It’s been a bit more fiddly than the old 801s was. That one was definitely a plug and play affair. This has required a bit of adjustment on the little potentiometer that’s built into the sensor. Not too bad though. I’ve invested probably a total of 15 minutes of time standing next to the dryer, staring at telemetry, while the dryer is running, or not. But in the end, it’s all working, and the notifications are happening once again.

One Crazy Summer

Hey automators!

Summer’s been absolutely nuts. Between work stuff, family stuff, running here and there, and of course, the odd project or two, I’ve been just plain stretched for time.

Stay tuned. I’ll be coming back around shortly. I’m working on some things. Preview?

Well, Remember how Logitech decided that the Harmony Remote, one of the best things ever to happen to the world of universal remotes was going to be taken out back and killed? Yeah, I was pretty mad about that too. So, I went looking for something else to solve some automation challenges with that. So, that’s coming.

What else? Tried to buy a Raspberry Pi lately? Heh. Yeah, me too. I decided to try a different fruit for a change. So far, so good. More on that later.

More still? There’s an update on that printer situation. The dryer too.

How about a Raspberry Pi-based network console server for my network equipment?

Hang in there family, it’s coming.

Embracing Simplicity. Again. This time, it’s DNS.

Public Enemy #1

I, like many, hate DNS. I tolerate it. It’s there because, well, I need it. There’s just only so many IP addresses one can keep rattling around inside one’s head, right? So, it’s DNS.

For years, I ran the old standard, BIND under Linux here at home. My old BIND config did a local forward to dnscrypt-proxy, which ran bound to a port on localhost, and then in turn pushed traffic out to external DNS servers like Cloudflare’s or IBM’s I didn’t think my ISP was entitled to be able to snoop on what DNS lookups I was doing. They still aren’t entitled to those, so I didn’t want to lose that regardless of what I ended up doing.

Out in the real world, my domain’s DNS was hosted by DNS Made Easy. They’ve got a great product. It’s reliable, and it’s not insanely expensive. It’s not nothing, but we’re not talking hundreds a year either. I think it’s about $50 a year for more domains and queries than I could possibly ever use. But, like many old schoolers, they’ve lagged behind the times. Yes, they’ve got things like a nice API, and do support DNSSEC, but DNSSEC is only available in their super expensive plans that start at $1700+ a year. That’s just not happening. So, I started looking around.

I landed on Cloudflare. They’ve got a free tier that fits the bill for me. Plenty of record space, a nice API, dare I say, a nicer API even. DNSSEC included in that free tier at no cost even. How do you beat free? I was using a mish-mash of internal and external DNS with delegated subdomains for internal vs external sites as well. It was (again) complicated – and a pain in the rear.

So, I registered a new domain to use just for private use. I did that through Cloudflare as well. As a registrar, they were nice to work with too. They pass that through at cost. Nice and smooth setup. So, internal stuff now consists of names that are [host/app] Traefik is setup using the Cloudflare dns-01 letsencrypt challenge to get certs issued to secure it all, and the connectivity, as discussed before in the other post is all by Tailscale. The apps are all deployed using Docker with Portainer. The stacks (ok, they’re just docker-compose files) in Portainer are all maintained in private GitHub repos. I’ll do a post on that in more detail soon.

Ok, so what did I do with the DNS at home? Did I just ditch the resolver in the house entirely? I did not. In the end I opted for dumping BIND after all these years and replacing it with Unbound. I had to do a bit of reading on it, but the configuration is quite a bit less complex, since I wasn’t configuring zone files any more. I was just setting up a small handful of bits like what interfaces did I want to listen to, what did I want my cache parameters to look like, and what did I want to do with DNS traffic for the outside world, which pretty much everything is? In my case, I wanted to forward it to something fast and secured. I was already crushing pretty hard on Cloudflare, so and were easy choices. I’m also using IBM’s as well. All of those are forwarding out using DNS-over-TLS, and DoT, or sometimes DOT. It worked for me first try.

Then I grabbed the Ubuntu certbot snap and told it to grab a cert for dns.home.$(newdomain).net, which is attached to this moon. After I got the cert issued, it was a piece of cake to turn up both DNS over HTTPS and DNS over TLS, and DoH and DoT.

It was fairly easy to get DoH working on a Windows 11 PC. It was also super easy to craft an MDM-style config profile for DoT that works great on IOS and iPadOS devices. Microsoft has Apple beat cold in this department. Well, in the Apple wold, if you configure a profile for DoT (the only way you can get it in there) you’re stuck with it until you get rid of it – by uninstalling and reinstalling.

On Windows? It was as easy as setting your DNS servers to manual, then crack open a command prompt as Administrator and then (assuming your DNS server is…

netsh dns add encryption https://my.great.server/dns-query

Once you’ve done that, you’ll be able to choose from a list under where you punch in DNS settings in the network settings and turn on Encryption for your DNS connection. It’s working great!

So, You Should Dump IPsec, Right?

Wrong. Probably.

So, Since I just posted the other day about dumping my pile of Python scripts and IPsec VPNs and moving to Tailscale for my personal use case, several folks have sparked conversations with me about the topic.

In my case, it made complete sense to do something else. I was using a solution that was the essentially held together with bubblegum, duct tape, and baling wire. It was fragile, it kept breaking, and let’s be real – I was bending the solution into a shape it wasn’t designed to be used in – which is why it kept breaking in the first place.

You see, IPsec tunnels are intended to work when you’ve got stable, fixed endpoints. Over time, things have been done so that endpoints can become dynamic. But typically just one endpoint. Suddenly, with two dynamic endpoints, results become… Unpredictable. I think that’s a kind way of putting it even. That right there, explains my repeated breakage problems.

So, if you’re still using traditional firewall & VPN in a more traditional use case, then yes, keep doing things more traditionally – keep on using IPsec VPNs. It’s quite honestly the best tool in the bag for what you’re hoping to accomplish in terms of securing data that’s in motion, provided you’re able to meet the bars of entry in terms of hardware support as well as supported feature set.

So, get rid of your firewalls? Not a chance. Get rid of my SRX firewalls and EX switches? No way, no how. You can have my Junos stuff when you pry it from my cold, dead hands. Heck, I make my living with Junos. But just like the whole story of the guy who only has a hammer and thinks everything is a nail, sometimes you’ve just got to use a different tool to do the job right.

But taking the time to think about how to break up with complexity and technical debt? Yeah, that’s totally worth your time. Sometimes that means saying goodbye to old friends, even when you forced them into places where they didn’t quite fit.

So, in the end the whole square-peg-round-hole thing? Stop doing that.

Data Visualization and You…

Sometimes there’s data. You’ve got a bunch of it, you need to work out how to represent it in a way that not only makes sense to you, but is also appealing in some fashion. I’m going to talk about a couple of different use cases in this post, each with their own unique data presentations. First, the sensors.

I’ve got a couple of SwitchBot Meter Plus sensors around the house. One is in my office, and the other is in the garage. There isn’t much to them, small little things, battery powered. Pretty much it’s a little monochromatic LCD screen with a temp/humidity sensor and a bluetooth radio. That won’t do, on its own, of course. So, I added SwitchBot’s Hub Mini to the party. It’s a little bridge device that plugs into the house’s AC mains, and has both BT and WiFi radios inside. While I haven’t cracked it open, the device shows up with a MAC address that suggests it’s little more than an ESP32 or ESP8266 microcontroller inside. With the hub in place, connecting the sensors to the SwitchBot cloud, a really important thing happens – the sensors become accessible via SwitchBot’s REST API. So, I’m using some custom-written Python code that runs under Docker to read the sensors. Turns out it was all surprisingly easy to put the pieces together. It was also a pre-cursor to another project I went on to do, where I helped a friend using a similar sensor to control a smart plug to operate a space heater.

So, what does one do with a sensor like this? You read it, naturally. You keep reading it. Over and over at some sort of fixed interval. In my case, I’m reading it every 5 minutes, or 300 seconds, and storing the data in a database. This type of data isn’t particularly well-suited to living in a SQL database like MariaDB, Postgres, etc. This is a job for a time-series database. So, I called on InfluxDB here. It’s relatively small, lightweight, and very well understood. The Python modules for it are pretty mature and easy to work with even, so it was easy to implement as well. Total win. So, read sensor (convert C to F, since I’m a Fahrenheit kind of guy), store in database, sleep(300), do it again. Lather, rinse, repeat. Just keep on doing that for roughly the next, forever. Or until you run out of space or crash. That’s the code right there, in a nutshell.

Sensors Data Visualization
Sensors Data Visualization

So, what are we visualizing? At the right, you can actually see what I’m graphing. The InfluxData team were nice enough to include some visualization tools right there in the box with InfluxDB, so I’m happy to take advantage of them. Many folks would prefer to use something a bit more flashy and customizable like Grafana, and that’s totally cool. I’ve done it too, even with this same dataset, and the data looks just as good. Heck, probably even looks better, but for me, it was just one more container to have to maintain with little extra value returned. The visualization tools baked into InfluxDB are good enough for what I’m after.

LibreNMS WAN Metrics
LibreNMS WAN Metrics

Next up? Keeping an eye on what’s up with my WAN router’s Internet-facing link. Here at the homestead, I’m running LibreNMS to keep an eye on things. Nothing nearly as custom here. It’s more off the shelf stuff here. It all runs (again) in Docker containers, and as you’d likely expect, uses SNMP to do the bulk of its monitoring duties. at the right, you can see some sample graphs I’ve got stuck to the dashboard page that give a last 6-hours view of the WAN-facing interface of my Internet router, a Juniper SRX300. You see the traffic report as well as the session table size. Within LibreNMS, I’ve got all sorts of data represented, even graphs of how much toner is left in the printer and the temperature of the forwarding ASIC in the switch upstairs in the TV cabinet. All have their own representations, each unique to the characteristics of the data.

Bottom line? Any time you’re dealing with data visualization, there is no one-size-fits-all. Spend the time with the data to figure out what makes the most sense for you and then make it so!

A Journey To A Smarter Dryer

This one was much more difficult. A lot more difficult.

If you recall from my post about the washer, I was able to pull off some fairly useful stuff without a ton of effort. Read a smart plug’s API to see how much power the washer is using to figure out when it turned on, then wait for it to turn off again, then let the fam know that the washer finished, and go take action so that the laundry doesn’t sit around for days, get funky and need to get re-washed. This was of course pretty easy simply because we were able to rely on the fact that the 120V motor in the washer draws well under 15A, the top end of the smart plugs I’m using, the Etekcity ESW15.

Sadly, when we moved into our house, we had an electric dryer. We’ve got natural gas in the house. Heck, in the same room even for the furnace and water heater even. But, back when the last washer sprang a leak and we needed a new washer in a hurry, and unfortunately at the time it was going to be months to get the matching gas dryer back in stock, so we just punted and stuck with the electric model. Sadly, this means for us this means we can’t take the same approach we did with the washer, since nobody makes a smart plug that works on 240V AC 30A circuits.

Unwilling to settle for relying on setting timers with Alexa, having to remind the kids to set timers, or just plain forgetting to do it, I started Googling about, looking for ways to go about monitoring the dryer. Monitoring energy use is the natural fit. When the dryer is in use, it’s consuming loads of energy, and when the clothes are dry, the energy use falls right off. This really shouldn’t be that hard to figure out, right? Right? Sadly, it was.

My next move was to play around with a split core current transformer clamp, and build a circuit with a burden resistor, reading the thing with a microcontroller. I read about the whole process in a handful of places online and it didn’t seem to ridiculous to build the circuit, so I sourced the parts. I got a little breadboard, some jumper wires, the resistors, capacitors, and the CT sensor clamp, and a sacrificial extension cord, which I’d use for my proof of concept test. You see, the CT clamp goes around a single conductor, not the whole cable assembly, so I needed to modify the cable slightly. Relax, the real cable was one of those “flat, side-by-side” types, so it would only mean peeling them apart, not really cutting anything. Sadly, I never made it to that phase. During my POC phase, I was able to get readings back from the sensor, but they never made sense. I was using an ESP32 microcontroller with MicroPython, so maybe that’s related. Or maybe I had a bum CT clamp. Or something else was wrong. We’ll never know, since I gave up after several evenings of bashing my head against the desk.

Failing at the “point” solution of energy monitoring, I moved on to looking at whole-house power monitoring. Hey, if we can’t kill this fly with a newspaper, let’s try a missile, right? Sense landed at the top of the pile. It had the API I was after, though they sort of keep that on the DL. Not in love with that, since those sometimes disappear. If I’m going to drop a couple of hundred bucks on something to use the API for something, it better not just disappear on a whim someday. Plus, our panel is in my home office, recessed in the wall, and there’s not exactly a clean way to get the Sense WiFi antenna back out without it looking really weird. I could make it clean, but then there’d just be a random RP-SMA antenna sticking out of my wall. Interesting decor choice. Sure to be a selling point when we sell the house some day.

Which brings me to the vibration sensor. I was reading one day, still searching, and I came across Shmoopty, and my problems were (half) solved. Sure, I had a Pi Zero W already laying around and I could have just built exactly what he had done, but what’s the fun in that? Remember, I’m already invested. It’s overkill time. So, I ordered up a couple of those 801s vibration sensors and got to work. You know, it was surprisingly hard to get one that met my needs at the time. Why? Most of the 801s units out there are analog-only. Since I’m using a Raspberry Pi, I wanted a digital output, so I didn’t need to mess around with the need for extra ADC (Analog to Digital Conversion) circuitry, just to read a simple sensor. So, I had to order from AliExpress and wait the long wait for shipping from China.

After my sensors finally turned up, I worked out the arrangement of my project boxes and so forth in the laundry room. I landed on a wall-mounted box for the Pi with a 1m pair of wires connected to the sensor, with the sensor inside another small box, which is stuck to the top of the dryer using a little strip of 3M VHB tape. Shmoopty’s Python made it easy to figure out how to read the sensor, so I was happy to be able to draw my inspiration from that. His approach is to keep it small, run on a Pi Zero W, even make it renter-friendly, while mine is more of a “go big” approach – building a Docker container to run it inside of.

Well, at the end of it all, it shares a lot of common philosophy with the plugmon tool, in that it loops infinitely, looking for start and stop conditions. Instead of watching power consumption, it’s watching for the dryer to start vibrating a lot. When that starts an appreciable amount of time, the dryer is declared to be “on”. Once it transitions back to “off”, it fires an event that causes a Telegram message to get sent to the family group chat, again, much like when the washer finishes!

Well, if you’ve made it this far, you’re ready to go check it out. So, get going and get reading. Smarten up that laundry room, report back what you did, and how you did it!

Juniper Switch Port Bounce

How many times do you want to bounce a switchport? Ok, it’s not every 5 minutes, I’ll grant you that. But when you need to, you need to. There’s a handful of strategies we can employ to do this.

Firstly, wild-west style. Just walk right up, yank the cable out, count to 10, and shove it back in. Did it work? Did I grab the right cable? Shoot, I hope so. Wait, Juniper starts counting at zero and Cisco starts counting at 1. Oh crap. I pulled the wrong cable. Let’s go back and do it again. Once more, with feeling, and the right cable this time.

Or, we could take the vastly more measured approach of writing up a full MOP, taking it to the Change Control team, getting it approved, scheduling a change window, coordinating with testing teams, double-checking that we’ve got the right cable, then pull it out, count to 10, plug it back in, have the testers verify that everything works correctly, close out the change window, and then go to bed. But that seems slightly excessive, especially if we really need to bounce that port right now, since the thing on the other end’s not responding and we’re troubleshooting because there’s no connectivity.

What if we take the middle-ground? What if we automated the process a bit to lower the risk of some of the human error factors? If we know what port we want to bounce, we can make that happen in a measured, programmatic way through the Junos Python API, which of course, uses NETCONF under the hood.

Enter the Python script I wrote last night. It’s written (naturally) in Python 3, since Python 2 is now EOL, as of a couple of years ago. Seriously gang, if you’re still writing in Python 2, stop. Anyhow, I’m on the road for a couple of days for work, and after a drive last night, and some time stuck in traffic, and some dinner with a work contact, I was just relaxing, and I wrote this.

Yeah, I know, weird way to relax, right? Ok, I had been pondering this the other day, and just sort of threw the idea in the background for processing at a low priority. You know how that goes. Wrote a bit of code, cranked up the VPN back to home, experimented with bouncing the link connected to a Raspberry Pi on the network at home a few times and here we are.

Feed the script a hostname/IP for the switch, (optionally) a username – if you don’t, it will default to whatever your environment resolves for $USER, (optionally) a password – if you don’t, it will expect to be trying to authenticate using SSH keys, and the port you’re looking to shut and turn back up. Using the Junos Python API, the script connects, does an exclusive config lock, disables the port, commits the config, rolls back, commits again, and finally unlocks the config.

At any rate, here it is, in all its splendor… I also copied and pasted most of the same code and at the same time wrote a “PoE Sledgehammer“. It disables PoE on the switch, then rolls back the change. Useful if you need to do something like simultaneously reboot every phone and/or WLAN AP connected to the switch at the same time. As the name implies, it’s kind of a blunt instrument. Use it with caution…

It Started With A Light Bulb…

One night a few years ago my wife and I sat in the living room watching something on TV, when suddenly one of the recessed lights went out. The bulb died. It wouldn’t be long before a great adventure would begin.

The next morning I trotted off to Lowes to pick up a replacement bulb. I decided that it was time to catch up with current tech and move from the power-gulping bulbs we had in the fixtures to newer LED replacements, so I picked up 4 of those retrofit kits. They’re simple to install. You pop off the old trim ring and unscrew the old bulb, screw in an adapter, connect the wires from the adapter to the LED/trim piece, and put LED/trim into position. Installation takes a good 30 seconds. Minutes later, the 4 cans in our living room had been completely modernized. After I finished and went back to the dimmer on the wall, I popped the little “slider” piece back in to re-energize the switch and the LEDs started flickering. Uh oh. So, the old Lutron dimmer in the box wasn’t ready for LEDs. It’s minimum load was too high, and so it was passing enough power that it was lighting up the LEDs.

Unwilling to go back at this point, I returned to Lowes in search of an updated dimmer. It was then that I was greeted by the Lutron Caseta family of products. On sale was the starter pack. For a small premium beyond the cost of the dimmer I was already going to buy I could get “Smart” switches that I could control from my phone with an app, and even worked with Apple’s HomeKit. I was sold already. Within a year most of the switches in our house had been converted to either Caseta dimmers or switches, except for the couple of spots where we’re using Hue lights.

Is there more? Oh yeah, there’s more. Wait until I tell you guys the story about the microcontroller, fire, the sensor from China, Python, and the Raspberry Pi.