I did some stupid math in a spreadsheet to place the string slots in the nut for my upcoming 8-string bass. Any constructive input is appreciated.
https://docs.google.com/spreadsheets/d/1KGS_eoy8JUUH_VKmlBYA5T5jHmtdW9s9ESHW9HvVpGg/edit?usp=sharing
Worldwide MQTT Without Exposing a Local Broker to the Public Internet
An article on Steve’s Internet Guide got me thinking about MQTT when on the road. No, seriously! Here’s my situation:
I have a sensor that reports the usual temperature and humidity inside my travel trailer, and it also provides the same information from inside the RV refrigerator thanks to a Ruuvi tag placed within. (Code for this sensor using ESPHome can be found in my Github repo.)
This sensor connects to my home WiFi network when the trailer is parked in the driveway, but obviously not when we are on a trip. We can use the Ruuvi app to read the fridge temperature using BLE when inside the unit, but if we are even across the campsite or farther away – no go.
Anyone who has relied on an RV refrigerator to keep their food safe knows that they can be less than reliable. Knowing the temperature is rising gives us a chance to take action before food items are spoiled.
My goal with this project was to provide a way to monitor the refrigerator temperature when the trailer is parked away from home, and to use Home Assistant to issue alerts that will inform us of any temperature issues.
WiFi First
Before I get into the MQTT configuration, let’s make sure we connect to WiFi when away. My trailer has a travel router that can bridge to RV park networks, a hotspot, or a tethered smartphone to bring Internet access inside. ESPHome allows one to set up more than one set of WiFi options, so I use this code:
wifi:
networks:
- ssid: !secret rv_ssid
password: !secret rv_password
- ssid: !secret wifi_iot_ssid
password: !secret wifi_iot_password
manual_ip:
static_ip: $ip_address
gateway: 192.168.37.1
subnet: 255.255.255.0
dns1: 8.8.8.8
dns2: 1.1.1.1
On the home network I use a static IP address to make connection time fast. This, along with deep sleep, helps limit the time the device is awake when using a battery operated-sensor.
MQTT in the Wild
In order to collect the telemetry data from this sensor when away, I set up a free account with the service flespi. However, one can use any one of many services offering an MQTT broker service, as my solution is simple and standards-based. Thanks to the good folks at flespi for making this service available for trials and hobbyists.
Flespi assigns a long token for authentication, and this can be provided as either a username or password when logging in. Here is my code:
mqtt:
id: mqtt_client
broker: !secret flespi_broker
port: !secret flespi_port
username: !secret flespi_token
birth_message:
topic: $device_name/birthdisable
payload: disable
will_message:
topic: $device_name/willdisable
payload: disable
(Sorry for all the secrets; gotta keep things private and portable)
MQTT Explorer shows the broker contents once the device is connected and transmitting.
The Bridge
Finally, we need to make a safe connection between the local broker and the public one. I use the Mosquitto broker locally, and it offers an MQTT bridge functionality that makes this task easy.
What we are going to do is have the local broker subscribe to the topics created on the external broker. In my example, the sensor uses topics with the prefix “trailer/”. In addition, ESPHome uses a topic prefix to send configuration information to Home Assistant. The default topic for this is “homeassistant/”.
To define these subscriptions, we will modify the mosquitto.conf file to connect to the external broker and subscribe. Add this code to your file:
# Bridge to public broker
connection <MQTT connection name>
address <broker url or IP with port number, such as example.com:1883>
remote_username <use your token here>
bridge_insecure false
try_private false
bridge_protocol_version mqttv50
topic homeassistant/# in 0
topic trailer/# in 0
The top section of this snippet sets up the broker to broker connection. Your version may be different if using another service. Note that I am using an unsecured connection; you may wish to beef things up.
The lower section is what defines the subscriptions. The topic “homeassistant/#” uses the octothorpe wild card symbol to grab anything under this topic prefix. The configuration information for the sensor is found here.
The “trailer/#” topic is used to identify the sensor readings. Whenever something is posted to this topic prefix, it will be retrieved through the bridge to the local broker – ready for consumption by Home Assistant or any other user.
Wrap Up
I was surprised that I could do this without any additional cost and limited coding. Now that the data is feeding to Home Assistant, I can write automations, alerts, etc. to keep us informed.
Thanks again to Steve of Steve’s Internet Guide for the inspiration and examples.
Garage Door Automation Rig
My first rule of home automation is not to ask why I need something automated, but if it is possible. It’s a hobby after all. However, with the garage door we had an actual need. (Plus, I could do it!)
The whole point of having a garage door is so we can close it when access isn’t needed. Otherwise there would just be a hole in the wall. However, in my house we have this bad habit of not closing it – sometimes when we are home and sometimes when we leave. This project was inspired by an all night “open to all” garage.
There were also a couple of articles that outlined how this could be done:
Wifi Garage Door Controller using a Wemos D1 Mini (I pulled much of my code from this project)
Open the Garage Door with a Phone and Particle Core | Make
Project goals:
- Ability to open and close the door remotely
- Report the door’s status
- Use Home Assistant to interface with MQTT to display status and issue commands
- Use automation to close the door when left open too long
- Just recently I added a temperature sensor to report the temp in the garage
Parts List:
- Wemos D1 Mini
- Wemos D1 Mini Relay Shield
- Wemos D1 Mini ProtoBoard Shield
- BMP280 Temperature/Pressure sensor
(Note that his is the 4 pin I2C version.) - 10KΩ Resistor
- Door Bell wire
- Magnetic Reed Switch
UPDATE: I’ve replaced the Magnetic Reed Switch with a much hardier commercial model. The Enforcer works much better!
One could substitute other WiFi enabled project boards, such as the Node MCU. I used the Wemos partly because of the availability of the shields which make for a compact build. The ProtoBoard shield was not required, but helped me get past my substandard soldering skills.
Pin Layout:
- 5V – Power to the relay
- 3.3V – Power to the BMP280
- GND – Ground
- D1 – Relay control (this is hardwired in the shield)
- D2 – Reed switch
- D5 – SCL for BMP280 (not the default pin)
- D6 – SDA for BMP280 (not the default pin)
I don’t usually solder all of the pins on the D1 shields; just those that are needed. However, there is no penalty for completeness. Be sure all of the pins listed above are soldered in all headers.
The Code
This code is doing three things each cycle through the loop. First, it checks the state of the door and reports any changes. Next, the temperature and pressure are checked and reported. Finally, the client loop will look for any commands from MQTT and handle accordingly.
The code is available on GitHub: https://github.com/dashdrum/garage_door
Check Door State
If the door is found to be open, it is not reported right away. Instead a counter is incremented, and only after five consecutive readings of open is that status returned. The goal here is to ignore spurious readings.
A return value of closed is assumed correct and set on first reading.
A change in the door status is immediately reported via MQTT. The process also reports the status each minute regardless of a change to ensure Home Assistant is always in sync.
Report Temperature and Pressure
This code is based on the example program that came with the library by github user mahfuz195 (found here). The readings are returned in JSON format using a library by github user bblanchon (found here)
Updates are limited to once every 10 minutes. No need to overwhelm Home Assistant with readings.
In the begin() function for the BMP, I specify which pins to use for the I2C interface. On the D1 Mini these default to D1 and D2, but those pins were already in use by the relay and reed swtich. D5 and D6 are recruited to take the job.
Callback
The callback routine catches a MQTT message. The payload of “OPEN” is the only command that will “push” the button – closing the realy for 6/10 of a second.
Other Code Features
I’m using a few other libraries in this program. WiFiManager handles the WiFi connection, storing the credentials in nonvolitile memory. It is important to note that WiFiManager will open a WiFi acces point if no credentials are found or the connection attempt fails. Connect to this AP using a laptop or phone, access 192.168.4.1, and follow on the onscreen instructions to setup your WiFi information.
ESP8266HTTPUpdateServer accepts program updates over the air. No need to climb up a ladder to connect your laptop!
Wiring the Reed Switch
Using a bell wire pair, connect one wire to the 3.3V pin, and the other to D2. Also, I’ve added a pull-down resistor between D2 and ground. The pull-down keeps the signal from “floating” (sending an unknown level). On the other end, use the NO (normally open) terminal for one wire and the common terminal for the other.
Wiring the Temperature Sensor
Connect the four pins of the BMP280 sensor as labeled. Rember that my code uses D5 and D6 for SDA and SCL, not the defaults of D1 and D2.
Wiring the Relay
Your garage door opener should have at least four terminals – two for the open button and two for the safety circuit. Follow the wires from your open button to find the two you need. Otherwise, use a short piece of wire to short the pairs of terminals together until the door opens.
Run a pair of bell wire from the opener terminals to the relay. Use the common terminal and the NO terminal on the relay.
Set Up Home Assistant
The GitHub repository includes a configuration file for Home Assistant. Integrate these entries into your configuration.yaml file. Note that this file itself is not a complete configuration file.
Once again the GitHub address is: https://github.com/dashdrum/garage_door
Please leave your questions or suggestions below.
Mini Django
I used the pico_django version of Mini Django, from Github user Tim Watts, to deploy a quickie XML feed today. Fun!
Thanks, Tim!!
Django, Gunicorn, Nginx, & HTTPS
I can find many examples of using Django, Gunicorn, and Nginx for an application, and also ways to implement HTTPS on Nginx, but not all together. This post will document my final configurations and explain my decisions.
First, the code:
nginx.conf:
worker_processes auto;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
## Rewrite http to https
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
return 301 https://$host$request_uri;
}
## Use https
server {
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";
listen 443 ssl;
server_name your.domain.xxx;
ssl_certificate <path-to-certificate>;
ssl_certificate_key <path-to-private-key>;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
location = /favicon.ico { access_log off; log_not_found off; }
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:443;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Proto https;
proxy_read_timeout 300s;
proxy_pass http://localhost:8000;
}
}
}
gunicorn.py:
##bind = ":8000"
workers = 3
timeout = 300
proc_name = "dupe"
user = "<your user>"
group = "<your group>"
raw_env = []
capture_output = True
loglevel = "debug"
errorlog = "<path-and-name-for-log-file>"
gunicorn.service:
[Unit]
Description=gunicorn daemon
After=network.target
[Service]
Environment=DJANGO_DEBUG=False
Environment=DJANGO_SECRET_KEY=<your SECRET_KEY>
Environment=DJANGO_ALLOWED_HOSTS=[*]
Environment=DJANGO_STATIC_ROOT=<application.path>/staticfiles
Environment=DJANGO_SETTINGS_MODULE=<project>.settings.devl
User=<your user>
Group=<your group>
WorkingDirectory=<application.path>
ExecStart=<virtualenv.path>/bin/gunicorn -c <application.path>/gunicorn.py --bind 0.0.0.0:8000 <project>.wsgi:application
[Install]
WantedBy=multi-user.target
Notes:
nginx.conf holds all of the interesting stuff
The first server section grabs any http traffic coming in on port 80 and redirects it to the https URL on port 443.
Handling https traffic was trickier. We need to pass all of the header settings from the incoming request on to gunicorn/Django at localhost. All of the proxy-set
attribute values in the location section came from a little bit of trial and error.
The setting for proxy-read-timeout
is admittedly high – five minutes! This application has one longer running transaction that was exceeding the 60 second default. This value needs to match the timeout
setting in the gunicorn settings. I’ll probably lower it to a more respectable two minutes before hitting production.
Finally, astute readers will notice that there is no code to handle requests for static resources. This is because I am using Whitenoise to route those calls through Django. This works only because my app just has eight users and Django can handle the load. Anything that is public facing should use Nginx to serve static files directly (or look into using a CDN, etc.)
Not much in gunicorn.py
I’m using 3 workers – plenty for my expected load. The timeout
is set to 300 seconds, which is the same found in nginx.conf above. These should match.
The bind
setting is commented and moved to gunicorn.service so that I can use this file for more than one instance of the application. A second service file can specify a unique port for its copy of gunicorn.
gunicorn.service – Instance specific settings
Since my app is stored in a git repo (on a company server, not github), I don’t want to include any security information like the SECRET_KEY in a versioned file. Therefore, this guy is not included there – although an example file is.
The Environment variables set here are read by Django at startup.
The ExecStart line starts by running the version of gunicorn found in the virtualenv for this instance. Setting up a virtualenv is beyond this scope of this post, but I hope that if you’ve gotten this far you know all about them.
Note that the bind parameter includes the port number used for this instance of gunicorn as mentioned above.
Questions? Comments?
I hope this helps someone setting up a Django app. Please drop a note in the comments with your experience. I’d also invite questions, as well as comments on how I can improve my configuration. Thanks all!
Installing Oracle Client on Ubuntu 11.10
(Another post written for personal documentation)
Get Software
Download these three packages from Oracle for the proper operating system (32 bit for me):
- Instant Client Basic-Lite
- Instant Client SDK
- Instand Client SQLPlus
Unzip and copy to /opt/oracle/11_2/instantclient
Set LD_LIBRARY_PATH
Create /etc/ld.so.conf.d/oracle_instantclient.conf:
#Oracle client LD_LIBRARY_PATH setting /opt/oracle/11_2/instantclient
Update cache:
sudo ldconfig -v
Symbolic Link to Library
ln -s libclntsh.so.10.1 libclntsh.so
Set ORACLE_HOME
export ORACLE_HOME=/opt/oracle/11_2/instantclient
Install Python Library
The library is called cx-oracle. Use your favorite installation method. (Don’t forget about your virtual environment!)
That’s It
Hope I remembered everything.
UPDATE
Found another post that outlines the procedure maybe a little better than I did, and includes notes on setting up tnsnames.ora
. See Install Oracle Instant Client 11.1 and cx_Oracle 4.4 on Ubuntu 8.04 on Technoblog.
Changing Hosts in Dreamhost
Last night, Dreamhost moved my account to a new host. Not a big deal, and it’s certainly understandable, but it did break my one running Django app – the Gentryart Gallery. Here’s what I did to fix it.
First, as suggested in the Dreamhost Server Moves Page, I recompiled Python to create a 64bit version. While I was at it, I upgraded from v2.5.2 to v2.6.5. This great page by Ryan Kanno outlined the steps. I just changed the version number. I also deleted my old installation of Python so there wouldn’t be any chance of confusion there.
Now, if I would have used the same python version and kept everything in the same locations, I would have been done. Since I didn’t, I had a little more to do.
Next, I needed to install all of the python packages I use. Here’s the list:
- Django (of course)
- MySQLdb (which requires setuptools)
- Python Image Library
- Photologue
Following the famous Jeff Croft post, I updated the admin_media
shortcut to point to the new Django files.
And the gallery is back in business!
Reporting in Django
The Django application I’m developing has a reporting need: paper (PDF) output of reports, both 8 1/2 X 11 and special forms (labels, name tags, etc.). So, I’ve begun evaluating my reporting options.
Unfortunately, I haven’t found very many methods to examine, probably because of my requirements:
- Python based
- Since I’m using Python/Django for the rest of the work, it would make sense to use Python for this component as well. Other folks that may someday support the application will appreciate having a single skill set to maintain.
- Contained in the Application
- I’ve found at least one system that runs as a separate service on a server, but that is more complexity than I would like to introduce into a client’s technology stack. This application will appeal to small offices that likely don’t have strong tech support. Adding a separate service (in this example, Java/Tomcat) on top of Django, MySQL and Apache is more than many can do, and I certainly don’t want to get into the server admin business on their behalf.
- Open Source
- Another issue that the clients won’t want to deal with is licensing to third parties. I’m using all open source tools to build and run the application, and it wouldn’t make sense to require a potential customer to license something before they could use it. Plus, would I have to become a reseller for the licensed software?
- Integrated with Django
- This is total ‘wish list’ stuff, but wouldn’t it be neat to have an interface similar to the Django templating system where I could bring in data from a view into a report ‘template’?
Looking through Google searches, I’ve found that the terms “reports” and “reporting” can mean different things to different people. Since I’m looking to generate paper, I have been looking at Geraldo, which uses the ReportLab library to generate PDF output. This application looks promising, but is still a work in progress. I’m finding that flexible formatting with variable length text is a difficult, and maybe impossible, thing to do.
I’d like to hear from others on this subject. Have you found a good package for reporting? Are you developing one? Please leave your thoughts in the comments.
Accessing the Value of a Form Field in a Template
(This is a ‘Note to Self’ post)
I’ve been trying for half the day to get radio buttons in a custom template to work. The buttons are generated in a for loop that builds a table row for each choice (iterating over a queryset called cls
), including this code for the row’s radio button:
input type="radio" name="reg_choice_1" value="{{ cls.id }}" {% ifequal srform.reg_choice_1 cls.id %}CHECKED{% endifequal %}
The goal is to have the ifequal
test tell me if the row’s cls.id
matches the value set in the form. However, I didn’t know how to access that value. After many Google searches, perusal of the Django documentation, and even a failed attempt to follow the template rendering code, I was stumped. As a wild guess, I tried the .data
attribute, and it worked! Here’s the updated (and functional) code:
input type="radio" name="reg_choice_1" value="{{ cls.id }}" {% ifequal srform.reg_choice_1.data cls.id %}CHECKED{% endifequal %}
I hope that I’ve put enough keywords in this post so that future stumped Djangonauts can find some help.
As always, leave your comments below.