Worldwide MQTT Without Exposing a Local Broker to the Public Internet

An article on Steve’s Internet Guide got me thinking about MQTT when on the road. No, seriously!  Here’s my situation:

I have a sensor that reports the usual temperature and humidity inside my travel trailer, and it also provides the same information from inside the RV refrigerator thanks to a Ruuvi tag placed within. (Code for this sensor using ESPHome can be found in my Github repo.)

This sensor connects to my home WiFi network when the trailer is parked in the driveway, but obviously not when we are on a trip.  We can use the Ruuvi app to read the fridge temperature using BLE when inside the unit, but if we are even across the campsite or farther away – no go.

Anyone who has relied on an RV refrigerator to keep their food safe knows that they can be less than reliable.  Knowing the temperature is rising gives us a chance to take action before food items are spoiled.  

My goal with this project was to provide a way to monitor the refrigerator temperature when the trailer is parked away from home, and to use Home Assistant to issue alerts that will inform us of any temperature issues.

WiFi First

Before I get into the MQTT configuration, let’s make sure we connect to WiFi when away.  My trailer has a travel router that can bridge to RV park networks, a hotspot, or a tethered smartphone to bring Internet access inside.  ESPHome allows one to set up more than one set of WiFi options, so I use this code:

wifi:   
networks:
  - ssid: !secret rv_ssid
    password: !secret rv_password
  - ssid:  !secret wifi_iot_ssid
    password: !secret wifi_iot_password
    manual_ip:
      static_ip: $ip_address
      gateway: 192.168.37.1
      subnet: 255.255.255.0
      dns1: 8.8.8.8
      dns2: 1.1.1.1

On the home network I use a static IP address to make connection time fast.  This, along with deep sleep,  helps limit the time the device is awake when using a battery operated-sensor.

MQTT in the Wild

In order to collect the telemetry data from this sensor when away, I set up a free account with the service flespi.  However, one can use any one of many services offering an MQTT broker service, as my solution is simple and standards-based.  Thanks to the good folks at flespi for making this service available for trials and hobbyists.

Flespi assigns a long token for authentication, and this can be provided as either a username or password when logging in.  Here is my code:

  mqtt:
    id: mqtt_client
    broker: !secret flespi_broker
    port: !secret flespi_port
    username: !secret flespi_token
    birth_message:
      topic: $device_name/birthdisable
      payload: disable
    will_message:
      topic: $device_name/willdisable
      payload: disable

(Sorry for all the secrets; gotta keep things private and portable)

MQTT Explorer Screen Capture

MQTT Explorer shows the broker contents once the device is connected and transmitting.

The Bridge

Finally, we need to make a safe connection between the local broker and the public one.  I use the Mosquitto broker locally, and it offers an MQTT bridge functionality that makes this task easy.

What we are going to do is have the local broker subscribe to the topics created on the external broker.  In my example, the sensor uses topics with the prefix “trailer/”.  In addition, ESPHome uses a topic prefix to send configuration information to Home Assistant.  The default topic for this is “homeassistant/”.

To define these subscriptions, we will modify the mosquitto.conf file to connect to the external broker and subscribe.  Add this code to your file:

#  Bridge to public broker

connection <MQTT connection name>
address <broker url or IP with port number, such as example.com:1883>
remote_username <use your token here>
bridge_insecure false
try_private false
bridge_protocol_version mqttv50

topic homeassistant/# in 0
topic trailer/# in 0

The top section of this snippet sets up the broker to broker connection.  Your version may be different if using another service.  Note that I am using an unsecured connection; you may wish to beef things up.

The lower section is what defines the subscriptions.  The topic “homeassistant/#” uses the octothorpe wild card symbol to grab anything under this topic prefix.  The configuration information for the sensor is found here.

The “trailer/#” topic is used to identify the sensor readings.  Whenever something is posted to this topic prefix, it will be retrieved through the bridge to the local broker – ready for consumption by Home Assistant or any other user.

Wrap Up

I was surprised that I could do this without any additional cost and limited coding.  Now that the data is feeding to Home Assistant, I can write automations, alerts, etc. to keep us informed.

Thanks again to Steve of Steve’s Internet Guide for the inspiration and examples.

Passing a Parameter to a Django Form

This is somewhat related to my previous post. I didn’t want to include my user profile value in the form and instead add it during validation. This caused the model validation to be skipped in part. I had to do some gymnastics to get it to run.

In this latest attempt, I have instead included that profile as a hidden field and compare it to the current user profile. Much cleaner! Now all I have to to is get that current value into the form.

It didn’t take long on Google to find several examples. Here is my rendition:

In the view, I overrode the get_form method to add the profile as a paramter

def get_form(self, form_class=None):

    if form_class is None:
        form_class = self.get_form_class()
    return form_class(**self.get_form_kwargs(),current_user_profile=get_profile(self.request.user))

Next, in the form I need to pop that value from the kwargs dictionary as assign to a object variable

class LinkForm(ModelForm):

        def __init__(self, *args, **kwargs):

            self.current_user_profile = kwargs.pop('current_user_profile')

            super(LinkForm, self).__init__(*args, **kwargs)

            self.fields['profile'].widget=HiddenInput()

I also set the profile field to hidden here.

Now I can write a clean method for the profile field like so:

def clean_profile(self):

        profile = self.cleaned_data.get('profile',None)

        if profile != self.current_user_profile:
            self.add_error(None,ValidationError('Web page altered. Try again.', code='wrong_profile'))

        return profile

This is much easier and clearer than my previous method. Not bad for working late on a Wednesday night.

Execute validate_unique when form doesn’t include all fields

Asked this question on Stack Overflow today:

https://stackoverflow.com/questions/48449821/execute-validate-unique-when-form-doesnt-include-all-fields

I recently had a situation where the validate_unique method from my model wasn’t running. This is because one of the fields involved in the unique test wasn’t included in the form. I tried many things in the form and the view before landing on this solution: I first injected the field into the object of the UpdateView, then ran the test in the Form in _post_clean.

models.py 

class Link(ModelBase):
	id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
	title = models.CharField(max_length=200,blank=True,null=False)
	url = models.URLField(max_length=400,blank=False,null=False)
	profile = models.ForeignKey('Profile',null=False,blank=False,on_delete=models.CASCADE)

	class Meta:
		unique_together = ('url','profile')

	class Admin:
		pass
forms.py 

class LinkForm(ModelForm):

	def _post_clean(self):
		''' Be sure that the instance validate_unique method is run including the profile field '''
		super(LinkForm,self)._post_clean()
		try:
			self.instance.validate_unique(exclude=None)
		except ValidationError as e:
			self._update_errors(e)

	class Meta:
		model = Link
		fields = ['title','url']
views.py 

class LinkUpdateView(UpdateView):
	model = Link
	form_class = LinkForm

	def get_form_kwargs(self):
		''' Add profile to self.object before kwargs are populated '''

		if hasattr(self, 'object') and self.object and self.profile:
				self.object.profile = profile

		kwargs = super(LinkUpdateView, self).get_form_kwargs()

		return kwargs

Is there a better way to do this that doesn’t involve overriding an internal function?

Got any help for me? Please post an answer here or on SO. I’ll share the solution here once I get one.

Garage Door Automation Rig

My first rule of home automation is not to ask why I need something automated, but if it is possible. It’s a hobby after all. However, with the garage door we had an actual need. (Plus, I could do it!)

The whole point of having a garage door is so we can close it when access isn’t needed. Otherwise there would just be a hole in the wall. However, in my house we have this bad habit of not closing it – sometimes when we are home and sometimes when we leave. This project was inspired by an all night “open to all” garage.

There were also a couple of articles that outlined how this could be done:

Wifi Garage Door Controller using a Wemos D1 Mini (I pulled much of my code from this project)

Open the Garage Door with a Phone and Particle Core | Make

Project goals:

  • Ability to open and close the door remotely
  • Report the door’s status
  • Use Home Assistant to interface with MQTT to display status and issue commands
  • Use automation to close the door when left open too long
  • Just recently I added a temperature sensor to report the temp in the garage

Parts List:

D1 Mini with Headers

UPDATE: I’ve replaced the Magnetic Reed Switch with a much hardier commercial model. The Enforcer works much better!

One could substitute other WiFi enabled project boards, such as the Node MCU. I used the Wemos partly because of the availability of the shields which make for a compact build. The ProtoBoard shield was not required, but helped me get past my substandard soldering skills.

Pin Layout:

  • 5V – Power to the relay
  • 3.3V – Power to the BMP280
  • GND – Ground
  • D1 – Relay control (this is hardwired in the shield)
  • D2 – Reed switch
  • D5 – SCL for BMP280 (not the default pin)
  • D6 – SDA for BMP280 (not the default pin)

I don’t usually solder all of the pins on the D1 shields; just those that are needed. However, there is no penalty for completeness. Be sure all of the pins listed above are soldered in all headers.

The Code

This code is doing three things each cycle through the loop. First, it checks the state of the door and reports any changes. Next, the temperature and pressure are checked and reported. Finally, the client loop will look for any commands from MQTT and handle accordingly.

The code is available on GitHub: https://github.com/dashdrum/garage_door

Check Door State

If the door is found to be open, it is not reported right away. Instead a counter is incremented, and only after five consecutive readings of open is that status returned. The goal here is to ignore spurious readings.

A return value of closed is assumed correct and set on first reading.

A change in the door status is immediately reported via MQTT. The process also reports the status each minute regardless of a change to ensure Home Assistant is always in sync.

Report Temperature and Pressure

This code is based on the example program that came with the library by github user mahfuz195 (found here). The readings are returned in JSON format using a library by github user bblanchon (found here)

Updates are limited to once every 10 minutes. No need to overwhelm Home Assistant with readings.

In the begin() function for the BMP, I specify which pins to use for the I2C interface. On the D1 Mini these default to D1 and D2, but those pins were already in use by the relay and reed swtich. D5 and D6 are recruited to take the job.

Callback

The callback routine catches a MQTT message. The payload of “OPEN” is the only command that will “push” the button – closing the realy for 6/10 of a second.

Other Code Features

I’m using a few other libraries in this program. WiFiManager handles the WiFi connection, storing the credentials in nonvolitile memory. It is important to note that WiFiManager will open a WiFi acces point if no credentials are found or the connection attempt fails. Connect to this AP using a laptop or phone, access 192.168.4.1, and follow on the onscreen instructions to setup your WiFi information.

ESP8266HTTPUpdateServer accepts program updates over the air. No need to climb up a ladder to connect your laptop!

Wiring the Reed Switch

High end magnetic reed switch
The Enforcer – Commercial Garage Door Magnetic Switch

Using a bell wire pair, connect one wire to the 3.3V pin, and the other to D2. Also, I’ve added a pull-down resistor between D2 and ground. The pull-down keeps the signal from “floating” (sending an unknown level). On the other end, use the NO (normally open) terminal for one wire and the common terminal for the other.
ProtoBoard with door switch connection

Wiring the Temperature Sensor

Connect the four pins of the BMP280 sensor as labeled. Rember that my code uses D5 and D6 for SDA and SCL, not the defaults of D1 and D2.
BMP280 temperature/pressure sensor

Wiring the Relay

Your garage door opener should have at least four terminals – two for the open button and two for the safety circuit. Follow the wires from your open button to find the two you need. Otherwise, use a short piece of wire to short the pairs of terminals together until the door opens.

Run a pair of bell wire from the opener terminals to the relay. Use the common terminal and the NO terminal on the relay.
D1 with relay shield

Set Up Home Assistant

The GitHub repository includes a configuration file for Home Assistant. Integrate these entries into your configuration.yaml file. Note that this file itself is not a complete configuration file.
Home Assistant status page
Once again the GitHub address is: https://github.com/dashdrum/garage_door

Please leave your questions or suggestions below.

Fun with Pagination – Back and Ahead by 5

Yesterday I wanted to beef up a ListView-based page with additional navigation controls.  Specifically, I decided to add buttons to jump back 5 and ahead 5 pages in the list.  With a little bit of class extension, this was easy to do.

Source:

from django.core.paginator import Page, Paginator

class PageFive(Page):class PageFive(Page):

    ''' Include info for next 5 and previous 5 pages '''

    def has_next_five(self): 
        return self.number < self.paginator.num_pages - 5 

    def has_previous_five(self): 
        return self.number > 6

    def next_five_page_number(self): 
        return self.paginator.validate_number(self.number + 5)

    def previous_five_page_number(self): 
        return self.paginator.validate_number(self.number - 5)


class PaginatorFive(Paginator):

 ''' Uses the PageFive class to report info for next and previous 5 pages
     Set pageinator_class in ListView to use '''

    def _get_page(self, *args, **kwargs): 
        """ 
        Return an instance of a single page using the PageFive object 
        """ 

        return PageFive(*args, **kwargs)

First step was to extend the Pageclass to add additional methods.  Mimicking the existing methods for next and previous pages, these four new functions return information on the previous five and next five pages of the list.

To use my new PageFiveclass, I also extended the Paginator class.  Lucky for me, the authors have included the _get_page()method as a hook to modify the Pageclass used.  Simple to override.

Next, I have to tell ListViewto call my redefined paginator.  The paginator_classvariable is used to make that change:

class MyListView(ListView):
    model = MyModel
    paginate_by = 10
    paginator_class = PaginatorFive

Finally, I can use the new methods of the PageFiveclass in the template:

<!-- PaginatorFive navigation buttons -->
 {% if is_paginated %}
   <ul class="pagination pagination-centered">
     {% if page_obj.has_previous %}
       <li><a href="?page=1">First</a></li>
       {% if page_obj.has_previous_five %}
         <li><a href="?page={{ page_obj.previous_five_page_number }}">Back 5</a></li>
       {% endif %}
       <li><a href="?page={{ page_obj.previous_page_number }}">Prev</a></li>
    {% endif %}

    <li class="active" ><a href="?page={{page_obj.number}}">{{page_obj.number}} of{{ page_obj.paginator.num_pages }}</a></li>

    {% if page_obj.has_next %}
       <li><a href="?page={{ page_obj.next_page_number }}">Next</a></li>
       {% if page_obj.has_next_five %}
         <li><a href="?page={{ page_obj.next_five_page_number }}">Ahead 5</a></li>
       {% endif %}
       <li><a href="?page={{ page_obj.paginator.num_pages }}">Last</a></li>
     {% endif %}
   </ul>
 {% endif %}

This was a fun little exercise.

 

(Sorry for the word wrap in the code.  Copy to your editor for a better view)

Django, Gunicorn, Nginx, & HTTPS

Django, gunicorn, nginx, and https logosI can find many examples of using Django, Gunicorn, and Nginx for an application, and also ways to implement HTTPS on Nginx, but not all together.  This post will document my final configurations and explain my decisions.

First, the code:

nginx.conf:

worker_processes  auto;

events {
    worker_connections  1024;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;

    ## Rewrite http to https

    server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;
        return 301 https://$host$request_uri;
    }

    ## Use https

    server {
        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";

        listen              443 ssl;
        server_name         your.domain.xxx;
        ssl_certificate     <path-to-certificate>;
        ssl_certificate_key <path-to-private-key>;
        ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers         HIGH:!aNULL:!MD5;

        location = /favicon.ico { access_log off; log_not_found off; }


        location / {
            proxy_set_header Host               $host;
            proxy_set_header X-Real-IP          $remote_addr;
            proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host   $host:443;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-Port   443;
            proxy_set_header X-Forwarded-Proto  https;
            proxy_read_timeout 300s; 
            proxy_pass http://localhost:8000;
        }
    }
}

gunicorn.py:

##bind = ":8000"
workers = 3
timeout = 300
proc_name = "dupe"
user = "<your user>"
group = "<your group>"
raw_env = []
capture_output = True
loglevel = "debug"
errorlog = "<path-and-name-for-log-file>"

gunicorn.service:

[Unit]
Description=gunicorn daemon
After=network.target

[Service]
Environment=DJANGO_DEBUG=False
Environment=DJANGO_SECRET_KEY=<your SECRET_KEY>
Environment=DJANGO_ALLOWED_HOSTS=[*]
Environment=DJANGO_STATIC_ROOT=<application.path>/staticfiles
Environment=DJANGO_SETTINGS_MODULE=<project>.settings.devl
User=<your user>
Group=<your group>
WorkingDirectory=<application.path>
ExecStart=<virtualenv.path>/bin/gunicorn -c <application.path>/gunicorn.py  --bind 0.0.0.0:8000 <project>.wsgi:application

[Install]
WantedBy=multi-user.target

Notes:

nginx.conf holds all of the interesting stuff

The first server section grabs any http traffic coming in on port 80 and redirects it to the https URL on port 443.

Handling https traffic was trickier. We need to pass all of the header settings from the incoming request on to gunicorn/Django at localhost. All of the proxy-set attribute values in the location section came from a little bit of trial and error.

The setting for proxy-read-timeout is admittedly high – five minutes! This application has one longer running transaction that was exceeding the 60 second default. This value needs to match the timeout setting in the gunicorn settings. I’ll probably lower it to a more respectable two minutes before hitting production.

Finally, astute readers will notice that there is no code to handle requests for static resources. This is because I am using Whitenoise to route those calls through Django. This works only because my app just has eight users and Django can handle the load. Anything that is public facing should use Nginx to serve static files directly (or look into using a CDN, etc.)

Not much in gunicorn.py

I’m using 3 workers – plenty for my expected load. The timeout is set to 300 seconds, which is the same found in nginx.conf above. These should match.

The bind setting is commented and moved to gunicorn.service so that I can use this file for more than one instance of the application. A second service file can specify a unique port for its copy of gunicorn.

gunicorn.service – Instance specific settings

Since my app is stored in a git repo (on a company server, not github), I don’t want to include any security information like the SECRET_KEY in a versioned file. Therefore, this guy is not included there – although an example file is.

The Environment variables set here are read by Django at startup.

The ExecStart line starts by running the version of gunicorn found in the virtualenv for this instance. Setting up a virtualenv is beyond this scope of this post, but I hope that if you’ve gotten this far you know all about them.

Note that the bind parameter includes the port number used for this instance of gunicorn as mentioned above.

Questions? Comments?

I hope this helps someone setting up a Django app. Please drop a note in the comments with your experience. I’d also invite questions, as well as comments on how I can improve my configuration. Thanks all!
 

Django and LDAP

LDAP GraphicA project at work requires that I authenticate users using an LDAP service. Who knew it would be so easy?!?

I’m using Django 1.9.2, but I imagine that this will work with other versions +/- a few releases. Python version is 3.5.

INSTALL

The library that does the magic is django-auth-ldap. For Python 3.x, pyldap is required, while python-ldap is used for Python 2.x. I also had to install some system libraries: libsasl2-dev, python-dev, libldap2-dev, and libssl-dev (on Ubuntu).

AUTHENTICATION_BACKENDS

The AUTHENTICATION_BACKENDS setting is not defined in the default Django settings.py file using v1.9.2. However, the code I use below expects the variable to exist. Therefore, I added it in with the default setting. You may have other backends already in use.

AUTHENTICATION_BACKENDS = (
    'django.contrib.auth.backends.ModelBackend',
)

LDAP SETTINGS

This code lives at the end of settings.py. Use your own LDAP URI, ou (Organizational Unit), and dc (Domain Component).

#-----------------------------------------------------------------------------#
#
#   LDAP Settings
#
#-----------------------------------------------------------------------------#

AUTHENTICATION_BACKENDS += ('django_auth_ldap.backend.LDAPBackend',) 

AUTH_LDAP_SERVER_URI = "ldaps://your.ldap.server"

AUTH_LDAP_USER_DN_TEMPLATE = "uid=%(user)s,ou=users,dc=example,dc=com"

The first line adds an additional authentication backend to the tuple. Next, we define the path to the server. Note that I’ve used a secure protocol (ldaps). This is essential to keep the credentials passed to the server encrypted. The third line sets up a direct bind with the supplied user.

In place of the direct bind method, the documentation also suggests a user search configuration. I was able to get this to work as well, but it requires that your password be included in settings.py (or you are performing some trickery to get the password). My LDAP admin is not fond of this method, but will create an application-only account if needed. Here is the user search configuration:

import ldap
from django_auth_ldap.config import LDAPSearch

AUTH_LDAP_BIND_DN = "<user>"
AUTH_LDAP_BIND_PASSWORD = "<password>"
AUTH_LDAP_USER_SEARCH = LDAPSearch("ou=users,dc=example,dc=com",
    ldap.SCOPE_SUBTREE, "(uid=%(user)s)")

I’m sorry to say that I’m not an LDAP expert at all, so I can’t supply much advice if this doesn’t work for you. Buy your LDAP admin a lunch sometime and ask your questions while waiting for the food.

Best of luck!!

Looking for OSS or Pro Bono Django Work

I’m settling in to a new job in a new town, and life is pretty good. Unfortunately, my duties don’t include any chance to program in Django. Our main software vendor is moving from Oracle Forms to Groovy/Grails, so I’m sure I’ll be learning more about that platform.

However, I would still like to keep my Django skills sharp, so I am offering my services to any non-profit group that needs an extra Django hand. No salary is expected, just a chance to get involved.

An open source project would be a great place for me to assist, and would also be a learning experience for me. Is there a project out there that needs a contributor with Django knowledge?

A charity or other non-profit would be another option. Perhaps there is an association developing OSS software for non-profits to use. With my experience supporting fundraising and higher education, I feel I could make a meaningful contribution.

If you have an opportunity fitting the description above, or know someone who does, please send me a message dan@dashdrum.com. I am ready to help!