As described in some earlier posts, I have a setup at home with IoT devices that publish measurement messages to a Raspberry Pi via MQTT. The RPi stores the data in a database and also forwards the messages to a cloud service (Adafruit IO).
In this post I have made a self-hosted data visualization web app that can be accessed from any browser-enabled device.
Overview
For the web front end and back end, I use my current favorite combination of frameworks for Raspberry Pi for web development – Flask and AngularJS. As I will involve a database and some charts, the total setup includes:
- Flask as a microframework for a REST API and for hosting the initial SPA html page.
- pymongo for accessing MongoDB via Python
- angular-chart.js for AngularJS directives with Chart.js
- AngularJS for creating the Single-Page Application
I have hosted the environment on a Raspberry Pi 3 with Raspbian OS, but any OS that has Python support should work.
The complete code for this project can be fetched from https://github.com/LarsBergqvist/IoT_Charts. To try this project out, fetch the code, install the JavaScript dependencies, install the Python dependencies and start the web server. Then, browse to http://<RASPBERRY_IP>:6001/ChartData.
About the project setup
Flask requires a certain folder setup:
- / – the root folder where the Flask web server Python script is placed
- /templates – the folder where html templates are placed (only one html file is needed for a SPA)
- /static – the folder where the AngularJS controllers, CSS and third-party JavaScript dependencies are placed
As Flask will render html pages with Jinja2 that uses {{}} for placeholders, a workaround for AngularJS has to be done by changing the default {{}}-binding syntax to something else, [[]] in my case.
The application uses angular-chart.js that depends on AngularJS and Chart.js. I will use npm for installing these libraries, so the first step is to install npm on the RPi (if not already done):
sudo apt-get install npm
Then, create a folder named static, move to this folder and fetch angular-char.js, AngularJS and Chart.js:
npm install angular-chart.js --save npm install angular --save npm install chart.js --save
A subfolder to static called node_modules will be created where the JavaScript libraries are downloaded.
Creating the back end with Flask
Flask is used for creating a REST API. If not installed for Python 3.* already, use pip to install it:
sudo pip3 install flask
As pymongo also will be needed, we need to install that library as well:
sudo pip3 install pymongo
Now, in the root folder, create a new Python script:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python3 | |
| from flask import Flask, render_template, jsonify, request | |
| from datetime import datetime, timedelta | |
| import sys | |
| import data_fake | |
| import data_mongodb | |
| app = Flask(__name__) | |
| def get_labels_and_values_for_topic(topic_name, numdays): | |
| if (numdays < 1): | |
| numdays = 1 | |
| if app.config['FAKE'] == False: | |
| repo = data_mongodb.MongoDBRepository | |
| else: | |
| repo = data_fake.FakeRepository | |
| return repo.get_data(repo,topic_name,numdays) | |
| @app.route("/ChartData") | |
| def index(): | |
| return render_template('index.html') | |
| @app.route('/ChartData/api/<string:location>/<string:measurement>') | |
| def get_measurements_as_labels_and_values(location,measurement): | |
| numdays = request.args.get('numdays', default=1, type=int) | |
| topic = "Home/" + location + "/" + measurement | |
| # Get all measurements that matches a specific topic from the database | |
| # Fetch data from today and numdays backwards in time | |
| # The measurements are split into two arrays, one with measurement times (=labels) | |
| # and one with the actual values. | |
| labels, values = get_labels_and_values_for_topic(topic,numdays) | |
| return jsonify({"measurements":{'labels':labels,'values':values}}) | |
| if __name__ == "__main__": | |
| for arg in sys.argv: | |
| if arg.lower() == "–fake": | |
| print("Using fake data") | |
| app.config['FAKE'] = True | |
| else: | |
| app.config['FAKE'] = False | |
| app.run(host='0.0.0.0', port=6001,debug=False) |
There is a main entry <IP>:6001/ChartData that serves up the initial html template for the Single Page application. The REST API URI:s are built up from <IP>:6001/ChartData/api/<location>/<measurement>. This categorization is based on how I have defined my IoT data (see my previous MQTT IoT posts). For example:
ChartData/api/Outdoor/Temperature - gets the data from the temperature sensor located outdoors. ChartData/api/GroundFloor/Humidity - gets the data from the humidity sensor on the ground floor.
The data can be fetched from MongoDB (how this data is stored is described in https://larsbergqvist.wordpress.com/2016/06/26/a-self-hosted-mqtt-environment-for-internet-of-things-part-3/) or from a fake repository (just to get some data for the front end development without a complete database).
The repository implementations are defined in two separate files:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from datetime import datetime, timedelta | |
| from repository_base import Repository | |
| class FakeRepository(Repository): | |
| def get_data(self,topic_name, numdays): | |
| today=datetime.today() | |
| # Create some test data | |
| # Three values from yesterday and three values from today | |
| dataPoint = ( | |
| (20.1,today–timedelta(1.8)), | |
| (19.1,today–timedelta(1.4)), | |
| (22.1,today–timedelta(1.2)), | |
| (23.3,today–timedelta(0.9)), | |
| (19.1,today–timedelta(0.5)), | |
| (30.1,today–timedelta(0.1)) | |
| ) | |
| values = [] | |
| labels = [] | |
| for value,time in dataPoint: | |
| if (time > (today – timedelta(numdays))): | |
| values.append(value) | |
| labels.append(super(FakeRepository,self).date_formatted(time)) | |
| return labels, values |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pymongo | |
| from datetime import datetime, timedelta | |
| from repository_base import Repository | |
| class MongoDBRepository(Repository): | |
| def get_data(self,topic_name, numdays): | |
| mongoClient=pymongo.MongoClient() | |
| db=mongoClient.SensorData | |
| yesterday=datetime.today() – timedelta(numdays) | |
| cursor = db.home_data.find({"topic":topic_name,"time":{"$gte":yesterday}}).sort("time",pymongo.ASCENDING) | |
| values = [] | |
| labels = [] | |
| for r in cursor: | |
| values.append(r['value']) | |
| labels.append(super(MongoDBRepository,self).date_formatted(r['time'])) | |
| return labels, values |
These classes are based on an abstract base class that defines the common interface and the shared base method:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from datetime import datetime, timedelta | |
| class Repository: | |
| """Abstract base class for the repositories. | |
| The sub classes should implement the get_data method.""" | |
| def date_formatted(date): | |
| return date.strftime('%d %b %H:%M') | |
| def get_data(self,topic_name, numdays): | |
| raise NotImplementedError( "Should have implemented this" ) |
With these back end files in place, we can test the REST API. First start the Flask server:
python3 charts_server.py --fake
The –fake argument makes the implementation use the fake repository so that you don’t need a database with data to test the API. When browsing to http:<IP>:6001/ChartData/api/Outdoor/Temperature, you will get JSON back:
{
measurements: {
labels: [
'09 Jul 22:24',
'10 Jul 08:00',
'10 Jul 17:36'
],
values: [
23.3,
19.1,
30.1
]
}
}
As you can see, the measurements are divided into an array of values and an array of labels. This is because the api provides ChartData that will fit the front end charts easily.
The query parameter numdays can be used for fetching data older than 24 hours, e.g. <IP>:6001/ChartData/api/Outdoor/Temperature?numdays=3.
Creating the front end
The front end consists of a html page in the templates folder and a JavaScript controller in the static folder:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html ng-app="app" lang="en"> | |
| <head> | |
| <meta charset="utf-8" /> | |
| <title>Sensor data</title> | |
| <script src="/static/node_modules/angular/angular.js"></script> | |
| <script src="/static/node_modules/chart.js/dist/Chart.min.js"></script> | |
| <script src="/static/node_modules/angular-chart.js/dist/angular-chart.min.js"></script> | |
| <script src="/static/app.js"></script> | |
| </head> | |
| <body> | |
| <h1>Sensor data</h1> | |
| <div ng-controller="ChartCtrl"> | |
| <label>Number of days: | |
| <input type="number" min="1" ng-change="numdaysChanged()" ng-model="numdays" /> | |
| </label> | |
| <div ng-repeat="item in data"> | |
| <h2 ng-bind="titles[$index]"></h2> | |
| <canvas id="line" class="chart chart-line" chart-data="data[$index]" | |
| chart-labels="labels[$index]" chart-series="series" chart-options="options"" chart-dataset-override="item.datasetOverride" chart-click="onClick"> | |
| </canvas> | |
| </div> | |
| </div> | |
| </body></html> | |
| Status |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| var myApp = angular.module('app', ["chart.js"]) | |
| .config(function($interpolateProvider) { | |
| $interpolateProvider.startSymbol('[[').endSymbol(']]'); | |
| }); | |
| myApp.controller("ChartCtrl", function ($scope,$http) { | |
| $scope.numdaysChanged = function() { | |
| requestNewData(); | |
| }; | |
| var lastVal = function(array, n) { | |
| if (array == null) | |
| return void 0; | |
| if (n == null) | |
| return array[array.length – 1]; | |
| return array.slice(Math.max(array.length – n, 0)); | |
| }; | |
| var getData = function(index,topic,numdays) { | |
| $http({ | |
| method: 'GET', | |
| url: "/ChartData/api/" + topic + "?numdays=" + numdays | |
| }).then(function successCallback(response) { | |
| values = response.data.measurements.values; | |
| labels = response.data.measurements.labels; | |
| $scope.data[index] = [values]; | |
| $scope.labels[index] = labels; | |
| title = topic + ": " + lastVal(values,1) + " (" + lastVal(labels,1) + ")"; | |
| $scope.titles[index] = title; | |
| }, function errorCallback(response) { | |
| }); | |
| }; | |
| $scope.numdays = 1; | |
| var requestNewData = function() { | |
| $scope.data = []; | |
| $scope.labels = []; | |
| $scope.titles = []; | |
| $scope.series = ['Sensor']; | |
| $scope.datasetOverride = [{ yAxisID: 'y-axis-1' }]; | |
| $scope.options = { | |
| scales: { | |
| yAxes: [ | |
| { | |
| id: 'y-axis-1', | |
| type: 'linear', | |
| display: true, | |
| position: 'left' | |
| } | |
| ] | |
| } | |
| }; | |
| getData(0,'Outdoor/Temperature',$scope.numdays); | |
| getData(1,'GroundFloor/Temperature',$scope.numdays); | |
| getData(2,'Garage/Temperature',$scope.numdays); | |
| getData(3,'Outdoor/Humidity',$scope.numdays); | |
| getData(4,'GroundFloor/Humidity',$scope.numdays); | |
| getData(5,'Garage/Humidity',$scope.numdays); | |
| }; | |
| requestNewData(); | |
| }); |
The controller makes asynchronous http requests to the REST API and updates the bound $scope properties with the result when the data has been fetched.
With the front end in place and using the fake data, you will get a highly responsive SPA app that display 3 data points for today and yesterday for each sensor that is defined (the same fake is used for each sensor).
With real data, you will get graphs like these, with a data point per recorded sensor value:
For each graph, there is a title that contains the sensor topic plus the latest value and its recording time.
If you want to play with real data, I have made a JSON export from my MongoDB database. It can be downloaded from:
https://github.com/LarsBergqvist/IoT_Charts/tree/master/DataBaseExport
Addendum, March 2017
I’ve started storing my sensor data in an InfluxDB time series database. For the chart app I have made an additional repository that fetches the data from this source. InfluxDB is now the default source for the charts app, but MongoDB or a fake repository can still be selected with startup arguments for the app.
The code is updated in GitHub:
https://github.com/LarsBergqvist/IoT_Charts
Thingsmatic 
2 Thoughts