Stumbles is Ben Sturmfels

Getting started with the GnuBee Personal Cloud 2 (or 1)

2018-02-01T00:00:00+11:00

I just received my GnuBee Personal Cloud 2 in the post. It's a device newly manufactured specifically to run completely free software, making in completely unique. The device is designed for network accessible storage, providing slots for six 3.5" hard drives.

As a brand new product, the documentation for getting started is nearly non-existant though, so here's my journey so far.

Assembly

I was surprised to see that the holes for the circuit board mounting brackets didn't all line up. It turns out that the centre brackets are oriented differently to the end brackets. Look carefully at the photo below.

The GnuBee Personal Cloud 2 (from below)

Getting started

The GBPC comes with the LibreCMC operating system installed in the on-board flash memory.

Based on the Background information for GnuBee PC1, I plugged a network cable in to the black socket and plugged the other end in to my computer. I then tured on the GBPC using the silver switch (red button on GBPC1). No SDCard was installed at this point.

Once it had started up, the computer connected to the GBPC's network. Visiting http://192.168.10.1/ in a web browser showed the LibreCMC login screen. Although it's not made clear, you can initially log in as user "root" by leaving the password blank. You can also connect with SSH with no password by running ssh -o StrictHostKeyChecking root@192.168.10.1 with no password. I needed the -o StrictHostKeyChecking as I also have a GBPC1 that uses the same IP address.

Proceeding to the System/Administration section allows entry of a root password. I set up key-based SSH access at the same time. Leave "Interface" as "unspecified", uncheck password authentication and enter your SSH public key fingerprint (from .ssh/id_rsa.pub). Select "save and apply" at the bottom of the page. I was then able to log out and log back in with my new password, and connect with SSH to root@192.168.10.1.

Unfortunately I notied that after switching the device off and back on again, the password and SSH changes were reset. I'm not sure why.

Resources

The GnuBee devices are not well documented at the moment, but at present the best information can be found at:

Installing hard drives

I installed a hard drive in slot 1 and after resetting the GBPC with the black reset button, noted that the new drive was listed under the System/Mount Points section as filesystem "crytpo_LUKS". The second drive was an unencrypted "ext4" drive. That didn't show up under "Mount Points" automatically and didn't seem to mount when I selected "Add", chose the drive from the list and defined a custom mount point /backup. There was no problem mounting this drive from the SSH command line though.

Further work

At this point I still have a lot of questions, and not enough time to investigate them all. If you know the answers, please get in touch.

Do I need to install a different operating system to do something useful with the device? I was a little suprised to find that the GBPC doesn't do anything much out of the box. Not that this in any way limits the device's potential, but it's nice not to have to do everything yourself. It seems that all drive mounting must be done manually, that there's no file sharing services or even rsync installed by default.
How do you install software on LibreCMC? I tried the System/Software section of the web interface, but installing or searching for packages fails with an error connecting to the package repositories.
How do you boot from the SDCard included in my "deluxe pack"? The card appears to have Debian installed, but after insterting the SDCard and resetting, the device still seem to boot to LibreCMC. Noting that the SDCard is auto-mounted if inserted while LibreCMC is running.
What do the different network ports do? Black is a LAN port. Blue seems to be the WAN port as the device can ping the outside world once plugged in. The yellow port doesn't seem to do anything.
Can drives be made to auto-mount? I noticed that plugging in a USB drive are mounted automatically under /tmp/run/mountd/.
How do you connect with the USB-to-UART cable? I tried plugging it in, connecting with picocom /dev/ttyUSB0 9600 and resetting the GBPC, but only see garbage on the screen. Clearly I need different connection settings.
Why don't password changes stick? Cleary the device is accepting some changes, because if I mkdir /mnt/backup, the directory is still there after a power off and on.
Can it be configured to be a DHCP client, rather than a DCHP server?

Update 5 Feb 2018

I learnt that USB-to-UART works via minicom, eg. sudo minicom -b 57600 -D /dev/ttyUSB0. See the GnuBee documentation. I presume that minicom must use different defaults to picocom.

It appears that a (firmware update)[https://groups.google.com/forum/#!topic/gnubee/vbKwd7r-8_8] is required to allow booting from the provided SDCard.

The network interface for LibreCMC can be changed from "static" to "DHCP client".

I'm having trouble mounting some encrypted drives. It appears that missing kernel support for "aes-xts-plain64" may be the cause.

Update 7 Feb 2018

Plugged two RAID1 drives into the GBPC2 and it correctly recognised the three RAID partitions and allocated them to /dev/md125, /dev/md126 and /dev/md127.

The opkg update command, as well as the web interface fail to update the package list with:

Downloading https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/ramips/mt7621/packages/Packages.gz
*** Failed to download the package list from https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/ramips/mt7621/packages/Packages.gz

Downloading https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/packages/mipsel_24kc/base/Packages.gz
*** Failed to download the package list from https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/packages/mipsel_24kc/base/Packages.gz

Downloading https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/packages/mipsel_24kc/packages/Packages.gz
*** Failed to download the package list from https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/packages/mipsel_24kc/packages/Packages.gz

Collected errors:
 * opkg_download: Failed to download https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/ramips/mt7621/packages/Packages.gz, wget returned 5.
 * opkg_download: Failed to download https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/packages/mipsel_24kc/base/Packages.gz, wget returned 5.
 * opkg_download: Failed to download https://librecmc.org/librecmc/downloads/snapshots/v1.4.1/packages/mipsel_24kc/packages/Packages.gz, wget returned 5.

In contrast, my LibreCMC-based router from ThinkPenguin updates and installs packages with no problems, so clearly something just isn't configured correctly.

Password changes do appear to be saved, and persist across reboots and powering off. It appears though that the small black button does a "factory reset", which resets the LibreCMC login credentials.

The hard drive idling plugin available through the web interface works well.

Slot #1 on the GBPC2 is blocked by the two 12V plastic clips. This is a minor design fault, but you can simply bend the four plastic tabs firmly down to fix the problem.

Configuring Android K-9 Mail from the command line

2017-10-06T00:00:00+11:00

I hate having to re-enter my email settings in a new or reinstalled device. Secure passwords are particularly painful to transcribe via a phone keyboard, but it's not just the basic settings. It's also those extras like signature, disabling notifications and folders that buy you if they're not right. I've set it all up dozens of times before and will no doubt do it many more.

On my laptop and desktop I have a permanent, version controlled copy of my email settings that I can apply automatically. Surely you could do the same on Android/Replicant? I couldn't find any useful information about people doing this with configuration management tools like Puppet, Salt, Ansible or anything else so here's my first attempt:

#!/usr/bin/env bash

set -x

# Set the K-9 incoming and outgoing account details via ADB and SQLite3.
#
# Does not yet *create* the accounts, so you still need to click through the
# accounts wizard and enter dummy information.#
#
# K-9 account settings live in an SQLite3 database and have a unique identifier
# for each account. Records in the database look like:
#
# primkey                                           | value
# ba46a4cf-bde0-4d5d-aa6d-9da12a8367df.transportUri | xxx
#
# The transportUri and storeUri represent the server settings and are base64
# encoded, presumably to obfuscate the password.

PREFERENCES_DATABASE='/data/data/com.fsck.k9/databases/preferences_storage'
ACCOUNT_UUID_PERSONAL='ba46a4cf-bde0-4d5d-aa6d-9da12a8367df'
ACCOUNT_UUID_WORK='83a40e3e-a4ef-4351-a9f2-cd2fb8510654'

# These weird looking URLs came directly from the database (base64 decoded), eg.
# sqlite3 preferences_storage "select * from preferences_storage"
PERSONAL_STOREURI='imap+ssl+://PLAIN:ben%2540stumbles.id.au:password1@imap.fastmail.com:993/1%7C'
PERSONAL_TRANSPORTURI='smtp+ssl+://ben%2540stumbles.id.au:password1:PLAIN@smtp.fastmail.com:465'
WORK_STOREURI='imap+ssl+://PLAIN:ben%2540sturm.com.au:password2@imap.fastmail.com:993/1%7C'
WORK_TRANSPORTURI='smtp+ssl+://ben%2540sturm.com.au:password2:PLAIN@smtp.fastmail.com:465'
WORK_SIGNATURE=$(<~/dotfiles/.signature)

read -r -d '' QUERY <<EOF
    UPDATE preferences_storage SET VALUE = 'Ben Sturmfels' WHERE primkey = '${ACCOUNT_UUID_PERSONAL}.name.0';
    UPDATE preferences_storage SET VALUE = 'Personal' WHERE primkey = '${ACCOUNT_UUID_PERSONAL}.description';
    UPDATE preferences_storage SET VALUE = '$(echo -n $PERSONAL_STOREURI | base64)' WHERE primkey = '${ACCOUNT_UUID_PERSONAL}.storeUri';
    UPDATE preferences_storage SET VALUE = '$(echo -n $PERSONAL_TRANSPORTURI | base64)' WHERE primkey = '${ACCOUNT_UUID_PERSONAL}.transportUri';
    UPDATE preferences_storage SET VALUE = 'false' WHERE primkey = '${ACCOUNT_UUID_PERSONAL}.notifyNewMail';
    UPDATE preferences_storage SET VALUE = '' WHERE primkey = '${ACCOUNT_UUID_PERSONAL}.signature.0';
    UPDATE preferences_storage SET VALUE = 'Archive' WHERE primkey = '${ACCOUNT_UUID_PERSONAL}.sentFolderName';
    UPDATE preferences_storage SET VALUE = 'Junk Mail' WHERE primkey = '${ACCOUNT_UUID_PERSONAL}.spamFolderName';


    UPDATE preferences_storage SET VALUE = 'Ben Sturmfels' WHERE primkey = '${ACCOUNT_UUID_WORK}.name.0';
    UPDATE preferences_storage SET VALUE = 'Work' WHERE primkey = '${ACCOUNT_UUID_WORK}.description';
    UPDATE preferences_storage SET VALUE = '$(echo -n $WORK_STOREURI | base64)' WHERE primkey = '${ACCOUNT_UUID_WORK}.storeUri';
    UPDATE preferences_storage SET VALUE = '$(echo -n $WORK_TRANSPORTURI | base64)' WHERE primkey = '${ACCOUNT_UUID_WORK}.transportUri';
    UPDATE preferences_storage SET VALUE = 'false' WHERE primkey = '${ACCOUNT_UUID_WORK}.notifyNewMail';
    UPDATE preferences_storage SET VALUE = '--
${WORK_SIGNATURE}' WHERE primkey = '${ACCOUNT_UUID_WORK}.signature.0';
    UPDATE preferences_storage SET VALUE = 'Archive' WHERE primkey = '${ACCOUNT_UUID_WORK}.sentFolderName';
EOF

# As my config grew, ADB started telling me "error: service name too long", so I
# had to write the configuration to a shell script, transfer it to the phone and
# run it.
TEMPFILE=$(tempfile)
echo "sqlite3 $PREFERENCES_DATABASE \"$QUERY\"" > $TEMPFILE
adb push $TEMPFILE /data/local/tmp
adb shell sh /data/local/tmp/$(basename ${TEMPFILE})

The idea is that you'd plug in your Android/Replicant phone by USB with ADB debugging enabled and run the above shell script. I gather that you need to kill K-9 mail for the settings to take effect (long hold on primary button and close K-9).

Alright, now I'll freely admit that this took quite a bit more time than it would of to just complete the K-9 settings wizard, but it's the principle that counts! The great thing about Free Software is that I can have confidence that I'll be using K-9 for many years to come, so this investment of time is worthwhile.

I haven't tried creating accounts from scratch. It could be just generating a UUID identifier and setting [UUID].accountNumber to say 2 for a third account. Or there might be some place you need to register the UUID. That's work for another day.

Disabling Django's password strength checking for development

2017-03-05T00:00:00+11:00

Django 1.9 comes with a feature that enforces strong passwords. It's an excellent security feature, but password complexity is not what you want in development. Here's how to turn the feature off to allow simple passwords.

It goes like this. You've created a new database, run the migrations and are about to create a superuser account:

$ python manage.py createsuperuser
Username (leave blank to use 'user'):
Email address: user@example.com
Password:
Password (again):
This password is too short. It must contain at least 8 characters.
Password:
...

Gahhh! All I want to be able to do is use the same one-letter password for all my development environments. The story is similarly frustrating when creating new user accounts via the Django Admin. You're probably creating a test account for a colleague. You want something short and simple.

Django Administration password validation

The solution is to set the following in your development settings. If you're following best practises, will be something like settings/local.py or settings/dev.py:

AUTH_PASSWORD_VALIDATORS = []

Just promise me you won't do this in production!

Python, FTPS and mis-configured servers

2016-12-09T00:00:00+11:00

Today's project involves automatically uploading electrical metering data to an FTPS server (explicit FTP over TLS, otherwise knowns as ESFTP). Shouldn't be a problem, since Python supports FTPS out of the box. Only it doesn't work. Here's the code:

import ftplib
ftp = ftplib.FTP_TLS('host', 'user', 'password')
ftp.set_debuglevel(1)
ftp.cwd('directory')
with open('filename', 'rb') as f:
    ftp.storbinary('STOR filename', f)
ftp.quit()

After a successful initial connection, the data transfer connection fails:

*cmd* 'PASV'
*resp* '227 Entering Passive Mode (10,200,0,100,255,150).'
*cmd* 'CWD collect/CSV'
*resp* '250 CWD command successful'
*cmd* 'TYPE I'
*resp* '200 Type set to I'
*cmd* 'PASV'
*resp* '227 Entering Passive Mode (10,200,0,100,255,123).'

Traceback (most recent call last):
  File "metering/__main__.py", line 143, in main
    ftp.storbinary('STOR {}'.format(filename), stream)
  File ".../lib/python3.5/ftplib.py", line 503, in storbinary
    with self.transfercmd(cmd, rest) as conn:
  File ".../lib/python3.5/ftplib.py", line 398, in transfercmd
    return self.ntransfercmd(cmd, rest)[0]
  File ".../lib/python3.5/ftplib.py", line 793, in ntransfercmd
    conn, size = FTP.ntransfercmd(self, cmd, rest)
  File ".../lib/python3.5/ftplib.py", line 360, in ntransfercmd
    source_address=self.source_address)
  File ".../lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File ".../lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

The most confusing aspect was that the transfer worked perfectly well via FileZilla or Gnome "Connect to Server".

I eventually noticed a message in the FileZilla logs, Server sent passive reply with unroutable address. Using server address instead. It turns out that the FTPS server was mis-configured and was replying to the PASV command with an internal IP address that was not accessible from the public internet. It seems that this is a common enough configuration issue that some FTP clients detect the problem and use the existing server address instead. Python's FTP client doesn't do this though.

The solution was to sub-class the FTP_TLS class and force it to ignore the response:

class FTP_TLS_IgnoreHost(ftplib.FTP_TLS):
    def makepasv(self):
        _, port = super().makepasv()
        return self.host, port

ftp = FTP_TLS_IgnoreHost('host', 'user', 'password')

The makepasv method parses the remote server's response to PASV and returns a host and a port. We're extending this method to throw away the returned host and use the one we already have from the original connection. The underscore is just a convention to indicate that we don't care about the first item in the tuple, the host (it's special in some languages like Prolog, but not in Python). Note of course that this approach doesn't attempt to detect unroutable addresses like FileZilla, it just assumes.

There are no doubt third-party packages that provide this functionality and more, but for such a small extension, it wasn't worth the hassle.

Join us for a swim at Brown Hill

2016-12-03T00:00:00+11:00

The water may be a little chilly, but to coincide with today's opening of the Brown Hill outdoor pool for the summer, we're launching the first page of the Brown Hill Community Hub website — a page about the pool.

We've been getting more involved the local community over the past 6 months or so, following the formation of the Brown Hill Partnership. This is a City of Ballarat initiative to involve a wide range of community groups in choosing projects to fund. So far we've seen the publication of the Brown Hill Community Newsletter, the inaugural Brown Hill Community Festival and today, the launch of the Brown Hill Community Hub. Over the next few months we'll be extending the website to include information about local businesses, sporting clubs and community grounds as well as info about the Hall, the Progress Association, the History Project and Fire Aware.

Phwew, that's certainly a lot of "community". But with the weather warming up and the festive season just around the corner, it's an exciting time of the year to be out and about. Don't forget your togs!

Angular eats my default form values

2016-07-26T00:00:00+10:00

Today, AngularJS isn't doing what I expect. I've auto-generating a really long form in Django, and adding ng-model attributes so I can summarise the form. Unfortunately, Angular is overwriting my defaults. Here's a shortened example:

<html>
    <head>
    </head>
    <body>
        <form ng-app="testApp" ng-controller="TestCtrl">
            <select name="colour" ng-model="colour">
                <option value="red">Red</option>
                <option value="green" selected="selected">Green</option>
                <option value="blue">Blue</option>
            </select>
            <p>You favourite colour is <span style="color: {{ colour }}">{{ colour }}</span>.</p>
        </form>
        <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.7/angular.min.js"></script>
        <script>
         angular.module('testApp', [])
                .controller('TestCtrl', function($scope) {
                    $scope.$watch('colour', function(newValue) {
                        console.log('Colour is: ' + newValue );
                    });
                });
        </script>
    </body>
</html>

It looks something like this:

After marvelling at how little code you need to do something useful, I note that although the HTML form has Green "selected", the select box is initially has no value selected. Why is that? The reason is that once Angular has loaded, it sets the <select> field to its own initial value of undefined, overwriting the HTML default value.

Enter: Angular Auto Value

Thankfully, John Wright and Glauco Custódio have both come up with libraries to make Angular do what we expect. These are Angular Auto Value and Angular Initial Value respectively. Here's our code after modification to use Angular Auto Value (extra <script> element and 'auto-value' argument to angular.module):

<html>
    <head>
    </head>
    <body>
        <form ng-app="testApp" ng-controller="TestCtrl">
            <select name="colour" ng-model="colour">
                <option value="red">Red</option>
                <option value="green" selected="selected">Green</option>
                <option value="blue">Blue</option>
            </select>
            <p>You favourite colour is <span style="color: {{ colour }}">{{ colour }}</span>.</p>
        </form>
        <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.7/angular.min.js"></script>
        <!-- Extra script -->
        <script src="https://cdn.jsdelivr.net/angular.auto-value/latest/angular-auto-value.min.js"></script>
        <script>
         angular.module('testApp', ['auto-value']) // Extra 'auto-value' dependency
                .controller('TestCtrl', function($scope) {
                    $scope.$watch('colour', function(newValue) {
                        console.log('Colour is: ' + newValue );
                    });
                });
        </script>
    </body>
</html>

I often reach for a pinch of Angular as a "jQuery substitute". A touch of auto-completion or form validation adds a lot of value to a server-side web app. Like jQuery, there's no assumption that your whole system is written in JavaScript. Unlike jQuery, it remains maintainable as you add more and more features.

Bruising first experience with ClojureScript

2016-05-20T00:00:00+10:00

"Bruised" is a little how I feel about my first experience porting a program to ClojureScript — configuring production builds has a few traps for newcomers. That said, I am very pleased with the results and excited to do more working with ClojureScript.

My Dad is a soil conservation officer with DELWP. Many years ago I wrote a small Java applet for use in his workshops. It help farmers estimate how much water can flow down an earthen channel without causing soil erosion. Here's a screenshot of the original Java version:

Channel flow calculator (Java version)

Despite its simplicity, the channel flow calculator has been a useful and popular tool. The only drawback has been the inconsistent support for Java applets in web browsers; particularly in government departments. To solve this, we recently decided to rewrite it in HTML and JavaScript. It's also a chance to indulge my interest in ClojureScript!

Why ClojureScript?

Well, less JavaScript for a start. Rather than trying to incrementally fix JavaScript's lack of modules, non-existent standard library, unexpected type coercion rules and inconsistent browser support, ClojureScript starts afresh with solid language fundamentals and good design choices based the Clojure Lisp dialect.

ClojureScript compiles down to lowest-common-denominator JavaScript, meaning that my code will run consistently on any modern browser. I can confidently use the full breadth of the elegant language features, powerful data-types and the convenient standard library. I get real module support. Last but not least, thanks to the optimising compiler, I can fearlessly pull in large libraries, knowing that my visitors will only download the code they need.

ClojureScript trap 1: Unwanted `:main`

The basic translation from Java to ClojureScript was straightforward; even the Java and JavaScript canvas-drawing APIs are nearly identical. After some messing around, I even managed to get Lein, Clojure's de-facto build tool, auto-rebuilding and live-updating my web browser (via plugins lein-cljsbuild and lein-figwheel). The trouble came when I needed to produce the optimised production version for publication.

ClosureScript utilises the Google Closure compiler to remove unused JavaScript code, turning around 1MB into 90kB (20kB Gzipped). All this by simply setting :optimizations :advanced in the build settings.

The first trap was that the :main setting used in development builds can't be left in place when you switch on :optimizations :advanced. Not that Clojure gives up this information willingly. Here's my faulty build script:

;; build.clj
(require 'cljs.build.api)

(cljs.build.api/build "src"
  {:main 'channel_flow.core
   :output-to "out/main.js"
   :optimizations :advanced})

(System/exit 0)

And here's the output:

$ java -cp cljs-1.8.51 clojure.main build.clj
Exception in thread "main" java.lang.AssertionError: Assert failed: No file for namespace channel_flow.core exists

Not so helpful. That error kept me busy for nearly two hours, including diving into the ClojureScript source code to figure out what was going on. After removing the :main setting, the error disappears. It's worth noting that if you're building through lein-cljsbuild, it does the right thing and ignores the unnecessary :main setting.

ClojureScript trap 2: Symbol renaming

The second trap was that after switching from :optimizations :simple to :optimizations :advanced, I was seeing JavaScript errors like document.ga is undefined. It's a bizarre feeling when your code was working only moments ago. Optimisation isn't meant to change the behaviour of code, right?

Turns out it does, if you're not abiding by certain rules. I was using the following HTML form:

<form name="mannings">
<label>Top width: <input type="number" name="w1" /></label>
…

I was referencing this in ClojureScript to trigger the recalculation when inputs were changed:

;; Don't do this
(.addEventListener document.mannings "input" redraw)

The Google Closure compiler was helpfully renaming document.mannings to document.ga to make my code shorter; thereby breaking the reference to the HTML form:

// "Optimised" (broken) JavaScript
document.ga.addEventListener("input",Oe);

All this makes sense in hindsight; I just didn't expect that enabling compilation optimisation do anything other than slow down the build and produce a smaller output file.

The solution is to explicitly advise the Google Closure compiler that document.mannings is something external to this file and not be renamed. Create an externs.js file that references the variables:

// externs.js
document.mannings = {
    w1: null,
    w2: null,
    cd: null,
    s: null,
    m: null,
    fd: null,
    v: null,
    q: null
};

Declare the externs to the compiler in your build script:

:optimizations :advanced
:externs ["externs.js"]

The results

Channel flow calculator (HTML/JavaScript version)

After clearing these initial hurdles, the updated channel flow calculator is now working well (picture below). A few weeks on, now that my frustration has faded, I'm actually really looking forward to my next ClojureScript project.

Nginx dynamic image resizing with caching

2016-04-21T00:00:00+10:00

At Sturm we build web apps that perform beautifully on both tiny mobile devices and huge desktop monitors. To achieve these, we use Nginx to resize images on-the-fly. This approach was used in our recent Course Finder project with Federation University.

The Nginx web server comes with a handy image filter module, but most tutorials don't configure it with performance in mind. Performance means caching.

Let's start with the basics:

server {
    # Basic image resizing server, no caching.
    server_name example.com;

    location ~ "^/media/(?<width>\d+)/(?<image>.+)$" {
        alias /var/www/images/$image;
        image_filter resize $width -;
        image_filter_jpeg_quality 75;
        image_filter_buffer 8M;
    }
}

Don't forget to reload your Nginx configuration:

$ sudo service nginx force-reload

This server block will respond to a request like http://example.com/media/768/mountains.jpg with the file /var/www/images/mountains.jpg. It will resize the image down to 768 pixels wide if necessary and compress it using JPEG quality 75. Requests where the original image is 8MB will be rejected.

Unfortunately our simple example doesn't perform so well, since it does a resize for every single request. Let's test it using Apache Bench (from apache2-utils on a Debian-based GNU/Linux). We'll make 100 requests using 5 concurrent connections:

$ ab -n 100 -c 5 http://example.com/media/768/mountains.jpg
...
Requests per second:    4.14 [#/sec] (mean)
...

Oh no, only four requests per second!

The solution is to cache the resized image so that the work is only done once a day. Since Nginx can't cache and resize at the same time, we'll need a trick. That trick is to use two server blocks. The first will be an internal-onlyserver that resize images. The second will be a public server that proxies requests to our internal server and then caches the result:

server {
    # Internal image resizing server.
    server_name localhost;
    listen 8888;

    location ~ "^/media/(?<width>\d+)/(?<image>.+)$" {
        alias /var/www/images/$image;
        image_filter resize $width -;
        image_filter_jpeg_quality 75;
        image_filter_buffer 8M;
    }
}

proxy_cache_path /tmp/nginx-images-cache/ levels=1:2 keys_zone=images:10m inactive=24h max_size=100m;

server {
    # Public-facing cache server.
    server_name example.com;

    # Only serve widths of 768 or 1920 so we can cache effectively.
    location ~ "^/media/(?<width>(768|1920))/(?<image>.+)$" {
        # Proxy to internal image resizing server.
        proxy_pass http://localhost:8888/media/$width/$image;
        proxy_cache images;
        proxy_cache_valid 200 24h;
    }

    location /media {
        # Nginx needs you to manually define DNS resolution when using
        # variables in proxy_pass. Creating this dummy location avoids that.
        # The error is: "no resolver defined to resolve localhost".
        proxy_pass http://localhost:8888/;
    }
}

Now that we're caching the resized images, Apache Bench shows a slow first request, but thereafter serves over 7000 requests per second!

Lots of love for Conservancy at LCA 2016

2016-02-09T00:00:00+11:00

It was wonderful to see an outpouring of support for Conservancy last week at LCA 2016. A number of speakers highlighted Conservancy during their talks, Bradley Kuhn gave a talk about GPL compliance, we held a Supporter lunch (photo below), and both the conference organisers and the new Linux Australia President highlighted Conservancy in the conference closing.

Software Freedom Conservancy is a tiny organisation doing amazing work to support the development of Free Software. They assist community free software projects with their administrative work (financial, legal, etc.) so that the developers can concentrate on making great software. Conservancy's member projects include BusyBox, Git, Inkscape, BusyBox, Mercurial, PyPy, Samba, Twisted and Wine.

Speaking on MediaGoblin at linux.conf.au 2016

2016-02-02T00:00:00+11:00

I'm very excited to be speaking about GNU MediaGoblin this Friday 5 February at linux.conf.au in Geelong, Australia. MediaGoblin is a media publishing tool for artists.

I began using MediaGoblin nearly four years ago as a way to share photos of our new baby with our family and close friends — a popular use case from all accounts! The two compelling features at the time were the ability to upload both images, audio and video to the same service and the automatic resizing, transcoding and compression. Being a technical person, I theoretically had all the tools and knowledge to convert the media and publish the HTML, but in practise it simply wouldn't have happened. With MediaGoblin, the publishing process is a joyful one.

A lot has changed in MediaGoblin since then. GStreamer 1.0 has brought more robust transcoding with support for more formats. The client-to-server API allows publishing media from a pump.api client. There's great new features for admin users including reprocessing of failed transcoding, command-line uploads, bulk uploads, reporting/moderation, PDF/document media type, 3D model media type and much more. These advances have allowed me to also use MediaGoblin to publish audio, video and slides from my past presentations on Emacs, Python, free software and software patents.

Thanks to Jessica Tallon's work, funded entirely by your donations, the project is also looking forward to announcing federation in the near future. This will bring social features such as sharing, commenting and notifications across different distinct MediaGoblin servers. I can't wait to talk more about these exciting developments on Friday. I'm very grateful to Chris Webber and Jessica for walking me through some of the details of their current work.

Hope to see you at 10:40am in D4.303 Costa Theatre!

Grunt build hangs on ngtemplates:dist

2015-10-28T00:00:00+11:00

I just ran into a strange situation where grunt build on a Yeoman Angular project was hanging at the following step:

$ grunt build
⋮
Running "ngtemplates:dist" (ngtemplates) task

The cause turned out to be an extra double quote symbol in one of the HTML view templates:

<form method="get" action="#" onsubmit="window.location='#/results/' + query.value; return false;"">

A git bisect helped reveal the problematic revision, but even so it took a lot of trial and error to find the cause. The challenge was that Grunt gave no clues as to what might be causing it to stall. It also didn't crash, rather sat there in an infinite loop merrily consuming the CPU.

Interesting, the problem doesn't occur for just any old extra quote, so I'd guess that means that Angular is trying to parse this snippet of JavaScript and getting stuck. And yes, I agree that this is a very un-Angular way to submit a form. :)

Auto-starting Emacs smerge-mode for Git

2015-09-28T00:00:00+10:00

I love that Emacs notices merge conflicts in Bazaar versioned files, highlighting the differences in green and pink. It's a small thing, but I really miss this automation for when working with Git — I have to manually type M-x smerge-mode.

The feature works by looking for conflict files and markers whenever you open a Bazaar versioned file:

;;; Copied from Emacs 24.3.1: vc/vc-bzr.el.gz:472
(defun vc-bzr-find-file-hook ()
  (when (and buffer-file-name
             ;; FIXME: We should check that "bzr status" says "conflict".
             (file-exists-p (concat buffer-file-name ".BASE"))
             (file-exists-p (concat buffer-file-name ".OTHER"))
             (file-exists-p (concat buffer-file-name ".THIS"))
             ;; If "bzr status" says there's a conflict but there are no
             ;; conflict markers, it's not clear what we should do.
             (save-excursion
               (goto-char (point-min))
               (re-search-forward "^<<<<<<< " nil t)))
    ;; TODO: the merge algorithm used in `bzr merge' is nicely configurable,
    ;; but the one in `bzr pull' isn't, so it would be good to provide an
    ;; elisp function to remerge from the .BASE/OTHER/THIS files.
    (smerge-start-session)
    (add-hook 'after-save-hook 'vc-bzr-resolve-when-done nil t)
    (message "There are unresolved conflicts in this file")))

The above looks a little complicated, but the important part is the line (smerge-start-session). This starts smerge-mode to highlight merge conflicts.

Let's do something similar with Git by defining a function vc-git-find-file-hook:

(defun vc-git-find-file-hook ()
  (when (save-excursion
      (goto-char (point-min))
      (re-search-forward "^<<<<<<< " nil t))
    (smerge-start-session)))

Emacs vc-mode has a standard interface for version control systems, so simply defining this function is all we need to do. Emacs will look for and run the function if it exists (it doesn't by default).

Similar to the Bazaar version, this function looks for conflict marker text in the file, and starts smerge-mode if it finds any. The Git version is shorter because Git deals with conflicts differently, and also because I haven't hit any edge cases… yet.

John Oliver on Software Patents

2015-06-05T00:00:00+10:00

A few years back, only the legal and computing industry understood the dangers of software patents. Today I was amazed watch John Oliver's detailed and hilarious treatment of the subject on Last Week Tonight (YouTube). The narrator summed up our concerns beautifully:

In the last decade, the number of software patents has skyrocketed. The issue here, say experts, is that while patents for machines tend to be fairly specific, software patents can be so broad and vague that they may give someone the ability to later claim ownership for inventions they never dreamed of at the time.

I suppose it's tempting for reporters to dwell on patent trolls, but many of the most harmful patents come from large well-known companies and consortia. These patents make important fields such as video encoding almost unbearable to work in and leave the public all the poorer for lack of compatibly and choice. That aside, it's always satisfying to see Uniloc being berated for it's misuse of the patent system.

Rock Your Emacs video from LibrePlanet

2015-04-18T00:00:00+10:00

I've recently returned from LibrePlanet 2015 where I presented a workshop called Rock Your Emacs. The workshop is designed to help Emacs users over the initial barriers to writing their own customisations with Lisp code. While we're waiting for official videos to be published, here's the video, slides and exercises from my workshop.

Rock Your Emacs, LibrePlanet 2015

Slides (PDF)
Exercises (Emacs Lisp)

Do you love Emacs, but have never understood the strange code with lots of brackets? You're missing out on one of the great joys of Emacs — customizing it to work exactly the way you want. It turns out that Emacs is a full Lisp code interpreter, so once you know a little Emacs Lisp, almost anything is possible.

After attending this tutorial, you will know how to:

read basic Emacs Lisp code
modify Emacs Lisp code as well as writing your own
make persistent customizations to your Emacs
bind your favourite commands to keys
answer your own questions with the amazing help system
use built-in Emacs Lisp development tools like the debugger

Programming experience is helpful but not required. Bring your Emacs and type along.

Ben Sturmfels is a software engineer and free software activist from Ballarat, Australia. He organises Free Software Melbourne, leads the End Software Patents Australia campaign, rides bikes, flies kites and grows vegetables.

Controlling fan speed on Thinkpad X60

2014-04-09T00:00:00+10:00

There’s just one minor annoyance with my new LibreBoot Thinkpad X60 from GLUGLUG. Working it hard causes it to overheat and shut down. Thankfully, the fan turns out to have much more capability than the defaults allow; all but solving this problem.

In Trisquel GNU/Linux 6.0, the X60′s automatic fan controller runs at one of eight preset speeds (0-7), depending on the temperature. About 10 mins with both CPUs running at 100% will cause it to overheat (eg. two copies of python -c"while True: pass").

As described on ThinkWiki, you can manually override the fan control by adding the line options thinkpad_acpi fan_control=1 to /etc/modprobe.d/thinkpad_acpi.conf. Then after rebooting and as root, you can set the fan to run at maximum speed (~3700 RPM) with:

echo level 7 > /proc/acpi/ibm/fan

Only that’s not the true maximum speed. You can actually disable the rotation speed regulation and just supply full voltage to the fan (~4900 RPM):

echo level full-speed > /proc/acpi/ibm/fan

This setting is a bit noisier, but allows both CPUs to run at 100% with no signs of overheating. A good result.

Next I need to figure out how figure out how to enable this automatically when the machine gets hot. I’ve tried thinkfan, but it seems to only allow the default 0-7 speeds used by the default fan control.

Raising concerns about ACTA and TPP

2012-07-14T00:00:00+10:00

The recent resounding defeat of ACTA in the European Parliament has given me a lot of hope. Many citizens took the time to contact members of parliament and this seems to have made all the difference. I hope that Australian can also reject this harmful agreement. That won’t happen without some effort though.

This morning I had the opportunity to meet and speak with Australian Government MP Catherine King. I raised concerns about the ACTA and TPP agreements. She acknowledged my concerns and agreed to contact Minister Stephen Conroy about these issues. Here are the points I provided:

ACTA

Please defend our freedom by voting down this harmful agreement.

Negotiated in secret for over two years
Uses the guise of reducing commercial international-level copyright and trademark infringement (an acceptable aim) to introduce provisions that restrict the freedom of citizens
Serves the interest of large incumbent music and movie companies, preserving their outdated business models
Punishes Internet users with disconnection if accused of sharing
Prohibits software that circumvents digital-restrictions on text/audio/video (ie. “you must only use our locked-down e-book reader software to read this book”)
Encourages a culture of surveillance and suspicion in which freedom and creativity are seen as dangerous.
Following a huge public backlash, the European Parliament recently voted ACTA down, 478 votes to 39.

TPP

Please speak out against the nature and scope of this agreement.

Completely non-transparent process – NGOs could make submissions at stakeholder forums knowing almost nothing about the scope of the agreement
Leaked draft indicates that United States is using the TPP to spread it’s draconian copyright and patent laws around the world.
Expected to increase the duration and scope of copyright and patents and adding harsher infringement penalties. These laws are already a long way in the favour of publishing and media companies to the detriment of authors, artists and consumers.
Expected to explicitly include areas of computation and information processing in patentable subject matter (aka. software patents). These patents don’t work as a way to encourage innovation, actually stifling the Australian software industry (see our 2010 petition which received 1000 signatures).

Update 21 July 2012: Looks like I wasn’t paying attention. Australia signed up to ACTA in October 2011. The TPP is definitely still being negotiated though.

Choice, please speak out against DRM

2010-01-08T00:00:00+11:00

*The following is a letter written to Choice, the leading Australian consumer advocacy organisation.

I urge Choice to speak out against products like the Amazon Kindle electronic book reader. They use Digital Restrictions Management (DRM) technology to restrict your basic freedoms.

The Kindle uses DRM technology to give Amazon full control over what the customer can and can’t do with the product. It allows many freedoms of traditional books to be stripped off, such as sharing the book with friend or giving the book away when you’re finished with it. DRM also allows monitoring and censoring of what you read. For example, Amazon recently remotely deleted copies of George Orwell’s “Animal Farm” and “1984″. The customers involved were given no notice or choice. The breathtakingly irony is that “1984″ describes a fictional society where citizens are under constant surveillance and control.

Understandably, devices like the Kindle are widely considered to be unethical. DRM is also being added to many products like phones, music players, digital music, films and books. The result is we consumers lose. Please help defend our rights by considering these issues in future reviews and campaigns.

More information about DRM is available at the Defective By Design website.