Early in my academic career at the University of Waterloo, I was fortunate enough to land a co-op placement at The Working Group. Back then, the team was just over a dozen people. We were taking on our first mobile projects, and were starting to outgrow our old office at the Burroughes building – where we still had musical jam sessions with the partners every couple weeks. I learned more and had more fun in that four-month placement than I thought possible.

That was two years ago. In February 2013, I founded a software company that creates music apps that anybody can use. So far, our portfolio of products includes The Wub Machine, an automatic music remixing app, and Forever.fm, an app that creates an infinite DJ mix of the hottest songs on SoundCloud. These two apps have proven popular, and have already reached more than 1,000,000 people across the world. However, their development...

Let’s go back to the good old days of writing web applications in PHP for a paragraph or two. When running PHP under Apache or nginx, every HTTP request resulted in a clean interpreter with completely new state. Developers had to explicitly ask for state to be shared - through the $_SESSION global, by persisting state on disk, or by saving state to some backing data store. This made developing applications amazingly simple. A PHP page was something like a pure function, producing consistent, predictable output based on the state of the underlying data store.

Now, consider this little bit of Python code:

class PatternRemixer(Remixer):
    _samplecache = {}

    def remix(song):
         do some stuff
        for key in song:
            if key not in self._samplecache:
                self._samplecache[key] = self.render_audio()
            self.output(self._samplecache[key])

Any...

Pipelines are an extremely useful (and surprisingly underused) architectural pattern in modern software engineering. The concept of using pipes and filters to control the flow of data through software has been around since the 1970s, when the first Unix shells were created. If you’ve ever used the pipe (“|”) character in a terminal emulator, you’ve made use of the pipe-and-filter idiom. Take the following example:

cat /usr/share/dict/words |      Read in the system's dictionary.
grep purple |                    Find words containing 'purple'
awk '{print length($1), $1}' |   Count the letters in each word
sort -n |                        Sort lines ("${length} ${word}")
tail -n 1 |                      Take the last line of the input
cut -d " " -f 2 |                Take the second part of each line
cowsay -f tux                    Put the resulting word into Tux's mouth

When run with...

The modern trend of providing an API for everything is wonderful. With minimal effort, any developer with an internet connection can programmatically access a wealth of data and powerful functionality. Without APIs, many hackathons wouldn’t exist, and many new developers would languish in frustration instead of participating in the best part of software development - building fun stuff.

However, all of this convenience comes at a cost. Often, that cost is literal, if an API provider decides to charge for access. This is the entire business model of many companies, and there are now even companies that provide API-monetization-as-a-service. This has created a kind of purely digital marketplace, by literally allowing people to buy access to data and functions. (This is a Good Thing™, as it encourages competition and variety in the API market, and reduces time-to-ship for many developers.)

I attend the University of Waterloo, one of Canada’s most widely-known engineering schools. Waterloo is famous for a system they call co-op - a regimen of paid internships of 4-8 months in duration in a real-world work environment. Co-op is mandatory for all engineering students, and upon graduation, results in each student having worked at up to 6 different companies for a total of at least 24 months. Each “work term” can happen during the summer, fall, or winter, and can be within Canada or abroad. (We do often go abroad, primarily to Silicon Valley.) Here’s where my class went for internships this past summer:

Over the past year, a number of Waterloo interns have had the pleasure of interning at Khan Academy, the groundbreaking non-profit dedicated to “accelerate learning for students of all ages.” They’ve made such an impression on Sal Khan, its founder, that he’s gone on to speak...

Late last week, I officially launched forever.fm, an infinite, beatmatched radio stream powered by SoundCloud. This morning, I was happy to discover that it had been featured in Hack A Day - one of my favourite hack-centric blogs. However, such exposure resulted in one small issue:

That’s 25% of my little 512MB Linode’s monthly bandwidth allotment being used up in 6 hours. With Linode (as of this writing) charging $0.10/GB for bandwidth (allotted or through overages), that huge server load could get very expensive, very fast. (At that rate, each listener would cost me roughly $0.25 per day of constant listening. Not viable for a free service!)

So, this afternoon, I was faced with a dilemma. How do I quickly and easily make it cheaper for me to host the site at peak times? A tried and true CDN would be a good solution, but even simple CDNs like Amazon CloudFront would cost more than...

I’m very proud to announce the launch of my latest project - forever.fm, an automatic, infinite online DJ. Forever.fm is a beatmatched stream of the hottest tracks from SoundCloud, mixed together to sound awesome, and continuing forever. (No advertisements, DJ chatter, or breaks!) Check it out!

WARNING: Past this point, you’ll find only gory technical details of how forever.fm was made.

Overview

Forever is powered by a large number of technologies, some of which I stole from my previous music hack, the Wub Machine:

• For this project, I chose to use a 512MB Linode VPS, which has been running spectacularly.
• The entire site runs on Python and uses Facebook’s Tornado evented server.
• To stream track metadata, waveforms, and other live updates, I’ve used Socket.IO and tornadio2, its Tornado wrapper.
• the SoundCloud API provides the songs, metadata and audio streams that you hear.
• the...

A couple months ago, I posted a blog post explaining my use for low-quality
smartphone photos. It involved a smart image cropping algorithm written by Michael Macias, using ImageMagick and written in Ruby. I’ve actually used the algorithm quite a bit in preparing new photos for my homepage - although there’s one major problem - it’s amazingly slow. Take a look at the kind of processing it does:

On large JPEGs from my own photo library, like the one above, this Ruby script takes roughly 2 seconds to perform a smart 124px square crop on the most interesting part of the image:

Matched 9 images.
Originals/2012/NYC/IMG_7054.JPG => ./18.jpg in 1801.717ms
Originals/2012/NYC/IMG_7055.JPG => ./19.jpg in 1856.692ms
Originals/2012/NYC/IMG_7052.JPG => ./20.jpg in 1787.717ms
Originals/2012/NYC/IMG_7059.JPG => ./21.jpg in 1727.487ms
Originals/2012/NYC/IMG_7057.JPG => ./22.jpg in 1716.977ms

Every so often, I find myself in a camera store, gawking at beautiful, expensive cameras and lenses. DSLRs have dropped in price, and mirrorless interchangeable lens cameras (also known as micro four thirds) now fill the gap between cheap point-and-shoot and semi-pro. However, every single time I go to make such a purchase, I stop myself.

It’s not that I don’t want a good camera, it’s that I already have a camera good enough. Most of us have one on us at all times.

It’s my smartphone, and it can capture images like this:

This image isn’t pristine. It’s vibrant, although it could be moreso. It’s lacking in detail, a bit noisy, and somewhat compressed. (Compressing it for web didn’t help with the presentation, either.) However, none of that is important. It’s a beautiful reminder of that moment - a great lunch with a great person, on a bright and sunny day, in a crowded restaurant...

One of my web apps, The Wub Machine, is very computationally expensive. Audio decoding, processing, encoding, and streaming, all in Python. Naturally, my first instinct was to turn to the multiprocessing module to spread the CPU-bound work across multiple processes, thus avoiding Python’s global interpreter lock.

In theory, it’s simple enough, but I did run into a few very nasty problems
when dealing with multiprocessing in Python:

• The multiprocessing module, at least on *nixes, forks the current process and communicates with the child with a pipe. This works wonderfully if the data you’re transferring can be easily pickled, and if the child process doesn’t need to modify any global state in the parent. Unfortunately, certain useful constructs in Python can’t be pickled, including functions and lambdas (or pretty much anything callable).

  In my app, I had a peculiar use case - I...