Why I mostly hire generalists

I’ve had the pleasure of getting to build the in-house web development team at Rogers Media for a few years now up from just myself and one other developer to a staff of fifteen. I’m of the school of thought that software development teams are best kept small: It requires a smaller web of communication, individual team members maintain a high level of impact on the product they’re building (avoiding a loss of motivation due to the Ringelmann effect), and team stability (that is, people constantly joining or leaving the team) is usually greatly improved.

But there’s a potential downside that has to be addressed: Software is complex, and the requirements of web development often extremely so. How does a small team come through with everything that’s needed to make a project work? My answer has been to build the core of the team around generalists – people who tend to be very good in one area of development, but also knowledgeable and capable in most others as well.

Here’s an excerpt from the Pragmatic Programmer series titled Be A Generalist that speaks to why generalists are the answer to meeting a modern business product’s software needs:

For at least a couple of decades, desperate managers and business owners have been pretending that software development is a manufacturing process at heart. Requirements specifications are created, and architects turn these specifications into a high-level technical vision. Designers fill out the architecture with detailed design documentation, which is handed to robot-like coders, who hold pulp-fiction novels in one hand while sleepily typing in the design’s implementation with the other. Finally, Inspector 12 receives the completed code, which doesn’t receive her stamp of approval unless it meets the original specifications.

It’s no surprise that managers want software development to be like manufacturing. Managers understand how to make manufacturing work. We have decades of experience in how to build physical objects efficiently and accurately. So, applying what we’ve learned from manufacturing, we should be able to optimize the software development process into the well-tuned engine that our manufacturing plants have become.

In the so-called software factory, the employees are specialists. They sit at their place in the assembly line, fastening Java components together or rounding the rough edges of a Visual Basic application on their software lathes. Inspector 12 is a tester by trade. Software components move down the line, and she tests and stamps them in the same way each day. J2EE designers design J2EE applications. C++ coders code in C++. The world is very clean and compartmentalized.

Unfortunately, the manufacturing analogy doesn’t work. Software is at least as malleable as software requirements. Things change in business, and businesspeople know that software is soft and can be changed to meet those requirements. This means architecture, designs, code, and tests must all be created and revised in a fashion more agile than the leanest manufacturing processes can provide.

In this kind of rapidly changing environment, the flexible will survive. When the pressure is on, a smart businessperson will turn to a software professional can solve the problem at hand. So, how do you become that person whose name comes up when they’re looking for a superhero to save the day? The key is to be able to solve the problems that may arise.

What are those problems? That’s right: you don’t know. Neither do I. What I do know is that those problems are as diverse as deployment issues, critical design flaws that need to be solved and quickly reimplemented, heterogenous system integration, and rapid, ad hoc report generation. Faced with a problem set as diverse as this, poor Inspector 12 would be passed over pretty quickly.

The label jack-of-all-trades—master of none is normally meant to be derogatory, implying that the labelee lacks the focus to really dive into a subject and master it. But, when your online shopping application is on the fritz, and you’re losing orders by the hundreds as each hour passes, it’s the jackof-all-trades who not only knows how the application’s code works but can also do low-level UNIX debugging of your web server processes, analyze your RDBM’s configuration for potential performance bottlenecks, and check your network’s router configuration for hard-to-find problems.

And, more important, after finding the problem, the jack-of-all-trades can quickly make architecture and design decisions, implement code fixes, and deploy a new fixed system to production. In this scenario, the manufacturing scenario seems quaint at best and critically flawed at worst.

Another way in which the software factory breaks down is in that, although in an assembly line the work keeps coming in a steady flow, software projects are usually very cyclical. Not only is the actual flow of projects cyclical, but the work inside a project is cyclical. A coder sits on the bench while requirements are being specified, architected, and designed, or the coder multitasks across many projects. The problem with multitasking coders is that, despite the software factory’s intentions, when the rubber meets the road, the coders rely a great deal on context and experience to get their jobs done. Requirements, architecture, and design documents can be a great head start, but ultimately if the programmers don’t understand what the system is supposed to do, they won’t be able to create a good implementation of the system.

Of course, I’m not just picking on coders here. The same is true at nearly every spot on the software assembly line. Context matters, and multitasking doesn’t quite work. As a result, we have an inefficient manufacturing system. There have been various attempts to solve this problem of inefficiency without departing from the manufacturing-inspired system, but we have not yet figured out how to optimize our software factories to an acceptable level.

If you are just a coder or a tester or a designer or an architect, you’re going to find yourself sitting idle or doing busywork during the ebbs of your business’s project flow. If you are just a J2EE programmer or a .NET programmer or a UNIX systems programmer, you’re not going to have much to contribute when the focus of a project or a company shifts, even temporarily, out of your focus area. It’s not about where you sit on the perceived value chain of project work (where the architect holds the highest spot of royalty). It’s about how generally useful you make yourself.



The growth of mobile device usage for the Web isn’t at the expense of desktop usage

From ComScore – Is Mobile Bringing About the Death of the PC? Not Exactly, some interesting data:

Here we can see that mobile consumption has clearly exploded, with a 44% gain in mobile app usage and a 38% gain in mobile web usage. This has contributed to mobile growing its share of digital media time spent from 50% to 60% in the past year and owning a clear majority of time spent.

But importantly, desktop usage has held up in the face of increased competition from mobile for consumers’ attention, with a negligible overall decline in total time spent from 477 billion minutes to 466 billion minutes. The net result of is that we’ve seen a 20% overall increase in time spent on digital in the past year, which means a lot more opportunity to reach consumers and monetize content.

As a web developer, this tells me that websites with analytics that tell me that desktop users are predominant aren’t to be forgotten; mobile Web usage is growing, but native app usage is where people’s attention is.



Quotes: Innovation

The more innovative – the more breaking-the-mold – the innovation is, the less likely we are to figure out what it is really going to be used for.

– Robert Friedel

Regular expressions: Greedy regex versus lazy regex

Here’s a typical regex scenario: You’ve got a string in which you need to find/capture the HTML tags from. Let’s say our string is:

This is a <em>first</em> test

Typically, you’d write a regular expression to capture the tag by writing this:

var re = new RegExp("<(.+)>", "");

Unexpectedly, however, the result you’re going to get matched back is:


The reason for this is explained on the Regex Tutorial website:

The first token in the regex is . You should see the problem by now. The dot matches the >, and the engine continues repeating the dot. The dot will match all remaining characters in the string. The dot fails when the engine has reached the void after the end of the string. Only at this point does the regex engine continue with the next token: >.

What we need to do instead is force the dot character to be lazy by adding a question mark after the plus sign (or a star, or numbers in curly braces):

var re = new RegExp("<(.+?)>", "");

This time, we’ll get back:


Reference: Regex Tutorial – Repetition with Star and Plus

Hacking the Budweiser Red Light (Part I): Identifying the network traffic that activates the light

The Budweiser Red Light is one of the best pieces of marketing I’ve ever seen. And while I’m enough of a hockey fan to want to pick up one of these anyways, the real prize is in figuring out how to make the thing go off whenever I choose.

Where to get started? While I’ve seen an attempt to use/modify the Electric Imp card inside the light to accept requests, I thought my approach might be simpler: Sniff the network traffic to and from the light, replicate it to sound the alarm.

The instruments used to sniff the traffic were as follows:

  • A MacBook Pro (to set up Internet Sharing upon for the Red Light to connect to)
  • My Android OS smartphone (to install and pair to the Red Light using the app Budweiser developed for that purpose)
  • WireShark (a network protocol analyzer) installed on the MacBook Pro, which also requires XQuartz


I kept careful notes in case someone else wanted to replicate this experiment; those instructions are:

  1. Enable Internet Sharing on the MacBook Pro, and make sure you set no password or key. (At home where I have an Windows 8.1 PC, I had attempted to set up a Wi-Fi hotspot but the Red Light appeared to have difficulty getting on that network.) I named my new Wi-Fi access point “BudRedLight”.
  2. I had my Android OS smartphone join the Wi-Fi access point of “BudRedLight”, which it did without issue.
  3. In order to sniff its traffic, we’ll need to get the Red Light on the “BudRedLight” access point as well. This means installing and setting up the official Budweiser Red Lights app, and using the interesting flashing light method of sending the Wi-Fi connection details from the phone to the Red Light.
  4. Once that’s done, I left the app open on the phone, specifically staying on the screen that provided the “TEST YOUR LIGHT” button. The plan was to capture what was received by the Red Light once this was pressed.
  5. Now that the Red Light and my phone were both on the “BudRedLight” access point, it was time to boot up WireShark on the MacBook Pro (I had to start XQuartz first so WireShark would run).
  6. Let’s get WireShark listening to the traffic coming in and out of “BudRedLight”: Select the Capture menu option, and then Options. Unclick the “Use promiscuous mode on all interfaces option” (this will cut down on the amount of noise being captured). Instead, double-click on the Wi-Fi listing, and within it check the “Capture packets in promiscuous mode” option and hit OK. (Need a visual guide? Here’s a screenshot.)
  7. Okay – we’re now capturing traffic! Tap the “TEST YOUR LIGHT“; this will make your phone send a HTTPS request up to some remote server, which in turn appears to send a TCP [PSH, ACK] packet to the Red Light. In the screenshot of WireShark below, I pressed the “TEST YOUR LIGHT” twice, resulting in the two [PSH, ACK] packets listed:bud-red-light-tcp-psh-ack-traffic
  8. So what’s in that packet? Only 234 bytes of it contain actual data, so let’s see what that looks like (via WireShark):bud-red-light-packet-inspection

Next Steps

So we’ve got an example of the data used to set off the Red Light, but we don’t really know what’s contained in that data.

Deep packet inspection isn’t really my thing, so at this point I’ve started asking around for possibilities. Here are the early contenders:

  • Use tcpreplay, Ostinato or some other application to “replay” sending of the packet shown above to the Red Light; maybe we don’t really even need to know what’s in the packet and this will set it off.
  • Find and use some other utility (or person!) that can tell us how to further decipher what’s in the data seen above.

If you’d like to help out, you certainly can! I’ve uploaded a zipfile of a PCAP file containing the packet I’ve displayed above. Feel free to try and dicipher or replay sending of that packet on your own! Please leave a comment if you do so; it’d be great to solve this for everyone.

When “compass compile” leads to an ‘Invalid UTF-8 character “\xCA”‘ error

We’ve got some pretty large CSS files to work with on a few projects I work on, and use SCSS plus the Compass tool to make management of the styles a bit easier. Recently, though, Compass has been throwing us this error:

    error scss/sportsnet.scss (Line 969 of scss/_inc-controls.scss: Invalid UTF-8 character "\xCA")

The root cause appears to be non-ASCII characters in a SCSS file. That’s easy enough to root out by hand if the file is a manageable size, but if you need an automated filter instead, try the following:

iconv -t ASCII//IGNORE -f UTF8 < _inc-controls.scss > _inc-controls.ascii

This pipes the offending SCSS file through iconv, and spits out the file _inc-controls.ascii which should contain differing lines where non-ASCII lines are found.

Overwriting a branch in Git with remote results (and bypassing merge conflicts)

If you’ve ever tried to do a git pull to get a branch up to date and have run into conflicts that you just want to bypass and go with the contents of the remote branch, do the following:

# Fetch from the default remote of origin.
git fetch origin
# Check out the branch you want to override the conflicts upon.
git checkout master
# Reset your current branch of master to origin's master.
git reset --hard origin/master

Reference: Stack Overflow: Git pull from remote.. Can I force it to overwrite rather than report conflicts?

Get a list of files added, removed and modified in Git between two branches

It’s sometimes useful to know exactly what files you’ve made changes to in your feature branch as you’re getting ready to merge back into your trunk code. Git’s diff option allows for this to be done quick easily:

git diff --name-status master..newfeature

The output looks like the following:

M       .gitmodules
A       plugins/backplane.php
A       plugins/multiple-post-thumbnails
M       themes/sportsnet/css/scss/_inc-controls.scss
M       themes/sportsnet/functions.php
M       themes/sportsnet/header.php
M       themes/sportsnet/single-sn-article.php
M       themes/sportsnet/single-sn-blog-entry.php
M       themes/sportsnet/single-sn-signing.php
M       themes/sportsnet/single-sn-trade.php
A       themes/sportsnet/zones/articles-comment-form.php
A       themes/sportsnet/zones/global/user-account-links.php

Reference: Stack Overflow – Showing which files have changed between git branches

Where all of my placeholder images come from

Years back, I happened upon fractalsponge.net, a one-man effort in creating extremely high resolution Star Wars renders of the various spacecraft in that fictional universe.

Downloading more than 5,000 rows from Google Analytics at a time

I’ve been increasing the amount of click tracking I do via Google Analytics, but getting the data back out of that system can be a bit of a time when you’re talking large amounts of data.

Enter Download Analytics, which will ask for input of a URL to the report page you’re viewing, and will automatically prompt to e-mail you (when ready) a link that will allow you to download the entire list of rows in your report. Simple, fast, effective.