Hi, I'm Frederic.

Web Developer & Entrepreneur
blogging about life and programming in Node.js+React.

GithubTwitterEmailKeybase

On Motivation

In this post we’ll discuss the 5 factors that increase intrinsic motivation and how by gradually losing those while working on a too big project, my motivation eventually grinded to halt so did my productivity and happiness. (TD;DR at the bottom)

Introduction

Motivation is tricky. For some people it’s mostly hovers a steady level most of days. But, it seems that for more entrepreneurial, driven, type A, win or nothing persons it looks a lot more like an epic roller coaster, it’s swayed up or down a lot more.

I’ve been having a really hard time dealing with a large rewrite project, yes, the thing most experienced programmer recommends not doing, ever. The plan was, take 60% of what was written in the past year by 4-8 engineers, rewrite it from scratch, bring in new features other teammates add the old codebase and finally ship it effectively deprecating the old code.

Context

That endeavour may seem like really too big and impossible to achieve in mere months (by myself alone with a bit of help) but I actually believed it was doable mostly because:

  • I had now worked in most parts of the codebase and knew pretty well how it clicked
  • I had already tried the refactor part by part approach and failed as they always became too big projects, it’s really difficult to replace an jet’s engine while the plane is flying
  • From those previous attempts I had code that could be reused
  • I planned on getting some time away from the office to concentrate on getting as much as possible done in the first 2 weeks

Now that we are done and shipping, 3 months later (and I got some help from coworkers down the road), I can see how 1-2 months could have been enough, if I worked at 200% productivity the whole time, like in the first 2 weeks. See, programmers never where good at estimating, we are too biased and there are too much variables involved.

Intrinsic Motivation

Intrinsic Motivation, compared to extrinsic motivation (earn external rewards / avoid punishments), is a lot more powerful in how motivating it is. We’ll discuss the 5 factors that increase it and how by gradually losing those, my motivation eventually grinded to halt so did my productivity and happiness.

In Making Learning Fun: A Taxonomy of Intrinsic Motivations for Learning Malone and Lepper (1987) identified the following 5 factors as increasing intrinsic motivation:

  • Challenge: Goals with a personal meaning, achieving the goal is possible but not certain
  • Curiosity: In this case, cognitive curiosity, the opportunity to learn
  • Control: Control of yourself, your environment and your task at hand
  • Cooperation and Competition: Helping others / Comparing your performance to others
  • Recognition: Having your accomplishments recognized by others

Cooperation and Competition

I started the project with a coworker, he was helping me out for the first 2 weeks, in retrospect, things went downhill after that point.

Having somebody to work with is such a game changer compared to working alone as it allows you to split the mental load in two, you are not bearing the stress all alone, you can talk about the project you are on, support each other. That sense that, not only you have help, but also that you have somebody you are competing against productivity wise is a huge motivator. People grossly underestimate the impact of peer pressure!

Once that lovely person I could compete with and share the hurdles with was gone it showed immediately in how much I accomplishing daily.

Recognition

See, the thing with programming is that not all work is equal in how much of it is visible. New features can be written in a very ugly way (and run slowly) but still work in a very short amount of time. In the first two weeks we got 90% of the pages / functionality rewritten, people where impressed, it was out of this world progress, compressing 6 man-year of work in potentially 2 man-months? Incredible! Except it wasn’t.

Software has this other category of work that most often takes up 80% of the time to produce: test, performance, best practices and edge cases. We skipped on writing tests as we where still shaping up how everything was to be structured and we wanted to show progress quickly. In and of itself this is OK, except I wasn’t ready to deal with the 2 months of that kind of work that came right after.

The lesson here is chunk your work up smaller pieces so that you avoid delaying all the work that gives you less recognition. You’ll then get motivation boosts from visible progress spread evenly on the project’s duration.

Challenge and Curiosity

Let’s associate both as I think they are very much linked when it comes to programming. Why? Because it seems that you can only be curious (learn, improve, read up, understand more) when something challenging comes up. If you are doing something you already know well it’s simply a question of applying knowledge and experience, nothing challenging and also nothing to be curious about.

Generally most projects are skewed in that regard as it’s a good idea to tackle the most difficult part upfront to avoid uncovering surprising unknowns later on during the project. That just means you either need to finds a really challenging project that involves mostly technologies / concepts that are new to you, but erring to far in that direction might just make you get lost and a lot less productive than working with know technologies. All a question of balance.

Control

Control is the last factor I lost, it’s a factor that I think, when working for a good employer, you most often have. In my case, because I started the project and headed it, I was responsible of bringing it to completion, abandoning it or trying to find teammate to finish it for me wasn’t really an option. Problem is: when motivation is rapidly decreasing and you can escape the task at hand, you quickly feel trapped and less and less in control of yourself, your results, your output.

Conclusion

  • Highs and lows are normal
  • Listen to you elders (or be ready to learn the hard way)
  • You can keep yourself really productive if you watch out for those 5 intrinsic motivation factors
  • It’s still all a question of balance, work is not always 100% rainbow and unicorns, just try to keep an equal amounts of that kind of work spread out, don’t do it all upfront, you’ll have a hard time with what comes next

TL;DR 1. Work with friends, 2. Chunk work in small pieces, 3. Always be learning, Actually do 1, 2 and 3 in balanced amounts.


References

Malone, T. W. & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learning, and instruction: III. Conative and affective process analysis. Hillsdale, NJ: Erlbaum.

Text editor implementation as a programming ├ętude

If you’re a programmer yourself and are looking to improve your skills, I would like to propose to generally “deliberately practice” your craft, in the sense “repeatedly attack problems at the edge of you capabilities in an exercise context, not at work”. And, as finding personal project ideas can be quite tricky for some, I would like to propose implementing a text editor as a really good project choice.

Now this proposition makes sense, in my opinion, because of the wide array of real hard problems related to so many different subjects of programming it has to offer. Also, there are good chances it’s quite different from what you work on daily (a majority of programmers are knee deep in the web of mobile applications these days) hence, will feel like a playful, new and exciting project.

Now hold your horses as, implementing a complete text editor that can rival with you current one (even if it’s Notepad or ed(1)) is a really big task. Try to start by setting your eyes on implementing a subpart of a one first.

Now for ideas on sub-parts that you could aim for:

  • A line editor: You know how your shell allows you to press backspace, delete characters to the beginning of the line, move the cursor? Well all of those are really nice thing but are quite novelties, most of there where not present in older shells. So, try your hand at implementing a program that asks for input but implements line editing features. Think of what a REPL does. Try for:
    • Typing characters
    • Backspace
    • Moving with arrows
    • Deleting to beginning/end of line
    • Moving to beginning/end of line
    • Moving word by word backward/forward
    • Entering in “replace mode” (like the insert key on keyboards)
  • Rendering a window tree of files: Try writing a terminal program that renders a tree of windows, each node being one of three types: horizontal split, vertical split, actual file/window. That one will get you thinking about recursing in a tree, caching information about location in files, calculating what is a line, how wide is a char, a tab, a Unicode char, how do you make it fast enough so that render wouldn’t block the editors main loop in a real editor. Try for:
    1. Reading files from disk
    2. Selecting a current position (so you get to implement scrolling)
    3. Creating a window tree so that all files have their own window split
    4. Rendering the window tree with nice window borders
    5. Framing and rendering the currently visible lines of the files
    6. Maybe have a status bar under each files showing stats line number of lines, current line, chars, file rights, file size, ….
    7. Make it fast by only rendering what’s needed when some file changes on disk
  • A file datastructure: Holding a file in memory, a task most editors need to do, is not an easy task. Getting it to be the right balance between: size in memory, insertion speed, deletion speed and interface complexity is a real struggle. The other problem you can attack after you have the basics right is testing operations on a file that is huge, bigger than 100MB and make that use case operations work in a decent time (< 100ms at least). Try exploring the following prior art in the matter:
    • Rope
    • Gap Buffer
    • Circular buffer
    • Red-Black tree
    • or a simple Linked List
    • or try the venerable Array
      Try testing how fast (and what’s the Big O of each?) of the following operations:
    • Inserting 1 character at the start/middle/end
    • Deleting 1 character at the start/middle/end
    • Inserting 10 000 characters at the start/middle/end
    • Deleting 10 000 characters at the start/middle/end
    • Inserting at the start the right after at the end
    • Loading in memory
    • Writing to disk
  • A command set/language: This one is a fun one for language lovers as it involves implementing a set of commands the user can use to edit files. You need to be able to parse an input string into an abstract syntax tree, then interpret it and execute it against a file contents. Here are good examples of editors that implemented a command set:
    • Vi - covers a lot, succinct [1][2][3]
    • Ed - not a visual editing, the grandfather of many others 1
    • Teco - really a language, also inspired a few others [1]
    • Emacs - less on point but think about keyboard keybindings and how natural they are to hit [1]
  • A plugin/configuration language: This one is all about implementing a full blown programming language (parser, interpreter, interface with host implementation language). A lot of toy editor project go without this one as it is a big chunk of work but a really crucial one in all popular editors theses day. You’ll be designing a language with the direct goal of exposing editor features, configuration and allowing the implementation of plugins that can change behaviour and call core editor methods. Take a look at the following languages that are used in popular editors:
    • ELisp
    • VimL
    • Lua
    • Python
    • Guile Scheme
    • Perl
  • A progressive rendering algorithm: This one is a bit smaller and needs quite a few pieces around it to make it work/visual. It consists in writing an algorithm that allows you to start a rerender of the screen following a user’s input but allows for stopping in the middle to handle user input then start back for where we where at the last rerender call making sure to invalidate parts that where just changed by the user’s action. Try reading this book at this point, it’s really one of the only books going in deep about many subject related to text editor implementation:

There a more smaller projects/parts that could be added to this list but at this point I would advise starting small but trying to build a complete editor and adding adding is all of those subparts together. Those first big goal being to be able to write the text editor with the new editor itself. :D

If you are interested in reading the implementation of a few toy editors with rather simple codebases you can look at:

Happy hacking & learning!

Testing web applications made fast and easy

Many other content already discusses about the advantages of testing. The advice almost always goes something like this: “Testing won’t add much to development time but will save your ass more than once on bugs and that’s before they even reach your customers. Plus, it has the nice side effect of making you write cleaner code if you write tests before code”.

I am here to talk about the two main pain-points developers have with writing tests, they are hard to write and lengthen the feedback loop (especially with large codebases).

Now those two factors are probably the biggest detractors of, first, the people looking to get into testing and TDD and, for the second, people writing test but hating it as their test suite is so slow it can take 1 hour to run.

That often leads to you pushing to the CI and hoping what you wrote didn’t break anything elsewhere, it’s an hour wait so you context switch to an other task, come back to it later, it failed, re-checkout the git branch fix the little details, push again…


Now, I am no different and here is what I propose: ~80% of tests you write are unit tests. That is 80% of tests you write and, in terms, ~80% of the running time of your test suite. Why not, forget about a real database, forget about HTTP, forget about all dependencies mock them all and only run (in Node.js’s case) pure JavaScript across just the few lines in the function you are currently testing, no other part of the codebase.

Here, let’s say you have this controller with a method fetching a list of users:

class UsersController {
  constructor(userRepository) {
    this.userRepository = userRepository;
  }

  users(req, res) {
    const limit = req.query.limit || 20;
    const order = req.query.order || 'created';

    return this.userRepository.find({limit, order})
      .then(users => {
        res.status(200);
        res.json({data: users});
      }, next)
  }
}

Now, the route most often taken to test this part of the code is to reach out for a library to do an http request and make sure you have a test database setup and also create few models so that you know what to look for in the controller’s response, all while making sure the database tables are cleared in between tests as you don’t want all those models from other tests showing up in the controllers response…

A lot to think about, a lot of setup, quite slow because of all the moving components and really, looks more like integration testing (which you should still be doing here and there) than unit testing.


How about creating a fake request object and a fake response, and just for this controller a fake userRepository that will help us verify that for a given input, the correct calls are made by the piece of code being tested.

// fake-request.js
class FakeRequest {
  constructor() {
    this.query = {};
    this.body = {};
    this.params = {};
  }
}
// fake-response.js
class FakeResponse {
  constructor() {
    this.statusCode = 200;
    this.jsonBody = null;
    this.endCalled = false;
  }

  json(value) {
    this.jsonBody = value;
  }

  status(code) {
    this.statusCode = code;
  }

  end() {
    this.endCalled = true;
  }

  // ...
}

Then with those two you can start writing a test specific to that controller’s users method:

// test/controllers/users.js
const assert = require('assert');
const FakeRequest = require('...');
const FakeResponse = require('...');
const UsersController = require('...');

describe('controllers:users', () => {
  let fakeUserRepository = {};
  let fakeRequest;
  let fakeResponse;
  let fakeNext;
  let usersContoller;

  beforeEach(() => {
    usersContoller = new UsersController(fakeUserRepository);
    fakeRequest = new FakeRequest();
    fakeResponse = new FakeResponse();

    fakeUserRepository.findOptions = null;
    fakeUserRepository.find = (options) => {
      fakeUserRepository.findOptions = options;
      return Promise.resolve([]);
    };

    fakeNext = (error) => {
      fakeNext.givenError = error;
    };
  });

  describe('users()', () => {
    it('calls userRepository defaulting to a limit of 20', () => {
      return usersContoller.users(fakeRequest, fakeResponse)
        .then(() => {
          assert(fakeUserRepository.findOptions);
          assert.equal(fakeUserRepository.findOptions.limit, 20);
        });
    });

    it('calls userRepository defaulting to ordering by creation date', () => {
      return usersContoller.users(fakeRequest, fakeResponse)
        .then(() => {
          assert(fakeUserRepository.findOptions);
          assert.equal(fakeUserRepository.findOptions.order, 'created');
        });
    });

    it('calls userRepository respecting query parameters', () => {
      fakeRequest.query.limit = 5;
      fakeRequest.query.order = 'name';

      return usersContoller.users(fakeRequest, fakeResponse)
        .then(() => {
          assert(fakeUserRepository.findOptions);
          assert.equal(fakeUserRepository.findOptions.limit, 5);
          assert.equal(fakeUserRepository.findOptions.order, 'name');
        });
    });

    it('calls next on userRepository error', () => {
      fakeUserRepository.find = () => Promise.reject('repo error');

      return usersContoller.users(fakeRequest, fakeResponse, fakeNext)
        .then(() => {
          assert.equal(fakeNext.givenError, 'repo error');
        });
    });

    it('sends the right response', () => {
      const result = [{id: 89, name: 'Jack'}, {id: 41, name: 'Dooey'}];
      fakeUserRepository.find () => Promise.resolve(result);

      return usersContoller.users(fakeRequest, fakeResponse)
        .then(() => {
          assert.equal(fakeResponse.statusCode, 200);
          assert.deepEqual(fakeResponse.jsonBody, {data: result});
        });
    });
  });
});

Sorry for the long file, but that’s all there is to it, running that test file takes no more that 2-3 milliseconds. A full test suite for a bigger project might be more like 20 seconds, a far cry from 20 minutes.

I know it isn’t perfect, there is a lot that happens in between component and you really can’t always write tests verifying exactly the right outputs but that’s why integration testing still has it’s place, to make sure everything integrates properly. Just don’t do it in place of actual unit tests and as 90% of the test suite.

As always, take this with a grain of salt, your projects are different than mines for sure, if you find that way of writing minimal unit tests promising try it out see if it sticks.

Bringing sanity to growing Node.js applications

It seems like most of the content written and blogged about Node.js, even now, 6 years in, takes a really basic approach to showing you how to build applications.

A lot of Node.js articles explain Express.js the leading web framework, the problem is, this framework is comparable to Sinatra in Ruby or Flask in Python or Silex in PHP. Good for small few pages website, basically gives you routing and an interface to HTTP but not much more.

Now Ruby, Python and others have bigger frameworks that are well suited for larger project where you benefits from more architecture, opinionated defaults and supporting modules (ORMs, utilities, rendering, mailing, background workers, assets pipelines). The story is a bit different in Node.js as it promotes small npm modules (i.e. gems, packages) that you put together by yourself, which, is a good thing, most experienced developers prefer libraries over frameworks, but, there is no literature or examples of how this can be done within the Node.js ecosystem.

So, to solve this, let’s try and define few libraries or simple files that can help us out with our growing codebase.

Goals

The goals here are to have something easier to maintain than an app.js file, a routes/, models/ and views/ folder that’s it. To achieve this we are going to go on a hunt and steal few time tested tricks from other ecosystems.

Dependency injection

Some people seem to dread this one, others love it. Having experienced it a lot in Laravel a great framework for PHP and in Java in quite a few places, dependency injection can help keep all our application parts and files decoupled. Leading to way easier unit testing and modification of dependencies.

The concept is, all of your request handlers/controllers have dependencies, your models/repositories/entities too, you could go and hard code them by requiring the right file and using it but if you let a dependency injection container do it for you, you can more easily change that required components by a different implementation of it and, when testing, you can directly pass in stubs/mocks without any trickery or magic.

So, how would we go about implementing this?

First step is to have a file that represents the global instance of the container. That is, where all instances will be stored and the tool that resolve needed dependencies when you’ll want to instantiate a controller.

It would look like this:

import Container from './lib/contrainer';
export default new Container();

Then, in your app.js you can register libraries you want to make available to the following classes you’ll register/use.

import express from 'express';
import container from './container';

let app = express();
// ... middlewares, config ...

// Manually setting intance
import EventEmitter from 'events';
container.set('events', new EventEmitter());

// Automatically resolving dependencies and setting an instance
container.load(require('./lib/config'));
container.load(require('./lib/db'));
container.load(require('./lib/auth'));
container.load(require('./repositories/user'));
container.load(require('./services/billing'));

// Using container to resolve dependencies but
// giving back the instance insted of setting it.
let requireUser = contrainer.get('auth').requireUserMiddleware;
let userController = container.create(require('./controllers/user'));
app.get('/users/:id', requireUser, userController.showUser);
app.get('/users/create', requireUser, userController.showCreateUser);
app.post('/users', requireUser, userController.createUser);

// ... error handling ...

app.listen(contrainer.get('config').get('port'));
container.get('events').emit('app:started');

(When you grow to have many more routes, extracting those to their own routes.js is a good idea)

The final piece being the DI container it-self. I tried making it as compact as possible.

import R from 'ramda';

export default class Container {
  constructor() {
    this.contents = {};
  }

  get(name) {
    if (!(name in this.contents)) {
      throw Error('Container has nothing registered for key ' + name);
    }
    return this.contents[name];
  }

  set(name, instance) {
    this.contents[name] = instance;
  }

  create(klass) {
    if (!('length' in klass.dependencies)) {
      throw new Error('Invariant: container can\'t resolve a class without dependencies');
    }

    var dependencies = R.map(function(dependencyName) {
      return this.get(dependencyName);
    }.bind(this), klass.dependencies);

    return applyToConstructor(klass, dependencies)
  }

  load(klass) {
    if (typeof klass.dependencyName !== 'string') {
      throw new Error('Invariant: container can\'t resolve a class without a name');
    }

    this.set(klass.dependencyName, this.create(klass));
  }

  unset(name) {
    delete this.contents[name]
  }

  reset() {
    this.contents = {};
  }
}

function applyToConstructor(constructor, args) {
  var newObj = Object.create(constructor.prototype);
  var constructorReturn = constructor.apply(newObj, args);

  // Some constructors return a value; let's make sure we use it!
  return constructorReturn !== undefined ? constructorReturn : newObj;
}

Repositories, Entities and Services instead of large Models

It’s been told on many blog posts and talks for a good while now that fat models are evil. The ActiveRecord pattern that’s so prevalent in Rails and many ORMs is easily replaced by separating concerns:

  • Data representation goes in models. Those are as dump as possible, optimally immutable.
  • Fetching/Saving/Database interactions are made in repositories. Those take plain models and knows how to persist them and query datastores.
  • Business logic goes into services. Services is the place where most of the complexity resides, it’s what controllers call with input, what validates business rules, what’s calling repositories and external apis.

To give concrete examples:

An entity is a simple POJO/PORO/POCO…

import R from 'ramda';

export default class InvoiceLine {
  constructor(params) {
    R.mapObjIndexed((v, k) => this[k] = v, R.merge(User.defaults, params));
  }

  taxAmount() {
    return this.price * this.taxes;
  }

  total() {
    return this.price + this.taxAmount();
  }
}
InvoiceLine.defaults = {price: 0, taxes: 0.15, created: Date.now()};

A repository will most likely take a database object in it’s constructor to be able to interact with the datastore. Repositories are singletons loaded once when stating the app using the container’s load method.

import User from '../entities/user';
const TABLE_NAME = 'users';

export default class UserRepository {
  constructor(db) {
    this.db = db;
  }

  findByEmail(email) {
    return this.db.select('id, name, email, ...')
      .from(TABLE_NAME)
      .where('email = ?', email)
      .limit(1)
      .exec();
  }
}
UserRepository.dependencyName = 'repositories:user';
UserRepository.dependencies = ['db'];

A service is the simplest of the 3 in form but the one in which most complexity will hide. It simply has instance methods and dependencies listed to it can be registered in the container for controllers to depend on.

export default class BillingService {
  constructor(userRepository, stripeService, mailer) {
    this.userRepository = userRepository;
    this.stripeService = stripeService;
    this.mailer = mailer;
  }

  createNewAccout(name, email, password, stripeToken) {
    // validate
    // create user
    // create stripe customer
    // update db user
    // send welcome email
    // ...
  }

  // ...
}
BillingService.dependencyName = 'services:billing';
BillingService.dependencies = [
  'repositories:user', 'services:stripe', 'mailer'
];

Slimmer Controllers in favor of Services

Now that we have a dedicated place to put business logic, you should aim to slim down those controllers to their essential job: mapping requests and the http protocol oddities to method calls/actions to be taken.

This simple action has the new benefit of decoupling yourself from the transport protocol enabling reuse of all that business logic by other consumers like: background workers, a websocket endpoint, a protobuff endpoint even a separate codebase if you decide to extract the core of your app into a library when you grow bigger.

Factories

As your project grows and your entities become more complex you may come to a point where you find yourself spending a lot of lines initializing entities in your services, it’s a good idea to extract those to factories. Those object will give you a clean way to encapsulate complex entity construction with many branches.

The lib folder still exists

Not everything fits into the concepts we just went over, there are few middlewares, really simple libs or wrapper and definitively have their place in your lib folder, just try to keep it lean and mean, most of your code is supposed to be elsewhere.

Conclusion

I hope this post gave you ideas on how to reduce the size and complexity of your routes files/folder. Code organization (/architecture) starts simple in a new project but needs to grow linearly as your project matures or your productivity will suffer quite a bit.

I would love to know how you deal with growing codebases too! DM on Twitter or send me an email.

Welcome to Hugo!

It’s been a little while I wanted to switch to a new bloging platform. My old solution was a tool a built myself (often unwise when a lot already exists) and was pretty basic. It got the job done, but did really have space for growth.

I am now joining the ranks of people using the static website generator called Hugo. Advantages it encompasses go from blazing fast compilation, by more than simply bloging, to simple in design but infinite in possibilities with concepts like content types (not plain blog posts) and taxonomies). Making themes for it was not an afterthought and was really well designed.

This was an occasion to try something new design wise, I wanted something more simple, a different layout, and, mainly, simple CSS and a serif font.

How hard is it?

Well, Hugo is a bit different in the way it approaches content but still easy to understand if you come from other static website generators like Jekyll or Metalsmith.

For a new site simply invoke (from the command like)

hugo new site <FOLDER>

When I wish to create a new post I type:

hugo new posts/2015-09-18-welcome-to-hugo.md

I will then base itself from the template at _archetypes/default.md and fill in the title and current date for me.

Next, firing up a development server is as simple as:

hugo server --buildDrafts -w

The --buildDrafts ensure you preview the posts that have draft = true in their metadata, allowing you to work on posts, put them on hold, and still keep publishing other posts.

When ready to publish something you simply invoke:

hugo

This will generate all the necessary html & static assets in the public directory that you can them upload or commit to Github Pages.

Hope this makes it less scary to get into! Hugo has good documentation as really is a seriously good option in the market of static site generators used for blogs.