Hi, I'm Frederic.

Web Developer & Entrepreneur
blogging about life and programming in Node.js, Go and React.

GithubTwitterEmailKeybase

Agile: Ideal vs. Reality

DISCLAIMER: This is the internet, a lot of people travel it’s seas. This article represents my own opinion based on my current/past work experience. It is ok if you disagree with me, it’s actually likely you will.

I’d like to argue that nowadays many companies adopted Agile as their “software development methodology” but have greatly diverged from Agile’s original intent. Enough so that it’s gotten hard to even relate the original Agile manifesto with how startups build software.

As a refresher here is the original Agile manifesto:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

Methodology or Values?

First of all, the Agile manifesto never was a methodology. It’s a set of values. The problem is that over the years it’s been heavily commercialized, books were written, formations were created, consultants were trained, roles were invented. This resulted in a very lucrative business promising the original benefits of Agile but consistently under-delivering on them.

To a fresh pair of eyes, the whole process of “doing Agile” seems so bloated and full complex processes and meetings. We made the process of building software into something rigid and structured which are exactly the prevailing properties of Waterfall that the Agile manifesto was trying to avoid.

Processes

Focusing on the first 25% of the manifesto: “Individuals and interactions over processes and tools”, the processes introduced by modern Agile are not all that bad. It still gets developers, managers, and PMs interacting in order to build software. Them problem is how codified those interactions have become. Sprint Planning, Daily Scrum, Sprint Review / Demo, Sprint Retrospective, Backlog grooming all add up when on a 2 weeks schedule but more important is how much value (or how little in some cases) is extracted from these.

Daily Scrum / Standup seems like a nice idea, keeps everybody in sync, it’s a good place to voice blockers. But, in reality when your team is made up of 3-5 developers working on the same project, constantly reviewing each other’s code: you know what they are up to. If you hit a blocker or even if you envision a possible blocker coming I would it’s expected that you speak up and go find the people/solution you need to unblock your work, waiting till 9 AM next morning makes no sense especially that the problem won’t even be addressed during standup as it’s not the place for longer 1-1 discussions. Standup gains in value when your team getting bigger and you don’t know what everybody is working on, but, that team size brings a lot of other pains with it, small self-organized and self-sufficient teams seem to work best in practice.

Sprint Review in the context of a startup/product company (in contrast with a consultancy) looses a lot of it’s meaning. Your clients are out there and by the thousand, they also don’t care about the technical tasks you’ve done this sprint. They don’t care for half done features either. Deploying a new feature and writing a blog post about it is the equivalent of a Sprint Review for a product company.

Sprint Retrospective is useful, but having a set time to do so seems backwards. Retrospectives should always be happening: when there is something to retrospect on! Team members should always be thinking about processes, wins, and failures. If something feels wrong or something seems worth celebrating you should feel empowered to speak up right there and then not a set one hour meeting every two weeks.

Sprint Planning’s goal is a useful exercise but the way it’s normally done seems more harmful than helpful, the next section will expand on why I think it’s that way…

Planning

There is no denying planning is valuable, it’s also very hard.

First, humans are quite ill-equipped to predict the future, it’s really difficult to divorce your thinking from the present and imagine what the future could look like.

Second, in software, there will always be surprises, things you initially thought was simple but happen to be a lot more involving.

Third, details and polish is extremely time-consuming, Power Law is in full effect here, you can get a minimal solution done in 20% of the time, it’ll do 80% of the job but getting to a point where you are satisfied with your solution, that last 20%, will take 4 times more time than that initial version.

So given it’s valuable but hard to get right, you want to do enough planning that you get value out of it but not so much that extra efforts expended are rendered useless (or worse detrimental).

The main knob you have control over is the level of details you go into while planning. At the very high level you have company goals, then increasing the level of detail: roadmap, project, feature, task, class, line of code. The further down that list you go and the more detailed you make each of those items the predicting the future you are trying to do. Don’t forget, surprises are inevitable.

Now, what’s the point in cutting up projects in detailed features and features in small tasks when the moment one of the smart engineers you hired is faced with starting to work on these he’ll painstakingly figure out the original assumptions were wrong and 50% of the tasks are wrong in their definition, 25% are not needed and can be deleted and 25% were missing and not even planned for.

To reiterate the Agile manifesto: “Responding to change over following a plan”.

Conclusion

I get that management wants a measure of how long projects will take. I get that for endeavors involving multiple teams they want to order dependencies to avoid teams waiting on other teams. But, I would argue that if you took all that process, all those meetings, all those conservative estimates, all that top-down task splitting, all that engineer protectionism and threw a whole project at them from the get go, you would be amazed by how fast it will materialize and how quick an initial version would be launched.*

It’s important to keep “Working software over comprehensive documentation” in mind, if your detailed plan starts with a perfect Part1 followed by a perfect Part2 … Part3 then at each stage there is nothing you can ship to customers. What you want is a skeleton of Part 1, 2 and 3, then improvements to all parts the next month and so on. This avoids a lot of re-work.

When planning in advance we try and think of all that’s needed but so much of that work goes down the drain as requirements will change, third party APIs will not work as you expect, and integration points with other teams are more complex than you envision. Letting smart engineers figure this out as they go is the most efficient way of making progress and limiting wasted time.

* That is, given you hired smart programmers, motivated by hard problems, able to handle autonomy.

Setting Goals

At time, months can seem to go by so fast. And months turn into years quick. We all dreamt of becoming X when younger. That X profession or occupation probably morphed over time. It became something less exciting but much more “realistic”.

A similar thing happens with dreams and aspirations. Most people have some secret aspirations they daydream about from times to times. The problem is they feel so big, so imposing, so far away, out of our reach.

Here’s the thing, every person you look up to didn’t get there by accepting their current reality. They got there by setting their mind on a goal and acting on it. All action towards a goal, even if it seems small, brings you closer to accomplishing it. It makes every action coming after easier. The compounding effect every one of those small steps is difficult to grasp but works wonders over the course of many years.

If you have a dream, you can spend a lifetime studying, planning, and getting ready for it. What you should be doing is getting started.
Drew Houston

So start now! Pick something you wish to do or become and write it down. Make sure it’s someplace you will see on the daily. Then, take the first step in the general direction of achieving it. However small it may seem, it’ll have a big impact when compounded with all the other steps you’ll take in the weeks, months, and years to come.

That’s it. Stop wishing and start moving by being deliberate about your goals, writing them down, and acting on them.

This post is best enjoyed while listening to Time from Hans Zimmer at very high volume. :P

To break out of character a bit… I am not used to writing those kind of inspirational posts but I do enjoy that kind of content from time to time. So I am trying my hand at it. Sometimes it’s exactly what you need to set yourself back on track ;)

On Motivation

In this post we’ll discuss the 5 factors that increase intrinsic motivation and how by gradually losing those while working on a too big project, my motivation eventually grinded to halt so did my productivity and happiness. (TD;DR at the bottom)

Introduction

Motivation is tricky. For some people it’s mostly hovers a steady level most of days. But, it seems that for more entrepreneurial, driven, type A, win or nothing persons it looks a lot more like an epic roller coaster, it’s swayed up or down a lot more.

I’ve been having a really hard time dealing with a large rewrite project, yes, the thing most experienced programmer recommends not doing, ever. The plan was, take 60% of what was written in the past year by 4-8 engineers, rewrite it from scratch, bring in new features other teammates add the old codebase and finally ship it effectively deprecating the old code.

Context

That endeavour may seem like really too big and impossible to achieve in mere months (by myself alone with a bit of help) but I actually believed it was doable mostly because:

  • I had now worked in most parts of the codebase and knew pretty well how it clicked
  • I had already tried the refactor part by part approach and failed as they always became too big projects, it’s really difficult to replace an jet’s engine while the plane is flying
  • From those previous attempts I had code that could be reused
  • I planned on getting some time away from the office to concentrate on getting as much as possible done in the first 2 weeks

Now that we are done and shipping, 3 months later (and I got some help from coworkers down the road), I can see how 1-2 months could have been enough, if I worked at 200% productivity the whole time, like in the first 2 weeks. See, programmers never where good at estimating, we are too biased and there are too much variables involved.

Intrinsic Motivation

Intrinsic Motivation, compared to extrinsic motivation (earn external rewards / avoid punishments), is a lot more powerful in how motivating it is. We’ll discuss the 5 factors that increase it and how by gradually losing those, my motivation eventually grinded to halt so did my productivity and happiness.

In Making Learning Fun: A Taxonomy of Intrinsic Motivations for Learning Malone and Lepper (1987) identified the following 5 factors as increasing intrinsic motivation:

  • Challenge: Goals with a personal meaning, achieving the goal is possible but not certain
  • Curiosity: In this case, cognitive curiosity, the opportunity to learn
  • Control: Control of yourself, your environment and your task at hand
  • Cooperation and Competition: Helping others / Comparing your performance to others
  • Recognition: Having your accomplishments recognized by others

Cooperation and Competition

I started the project with a coworker, he was helping me out for the first 2 weeks, in retrospect, things went downhill after that point.

Having somebody to work with is such a game changer compared to working alone as it allows you to split the mental load in two, you are not bearing the stress all alone, you can talk about the project you are on, support each other. That sense that, not only you have help, but also that you have somebody you are competing against productivity wise is a huge motivator. People grossly underestimate the impact of peer pressure!

Once that lovely person I could compete with and share the hurdles with was gone it showed immediately in how much I accomplishing daily.

Recognition

See, the thing with programming is that not all work is equal in how much of it is visible. New features can be written in a very ugly way (and run slowly) but still work in a very short amount of time. In the first two weeks we got 90% of the pages / functionality rewritten, people where impressed, it was out of this world progress, compressing 6 man-year of work in potentially 2 man-months? Incredible! Except it wasn’t.

Software has this other category of work that most often takes up 80% of the time to produce: test, performance, best practices and edge cases. We skipped on writing tests as we where still shaping up how everything was to be structured and we wanted to show progress quickly. In and of itself this is OK, except I wasn’t ready to deal with the 2 months of that kind of work that came right after.

The lesson here is chunk your work up smaller pieces so that you avoid delaying all the work that gives you less recognition. You’ll then get motivation boosts from visible progress spread evenly on the project’s duration.

Challenge and Curiosity

Let’s associate both as I think they are very much linked when it comes to programming. Why? Because it seems that you can only be curious (learn, improve, read up, understand more) when something challenging comes up. If you are doing something you already know well it’s simply a question of applying knowledge and experience, nothing challenging and also nothing to be curious about.

Generally most projects are skewed in that regard as it’s a good idea to tackle the most difficult part upfront to avoid uncovering surprising unknowns later on during the project. That just means you either need to finds a really challenging project that involves mostly technologies / concepts that are new to you, but erring to far in that direction might just make you get lost and a lot less productive than working with know technologies. All a question of balance.

Control

Control is the last factor I lost, it’s a factor that I think, when working for a good employer, you most often have. In my case, because I started the project and headed it, I was responsible of bringing it to completion, abandoning it or trying to find teammate to finish it for me wasn’t really an option. Problem is: when motivation is rapidly decreasing and you can escape the task at hand, you quickly feel trapped and less and less in control of yourself, your results, your output.

Conclusion

  • Highs and lows are normal
  • Listen to you elders (or be ready to learn the hard way)
  • You can keep yourself really productive if you watch out for those 5 intrinsic motivation factors
  • It’s still all a question of balance, work is not always 100% rainbow and unicorns, just try to keep an equal amounts of that kind of work spread out, don’t do it all upfront, you’ll have a hard time with what comes next

TL;DR 1. Work with friends, 2. Chunk work in small pieces, 3. Always be learning, Actually do 1, 2 and 3 in balanced amounts.


References

Malone, T. W. & Lepper, M. R. (1987). Making learning fun: A taxonomy of intrinsic motivations for learning. In R. E. Snow & M. J. Farr (Eds.), Aptitude, learning, and instruction: III. Conative and affective process analysis. Hillsdale, NJ: Erlbaum.

Text editor implementation as a programming Γ©tude

If you’re a programmer yourself and are looking to improve your skills, I would like to propose to generally “deliberately practice” your craft, in the sense “repeatedly attack problems at the edge of you capabilities in an exercise context, not at work”. And, as finding personal project ideas can be quite tricky for some, I would like to propose implementing a text editor as a really good project choice.

Now this proposition makes sense, in my opinion, because of the wide array of real hard problems related to so many different subjects of programming it has to offer. Also, there are good chances it’s quite different from what you work on daily (a majority of programmers are knee deep in the web of mobile applications these days) hence, will feel like a playful, new and exciting project.

Now hold your horses as, implementing a complete text editor that can rival with you current one (even if it’s Notepad or ed(1)) is a really big task. Try to start by setting your eyes on implementing a subpart of a one first.

Now for ideas on sub-parts that you could aim for:

  • A line editor: You know how your shell allows you to press backspace, delete characters to the beginning of the line, move the cursor? Well all of those are really nice thing but are quite novelties, most of there where not present in older shells. So, try your hand at implementing a program that asks for input but implements line editing features. Think of what a REPL does. Try for:
    • Typing characters
    • Backspace
    • Moving with arrows
    • Deleting to beginning/end of line
    • Moving to beginning/end of line
    • Moving word by word backward/forward
    • Entering in “replace mode” (like the insert key on keyboards)
  • Rendering a window tree of files: Try writing a terminal program that renders a tree of windows, each node being one of three types: horizontal split, vertical split, actual file/window. That one will get you thinking about recursing in a tree, caching information about location in files, calculating what is a line, how wide is a char, a tab, a Unicode char, how do you make it fast enough so that render wouldn’t block the editors main loop in a real editor. Try for:
    1. Reading files from disk
    2. Selecting a current position (so you get to implement scrolling)
    3. Creating a window tree so that all files have their own window split
    4. Rendering the window tree with nice window borders
    5. Framing and rendering the currently visible lines of the files
    6. Maybe have a status bar under each files showing stats line number of lines, current line, chars, file rights, file size, ….
    7. Make it fast by only rendering what’s needed when some file changes on disk
  • A file datastructure: Holding a file in memory, a task most editors need to do, is not an easy task. Getting it to be the right balance between: size in memory, insertion speed, deletion speed and interface complexity is a real struggle. The other problem you can attack after you have the basics right is testing operations on a file that is huge, bigger than 100MB and make that use case operations work in a decent time (< 100ms at least). Try exploring the following prior art in the matter:
    • Rope
    • Gap Buffer
    • Circular buffer
    • Red-Black tree
    • or a simple Linked List
    • or try the venerable Array
      Try testing how fast (and what’s the Big O of each?) of the following operations:
    • Inserting 1 character at the start/middle/end
    • Deleting 1 character at the start/middle/end
    • Inserting 10 000 characters at the start/middle/end
    • Deleting 10 000 characters at the start/middle/end
    • Inserting at the start the right after at the end
    • Loading in memory
    • Writing to disk
  • A command set/language: This one is a fun one for language lovers as it involves implementing a set of commands the user can use to edit files. You need to be able to parse an input string into an abstract syntax tree, then interpret it and execute it against a file contents. Here are good examples of editors that implemented a command set:
    • Vi - covers a lot, succinct [1][2][3]
    • Ed - not a visual editing, the grandfather of many others 1
    • Teco - really a language, also inspired a few others [1]
    • Emacs - less on point but think about keyboard keybindings and how natural they are to hit [1]
  • A plugin/configuration language: This one is all about implementing a full blown programming language (parser, interpreter, interface with host implementation language). A lot of toy editor project go without this one as it is a big chunk of work but a really crucial one in all popular editors theses day. You’ll be designing a language with the direct goal of exposing editor features, configuration and allowing the implementation of plugins that can change behaviour and call core editor methods. Take a look at the following languages that are used in popular editors:
    • ELisp
    • VimL
    • Lua
    • Python
    • Guile Scheme
    • Perl
  • A progressive rendering algorithm: This one is a bit smaller and needs quite a few pieces around it to make it work/visual. It consists in writing an algorithm that allows you to start a rerender of the screen following a user’s input but allows for stopping in the middle to handle user input then start back for where we where at the last rerender call making sure to invalidate parts that where just changed by the user’s action. Try reading this book at this point, it’s really one of the only books going in deep about many subject related to text editor implementation:

There a more smaller projects/parts that could be added to this list but at this point I would advise starting small but trying to build a complete editor and adding adding is all of those subparts together. Those first big goal being to be able to write the text editor with the new editor itself. :D

If you are interested in reading the implementation of a few toy editors with rather simple codebases you can look at:

Happy hacking & learning!

Testing web applications made fast and easy

Many other content already discusses about the advantages of testing. The advice almost always goes something like this: “Testing won’t add much to development time but will save your ass more than once on bugs and that’s before they even reach your customers. Plus, it has the nice side effect of making you write cleaner code if you write tests before code”.

I am here to talk about the two main pain-points developers have with writing tests, they are hard to write and lengthen the feedback loop (especially with large codebases).

Now those two factors are probably the biggest detractors of, first, the people looking to get into testing and TDD and, for the second, people writing test but hating it as their test suite is so slow it can take 1 hour to run.

That often leads to you pushing to the CI and hoping what you wrote didn’t break anything elsewhere, it’s an hour wait so you context switch to an other task, come back to it later, it failed, re-checkout the git branch fix the little details, push again…


Now, I am no different and here is what I propose: ~80% of tests you write are unit tests. That is 80% of tests you write and, in terms, ~80% of the running time of your test suite. Why not, forget about a real database, forget about HTTP, forget about all dependencies mock them all and only run (in Node.js’s case) pure JavaScript across just the few lines in the function you are currently testing, no other part of the codebase.

Here, let’s say you have this controller with a method fetching a list of users:

class UsersController {
  constructor(userRepository) {
    this.userRepository = userRepository;
  }

  users(req, res) {
    const limit = req.query.limit || 20;
    const order = req.query.order || 'created';

    return this.userRepository.find({limit, order})
      .then(users => {
        res.status(200);
        res.json({data: users});
      }, next)
  }
}

Now, the route most often taken to test this part of the code is to reach out for a library to do an http request and make sure you have a test database setup and also create few models so that you know what to look for in the controller’s response, all while making sure the database tables are cleared in between tests as you don’t want all those models from other tests showing up in the controllers response…

A lot to think about, a lot of setup, quite slow because of all the moving components and really, looks more like integration testing (which you should still be doing here and there) than unit testing.


How about creating a fake request object and a fake response, and just for this controller a fake userRepository that will help us verify that for a given input, the correct calls are made by the piece of code being tested.

// fake-request.js
class FakeRequest {
  constructor() {
    this.query = {};
    this.body = {};
    this.params = {};
  }
}
// fake-response.js
class FakeResponse {
  constructor() {
    this.statusCode = 200;
    this.jsonBody = null;
    this.endCalled = false;
  }

  json(value) {
    this.jsonBody = value;
  }

  status(code) {
    this.statusCode = code;
  }

  end() {
    this.endCalled = true;
  }

  // ...
}

Then with those two you can start writing a test specific to that controller’s users method:

// test/controllers/users.js
const assert = require('assert');
const FakeRequest = require('...');
const FakeResponse = require('...');
const UsersController = require('...');

describe('controllers:users', () => {
  let fakeUserRepository = {};
  let fakeRequest;
  let fakeResponse;
  let fakeNext;
  let usersContoller;

  beforeEach(() => {
    usersContoller = new UsersController(fakeUserRepository);
    fakeRequest = new FakeRequest();
    fakeResponse = new FakeResponse();

    fakeUserRepository.findOptions = null;
    fakeUserRepository.find = (options) => {
      fakeUserRepository.findOptions = options;
      return Promise.resolve([]);
    };

    fakeNext = (error) => {
      fakeNext.givenError = error;
    };
  });

  describe('users()', () => {
    it('calls userRepository defaulting to a limit of 20', () => {
      return usersContoller.users(fakeRequest, fakeResponse)
        .then(() => {
          assert(fakeUserRepository.findOptions);
          assert.equal(fakeUserRepository.findOptions.limit, 20);
        });
    });

    it('calls userRepository defaulting to ordering by creation date', () => {
      return usersContoller.users(fakeRequest, fakeResponse)
        .then(() => {
          assert(fakeUserRepository.findOptions);
          assert.equal(fakeUserRepository.findOptions.order, 'created');
        });
    });

    it('calls userRepository respecting query parameters', () => {
      fakeRequest.query.limit = 5;
      fakeRequest.query.order = 'name';

      return usersContoller.users(fakeRequest, fakeResponse)
        .then(() => {
          assert(fakeUserRepository.findOptions);
          assert.equal(fakeUserRepository.findOptions.limit, 5);
          assert.equal(fakeUserRepository.findOptions.order, 'name');
        });
    });

    it('calls next on userRepository error', () => {
      fakeUserRepository.find = () => Promise.reject('repo error');

      return usersContoller.users(fakeRequest, fakeResponse, fakeNext)
        .then(() => {
          assert.equal(fakeNext.givenError, 'repo error');
        });
    });

    it('sends the right response', () => {
      const result = [{id: 89, name: 'Jack'}, {id: 41, name: 'Dooey'}];
      fakeUserRepository.find () => Promise.resolve(result);

      return usersContoller.users(fakeRequest, fakeResponse)
        .then(() => {
          assert.equal(fakeResponse.statusCode, 200);
          assert.deepEqual(fakeResponse.jsonBody, {data: result});
        });
    });
  });
});

Sorry for the long file, but that’s all there is to it, running that test file takes no more that 2-3 milliseconds. A full test suite for a bigger project might be more like 20 seconds, a far cry from 20 minutes.

I know it isn’t perfect, there is a lot that happens in between component and you really can’t always write tests verifying exactly the right outputs but that’s why integration testing still has it’s place, to make sure everything integrates properly. Just don’t do it in place of actual unit tests and as 90% of the test suite.

As always, take this with a grain of salt, your projects are different than mines for sure, if you find that way of writing minimal unit tests promising try it out see if it sticks.