devlog 2. filters post-mortem

This post outlines a recent project's shortcomings and what to do next based on lessons from Shape Up - Basecamp's methodology for product development. I believe these principles can transform how we approach scope management at Querio and significantly improve our ability to ship meaningful work consistently.

Hey everyone! At the time of writing this I shipped my first big feature at Querio - server side filters. It ran too long, burnt me out and was almost a complete disaster in most other aspects. I will explain what happened and where 33.333333% of the engineering team is going from here.

querio server side filters

Server side filter means that user should be able to modify the SQL that describes how to fetch their data. I explain more of the concept in the previous post, albeit slightly manically.

what happened

I will attempt to retrace my thoughts and actions from the start of the project until the final commit was merged. This will make it easier for us to establish problematic patterns of behaviour and (hopefully) come up with some remedies. I don't know about you but the most terse storytelling format I know is greentext, don't f with me on this.

>be me
>implement server-side filters as soon as humanly possible
>i give it a week 😎
>actually...
>spent a week digging around <https://github.com/tobymao/sqlglot>, until settled on approach that will work
>3 days to work out data-model, rewrote everything 2-3 times, settled on a dto layer
>1-2 days spent on no-gos like sharing db with main app and syncing over pg pub/sub with main app
>3-4 days on 'demo app' (that has nothing to do the actual app)
>confused why team looks at me weird
>a little over budget, but integrating this will be easy...
>i have never integrated into this app before
>integration needs slightly above average amount of boilerplate (see <https://querio.ai/blog/websockets>)
>pff i used to write dotnet i aint scared of boilerplate
>3 days later...
>filters appear in multiple places in the app, integrate everything again outside of websocket context
>use trpc queries - requests fail due to excessive header size
>cant communicate to server securely
>try to pass the session in the header - cannot decode, discover encryption keys dont match across services
>
>slowly going insane
>ported ui from demo app - does not look anything like our figma
>write styles, sticking head into rabbit-holes like selecting multiple options, supporting out-of-model condition types, error handling
>fix everything, push to prod
>self hosted customers cannot access
>didnt increment the minor version to trigger image re-publish
>"it's done!"
>get a cold, take 2 days off work

what went wrong and what to do next

The same two things that always go wrong: 1) poor shipping technique and 2) scope creep. Old me would write off (1) to animal spirits and (2) to these pesky project managers not defining correct requirements har har har. Except at Querio we don't have a project manager...so this is all me.

Old me is dead, here's what the new guy thinks.

poor shipping technique. managing scopes and being brave

Ever since I stopped being handed tasks and started getting handed projects I had to come up with tasks by myself. Senior devs always told me that before a new project I am to:

1. sit down if i am standing up

2. figure out all the stuff I have to do

3. make a todo list

4. tick items off

5. ???

6. Profit!

From here it went one of two ways: 1) I have done this before and I actually know what to do, or, more commonly 2) I have no idea what I'm doing but I am going to pretend that I do anyway. If I was unlucky enough to hit (2) I would almost always blow way past the project estimate, leaving a sour taste in everyone's mouth.

I was facing a false equivalence between imaginary and discovered tasks. Imaginary tasks are what we think we'll need to do based on our initial understanding, while discovered tasks emerge as we actually work on the project and gain deeper understanding of the problem space. The mistake is treating both types as equal when estimating and planning work.

In the case of filters I knew I would at least have to:

• establish a data model for server side filters

• transform arbitrary where conditions into the data model

• integrate new functionality into the app, being careful about the self-host customers

• write styles

These are imaginary tasks - my experience dictates that this is the kind of thing I do to ship this kind of project.

As weeks went on, the to-do list grew with discovered tasks. I label rabbit-holes with ? and no-gos with ! - more on them later.

• establish a data model for server side filters

• handle queries that have multiple scopes (CTEs, subqueries, etc) - that is most of them

• be upfront about the SQL dialect used

• !share the SQL schema for customer data we are filtering

• data model has to be flat

• data model needs to correctly handle logical connectors

• data model needs to be splice-able into the original SQL that fetched the data

• transform arbitrary where conditions into the data model

• cannot use conditions directly because of their recursive structure (given to me by SQL parser)

• !handle SQL conditions that are not currently supported by the data model

• ?some filters support multiple options which need to be transpiled back into SQL conditions

• integrate new functionality into the app, being careful about the self-host customers

• bypass self-hosted customers

• implement session persistence between these two components of the app

• handle payload limits

• write styles

• ?styles need to actually map over the data model

By the end of the project discovered tasks outnumber the imaginary kind 3 to 1. Reading the items you will notice dependencies between tasks. Notice also that each task can be one of two flavours. Completing some tasks results in finished work, completing others like integrate new functionality results in sub-tasks like implement session persistence and handle payload limits. The list is missing dimensions.

When I say "dimensions," I mean that a flat to-do list fails to capture the hierarchical nature of work (some tasks spawn sub-tasks), the dependencies between tasks (what must be completed before other things can start), and the relative importance of each task to the core value of the feature. A one-dimensional list treats all tasks as equal, when they're clearly not.

Aside from dimensions, the list above is missing structure. Structure (I have re-re-re-discovered) is important, doubly so at a startup. Here’s why.

I would join the daily standup calls, I'd usually be last because I am hammering out the 13th re-write of the data model, so I'd give my update first. With a nervous grind I'd recite: "I am still working on filters" to the sound of palms hitting faces and a 3000 yard stare from my CTO. "You guys don't want implementation details do you? I am doing you a favour!", I said. This excuse became less and less effective as my self-imposed deadline retreated further into the past.

Let's try to separate the final list into scopes along the lines of the original imaginary list. Let's work in the two kinds of tasks from above - one that finishes work and one that generates more work.

I'm introducing two key concepts here:

• Downhill scopes (v): These contain tasks that are well-understood and complete discrete chunks of work. When working on these, you gain momentum because each task you finish brings you closer to completion.

• Uphill scopes (^): These contain tasks that tend to generate more tasks as you work on them. These are zones of discovery and uncertainty, where the more you dig, the more you find that needs to be done.

Understanding which scopes are uphill vs. downhill helps set expectations and structure work more effectively. If the new me was doing the project, the todo-list might have looked something like this

1. query-engine (v)

   • establish a data model for server side filters

   • handle queries that have multiple scopes (CTEs, subqueries, etc) - that is most of them

   • be upfront about the SQL dialect used

   • data model has to be flat

   • data model needs to correctly handle logical connectors

   • data model needs to be splice-able into the original SQL that fetched the data

   • transform arbitrary where conditions into the data model

   • cannot use conditions directly because of their recursive structure (given to me by SQL parser)

2. integration (^)

   • !share the SQL schema for customer data we are filtering

   • !handle SQL conditions that are not currently supported by the data model

   • ?some filters support multiple options which need to be transpiled back into SQL conditions

   • integrate new functionality into the app, being careful about the self-host customers

   • bypass self-hosted customers

   • implement session persistence between these two components of the app

   • handle payload limits

3. frontend (^)

   • write styles

If I picked a scope and reported as I ticked off the tasks I'd be able to gain momentum and stop dreading the daily standup.

But how to pick a scope to build first? There are three tenets. The thing to build first should be:

1. core - thing most central to the project and most important to prove early

2. small - finish something meaningful in a few days and build momentum

3. novel - something that I haven't done before and thus am more likely to meet resistance (remain uphill the longest)

With this, the above middle-of-the-road todo list might become:

1. query-engine (v) - CORE, SMALL, NOVEL

   • establish a data model for server side filters

   • data model has to be flat

   • data model needs to be splice-able into the original SQL that fetched the data

   • transform arbitrary where conditions into the data model

   • cannot use conditions directly because of their recursive structure (given to me by SQL parser)

2. complex queries (^)

   • handle queries that have multiple scopes (CTEs, subqueries, etc) - that is most of them

   • data model needs to correctly handle logical connectors

   • ?some filters support multiple options which need to be transpiled back into SQL conditions

3. error-handling (^)

   • !handle SQL conditions that are not currently supported by the data model

4. platform (^)

   • bypass self-hosted customers

   • implement session persistence between these two components of the app

   • handle payload limits

   • be careful about the self-host customers

5. integration (v)

   • integrate query-engine into app

   • be upfront about the SQL dialect used

   • !share the SQL schema for customer data we are filtering

6. frontend (^)

   • write styles

I've marked the first scope as meeting all three tenets: it's CORE (the fundamental functionality needed), SMALL (achievable in a few days), and NOVEL (something I haven't done before). Prioritising scopes this way allows us to tackle the riskiest, most valuable work first, while building confidence through quick, meaningful wins.

Right? Wrong!

In hindsight I know that doing the uphill work for the query-engine scope took at least a week, including the ramp-up time (time to get acquainted with the library I was using for SQL parsing). At the end of the week the core might be done but there is nothing to demo.

I would instead take a cross-layer slice of the project and integrate everything e2e into a simple dropdown filter inside the app. Integration work would be mostly downhill is demoable and would force resolution on 80% of integration issues I ended up facing towards the very end of the project. Nice way to build momentum.

I don't know about you, but just looking at the structure of this todo-list gives me a sense of inner peace I haven't experienced since my friend Alex wrote the Prolog coursework for us both at university. I feel like the angels are singing to me.

Hey, but what about the rabbit-holes and no-gos?

i'm a creep i'm a weirdo

I used to think it was project manager's work to eliminate scope creep, but as uphill and downhill scopes have shown us, creeping is what scopes do. It is the natural order of things. More importantly the creeping happens after the scope is considered defined by most project managers - it is some work's nature to produce more work.

This understanding fundamentally changes how we need to approach scope management. Instead of pretending we can define everything perfectly upfront (we can't), we need to actively manage scope throughout the project lifecycle. So it must be that it is I who has to be the shield that guards the realms of men from scope creep and our roadmap from being turned into a shitshow.

Every part of the product does not need to be equally prominent, equally fast and equally polished. Every use case isn't equally common, equally critical, or equally aligned with the market we are trying to sell to.

Like many developers I know, I am infected with the perfectionist mind-virus. When I write code I compare what I have with the ideal - a feature that might work perfectly when all the cases are handled and all the bugs squashed. Needless to say the likelihood of that kind of feature appearing before we run out of runway approaches zero. Going forward I will compare to baseline - it's a difference between "never good enough" and "better than what we have now".

When we come up with things to fix, add, improve, rewrite for the 5th time or redesign during a project we may ask:

• is this a must have?

• if we ship without this what happens?

• is this a new problem or pre-existing one that customers already live with

• how likely is this to happen?

• when something doesn't work well for a particular use case, how aligned is this use case with our intended audience?

That is scope hammering - actively cutting back scope to fit our appetite and focus on what matters. Scope hammering doesn't mean doing sloppy work - it means being strategic about what you choose to build and what you choose to defer or ignore. It means staying disciplined about focusing on what provides the most value rather than chasing every edge case or "nice-to-have" feature. Is it the cure for the perfectionist mind virus? Only time can tell.

they see me creepin', they hatin'

If you thought I'd end this log without dunking on project management you are wroooong! I have alluded to the designs for filters being inconsistent with the experience we were designing (forms in figma were not representative of affordances we wanted to provide to the user). Moreover, I said that I (me) have set the deadline. That means the rest of the team had a deadline in mind that they would prefer but were not prepared to enforce.

Problem is, we are prone to thinking of deadlines exactly backwards. Estimations (bad usually and especially so in murky waters) start with a feature and end with a number. You're guessing how long it would take to complete an unknown scope, whose nature is to grow as it is being discovered. The inputs and outputs are backwards.

This is where the concept of an "appetite" from Shape Up comes in. The number one most important input into a project's scope is the time we are willing to spend on it - the appetite. Instead of asking "how long will this take?", we should be asking "how much time is this worth?" and then designing the solution to fit within that timeframe.

The appetite could be any length of time, in Querio's case it may be 1-2 weeks - no more than 4 (I'd say). Important thing is that the appetite is picked once and honoured throughout the project lifecycle. This flips the traditional approach on its head - rather than letting the scope determine the timeline, we let the timeline (appetite) constrain the scope. This creates a healthier, more predictable development process.

The harsh deadline forces scope hammering. It also incentivizes finding rabbit-holes and no-goes during the design phase - what Shape Up calls "shaping" the problem, before time is committed to it.

Shaping is the critical work that happens before a project is committed to. It's about:

• Defining the problem clearly

• Setting boundaries on the solution

• Identifying risks and rabbit-holes upfront

• Creating enough structure for the team to build upon without over-specifying every detail

A well-shaped project outlines the rabbit-holes to avoid (and suggests how to handle them) and no-go zones to not bother with (do not try to connect filters to the database to share the user schema). The result is a project with clear boundaries. Good fences make good neighbours and clear boundaries make great relationships. This applies to both the relationship between team members working on a project and the relationship between our team and the goals/timeline we've committed to.