Revolutionizing Disease Data Access: Airfinity's Journey to Lightning-Fast Insights

During the global COVID-19 pandemic, I worked for Airfinity, a disease forecasting company that rapidly adapted to the fast-moving nature of the outbreak. As we collected and aggregated various data, it became clear that we needed to move quickly. Our small development team innovated to meet these challenges. Although the platform initially struggled, with some pages taking an unacceptable 28 seconds to load, we were able to solve these issues effectively.

In Search of the Best Way to Deliver Data

During this time, requirements changed within days, and the shape of data varied significantly. At its simplest, we provided tables with powerful search and filtering systems. We also delivered many custom dashboards, graphs, and analyses daily, often tailored to specific entities' needs.

To meet these demands, we built a custom platform and utilized other tools like Tableau and Shiny dashboards, depending on the best fit for the required data.

Configuration over Code

One key challenge was serving this ever-changing data in a timely manner. We addressed this through our "configuration over code" approach.

While this term isn't widely explained online, it essentially means preferring to use a set of settings instead of code to modify a program. For example:

Preferring:

const paragraphs = [
  "Hello",
  "World"
]

<>
    {paragraphs.map(paragraph => <p>{paragraph}</p>)}
</>

Instead of:

<>
    <p>Hello</p>
    <p>World</p>
</>

This simple example illustrates a concept we applied throughout the platform.

Using Typescript as our configuration object

A crucial decision was choosing the form of our configuration object. We opted for TypeScript for its type validation, allowing us to move quickly without unknowingly breaking things. We validated routes, data columns, and even smaller elements with features like string unions.

This approach made it easy to add content while maintaining strong type checks. Importantly, we could compose our configuration in various ways, abstracting and reusing similar configs across pages. However, we approached this carefully, as shared elements don't always warrant combination.

While this method proved great overall, it did have some limitations:

1. Using complex types makes it harder to onboard developers

In some parts of the system, we implemented strict checks to avoid common errors. This involved a pretty complex set of types. This worked well but required developers to understand the types to make changes, presenting a challenge for those without TypeScript knowledge. Despite this, we felt it was a fair trade-off.

2. Dead Configuration

The ease of adding new configurations led to daily additions. However, this also made it easy to leave unused elements in the system. Periodically, we had to clean up these unused configurations to maintain easier system maintenance.

3. Serializability

Using TypeScript allowed us to include non-serializable elements like functions and classes in the config. While convenient, this came at the cost of serializability and the ability to perform static analysis.

Managing Data Overload. From 28s -> 1.8s Load Time

Our rapid development occasionally required implementing quick solutions. One such instance occurred when users reported extremely long load times on certain data-heavy pages. The ideal solution of fetching only necessary data from the backend wasn't immediately feasible due to complex data manipulations and user customizations. So we made the call to handle this on the frontend.

We implemented an incremental loading and caching system using web workers and IndexedDB for storage. This system loaded data incrementally using exponential backoff through a web worker to avoid blocking the UI thread. The data was then cached in IndexedDB, allowing us to perform needed manipulations without significant performance penalties and allowing us to reuse data on subsequent visits. This approach also required us to manage cache invalidation, but that's a story for another time.

Implementation was challenging, with many moving parts and obstacles like the serializability issue mentioned earlier. The diagram below illustrates the approximate structure of this solution.

The Impact

After implementing these solutions, we achieved:

1. A 93.5% improvement in page load times, reducing them from 28 seconds to just 1.8 seconds in the worst cases
2. Maintain real-time data access crucial for timely decision-making
3. Scale the platform to handle the ever-growing influx of data
4. Kept the platform simple and easy to work with for future developers

Airfinity has since then won the Tech Champions 2023 award and was shortlisted for the DataIQ Awards 2023 ‘Best Data Story or Data Visualisation’ award. They have also expanded their operations to the US and Japan.