Scaling notifications at Grow Therapy: Deliverability and reliability at scale

This article was co-written by Evan McDaniel, Martin Cservenka, and Avery Huang. 

In healthcare, notifications are a critical component of the care journey. When they work well, clients can more reliably engage with their care, and providers can more easily manage their practice. When they arrive late, go to the wrong place, or don't show up at all, it affects both the provider’s ability to deliver care and the client’s willingness to show up and trust the process. 

But building a reliable notification system is a surprisingly difficult technical challenge, with complexity at every layer of the stack. We work with various carriers and email providers, and each of them have their own rules around message quality that can affect deliverability. We also use various third-party tools to send notifications, and each comes with its own quirks and rate limits. 

As we grew, we kept hitting the limits of what our notification setup could handle. Patching gaps over time only got us so far; eventually, we realized we needed to rebuild from the ground up.

In this blog post, we’ll walk through that journey and how we transformed from a fragmented setup to a centralized engine that ensures critical healthcare reminders are reliably delivered.

The limitations of our initial notification system

Today, we send notifications across three channels — email, push, and SMS — and they fall into four categories:

• Critical updates: Time-sensitive alerts like new appointment bookings, changes to existing appointments, and new message notifications. 

• Reminders: Nudges that remind clients of important events, such as upcoming appointments.

• Task prompts: Requests to complete key tasks, such as intake forms or clinical measures. 

• Marketing notifications: Outreach and promotional messaging.

Examples of the different types of notifications that we send at Grow Therapy. 

When we first added notifications, we implemented them within product flows by directly calling our third-party provider APIs. This decision allowed us to launch notifications quickly, but it meant that notification logic was scattered across the codebase — making it hard to track, audit, or enforce consistency. 

We started to experience these limitations at scale. Without a consistent way to verify phone numbers or email addresses, we sometimes sent texts to landlines or emails to undeliverable addresses — leading some messages to not reach their intended recipients and affecting our overall deliverability. We also didn’t have a clear feedback loop in place when notifications didn’t work as designed, which meant our team was reacting to reports rather than proactively resolving issues. 

Third-party rate limits introduced unpredictable delays during high-volume periods, reducing throughput and creating backlogs in our notification queues. This led to delayed or duplicate sends for some notifications, and scheduled reminder notifications (e.g., appointment reminders) sometimes failed silently. Additionally, because notifications were implemented synchronously within user flows, notification delay propagated upstream, causing timeouts in critical user flows (such as appointment booking) and degrading the client experience.

We implemented incremental fixes along the way, but it became clear that a patchwork approach was only making things more fragile and harder to maintain. We realized we needed to treat notifications as a first-class, production-grade system — one with centralized logic, reliable delivery, and observability — so we can catch problems before clients and providers ever notice them.

Designing our notification engine

We built an engine that unified all notification channels behind a single dispatch surface. As a result, product teams simply need to make an API call to our engine with the appropriate parameters, and it will handle the underlying complexities of routing, vendor APIs, verification, and compliance. Not only did this simplify the developer experience, but it also made our engine more configurable, making it easy to switch vendors or add new features, such as localization, without requiring teams to rewrite their code. 

One key architectural decision we had to make was the execution model. Early on, we decided to prioritize flexibility and leave it up to the individual product team. This decision led to inconsistencies, as some synchronous notification sends caused latency and timeouts in critical flows. To resolve this issue, we decided to enforce async sends across the board, decoupling notification delivery from product flows. 

A flowchart that shows the lifecycle of a notification through our notification engine. 

How teams use the engine

Developers can programmatically call the engine via an API call and pass in three parameters:

• Notification name: A simple identifier (like appointment_reminder) that maps to the correct configuration in each channel's third-party system. 

• Destination: The recipient's contact information for the relevant channel, like their phone number for an SMS message. 

• Custom variables: Any dynamic values needed to populate the notification template, like the client’s name or appointment time. 

Each API call enqueues an async Celery job that handles the full send pipeline: validating the destination, rendering the template with custom variables, and dispatching the notification to the appropriate third-party vendor (including retries for failures). Push notifications go through an additional encryption step: content is encrypted before it leaves our system and can only be decrypted on the app side using a derived key, helping protect sensitive health data and keep the pipeline HIPAA-aligned.

Of these steps, validation is the most involved because it occurs in layers. For email, we first check the syntax, ensure it isn’t on any blocklist, and then use a third-party vendor to confirm deliverability before using it. Similarly, for phone numbers, we validate the syntax using E.164 normalization, confirm they aren’t on any blocklist, and then use another vendor to verify their validity. We cache all results to avoid repeated validation on subsequent sends.

Hardening the engine

The core architecture provided a solid foundation for us to build on. But to truly deliver a reliable and smooth experience at scale, we needed to go beyond this and harden the system — ensuring that no message slipped through the cracks, regardless of carrier, channel, or volume. Here's what that looked like in practice.

Protect critical paths under load

Not all notifications carry the same level of time-sensitivity and urgency. For instance, a one-hour appointment reminder demands a different urgency than a "fill out your intake forms" nudge for an appointment next week. If clients miss time-sensitive notifications, they may miss critical information or appointments, which negatively affects their care. 

To address this, we implemented a three-tier queue system to prioritize messages into high, normal, and low tiers. Time-sensitive sends (like one-hour appointment reminders) go into the high tier, whereas bulk or non-urgent messages route to normal or low tiers. Each queue is processed by a dedicated Celery worker pool, ensuring that spikes in lower-priority traffic (e.g., bulk reminders) don’t starve higher-priority notifications of processing capacity. This isolation allows critical messages to be delivered reliably, even under heavy load.

A visual way to understand how our notification queues work. 

Making retries seamless

We’re leaning on third-party APIs for our notification sends, which have their own rate limits and quirks, which means retries are unavoidable. But without proper safeguards, we can easily end up with duplicate notifications and an inconsistent user experience.

That’s why we introduced idempotency. Anyone invoking a notification send can create their own idempotency key (e.g., appointment_ID-one_hour for a one-hour appointment reminder) to make each notification unique. If no key is provided, the engine will generate a random one. Behind the scenes, idempotency keys are scoped to the notification, recipient, and channel, allowing us to safely retry failed sends. Each successful send is recorded in an immutable NotificationSent log, which acts as a source of truth for what has already been delivered.

To handle transient failures from downstream providers (e.g., rate limits or temporary outages), we pair idempotency with exponential backoff. This approach spreads retries over time, smoothing traffic spikes and giving external systems time to recover, while guaranteeing eventual delivery.

Optimizing notification scheduled sends

To determine whether a client should receive a notification, we initially relied on asynchronous polling jobs that checked which clients were eligible for a particular notification. As we added more notification types and triggers, this approach became increasingly inefficient — we were constantly polling the database to check whether anything needed to be sent, even when nothing had changed. 

To address this, we introduced an event-based scheduling system that triggers notification sends when relevant events occur, such as a canceled appointment or newly assigned intake forms. This approach eliminated unnecessary polling, but as we scaled, we saw a new problem: performance. Sends were processed serially, and any additional API calls to retrieve and format data for individual notifications were made in-line. This slowed down the overall processing throughput, especially as we scaled. Additionally, because Celery has a maximum execution time per task, some jobs were killed mid-flight, resulting in timeouts and missed notifications for our customers. 

We solved this by adopting a bulk processing approach. Reminders are now processed in batches, with all relevant provider and appointment data fetched up front. Each batch is then dispatched as a separate Celery sub-task, allowing multiple workers to process reminders in parallel. We also fanned out sends to avoid overwhelming downstream resources. As a result, our system can now reliably schedule and send notifications, even at scale. 

Surfacing failures 

You can only fix what you can see, which is why we prioritized adding observability metrics and systems to monitor, detect, and respond to failures.  

We implemented custom Datadog metrics to track failed notifications, including raw volume and failure sources (e.g., invalid destination, rate limit exceeded), and built a dashboard that used these metrics to track system performance. We took monitoring a step further by implementing automated Slack alerts that notify relevant product teams if failures spike or specific notifications begin to degrade, giving them the chance to address issues before they affect large numbers of users. To help teams debug quickly when that happens, we added structured logging with high-cardinality data for cleaner Sentry error grouping.

The impact of our work

We’re 2 years into building out our engine, and over that time period, we’ve seen tremendous benefits. 

The biggest one: We created a system that can reliably send notifications to providers and clients.  In the last year alone, we’ve increased our successful sends by 78%, including a 67% decrease in invalid destination errors. Additionally, by making our notification sends async, we’ve decoupled them from user-facing request paths, eliminating timeouts in critical flows like appointment bookings. As a result, we’ve been able to significantly increase throughput, increasing our notifications by 4x over the last two years. 

From a developer experience standpoint, notifications are now far easier to work with. Teams can easily add new notifications through a few API calls, reducing time to launch and improving consistency. They also have visibility into their notification performance and overall system health, allowing them to easily detect and fix issues before they become larger outages. 

Finally, this work laid the foundation for future growth. The system is now flexible enough to support new channels, localization, and evolving product needs without requiring major architectural changes.

—

Notifications are a critical component of a client’s healthcare journey, which means the systems that power them deserve the same care and investment as any other core product surface.  The work isn't always straightforward — third-party vendors introduce constraints outside of our control, and edge cases can compound quickly. But by investing in the right architecture, safeguards, and observability, we turned a fragmented implementation into a reliable foundation for patient communication.

If you’re interested in solving complex infrastructure problems that drive real impact, come work for us. Check out our open roles.