How a concurrency bug broke our push notifications

Sending push notifications to users is a common task in every Android and iOS app, as it is for our app. The notifications are not encrypted for us, meaning our backend just sends clear text to the Firebase Cloud Messaging servers, and at some point, those clear text messages will be delivered to the users’ the mobile phones.

At the time, we weren’t sending sensitive data, and since no one had questioned the implementation for a long time, it seemed fine.

This changed in May 2025. Our data privacy consultant asked us developers, if we encrypt the push notification data. Multiple developers were involved in the group, and obviously, we answered no.

While we defined it as acceptable, even from a data privacy perspective, we all agreed that we actually should and want to encrypt those messages. Just for the sake of correctness. It just feels more right to do so.

Since we decided, we want to tackle this, the ticket was discussed in our backlog grooming meeting. We came up with two different solutions:

1. Encrypting the data we sent
2. Sending only an ID and fetching the message from the backend on demand

Eventually, we decided to go with the encryption solution. Mainly because the core logic has been in place for more than six years. We already encrypt and decrypt vehicle positions. We decided to basically just reuse the existing code and “only” move some classes and files around.

This was also why we estimated it could be implemented relatively quickly. Likely in one sprint, at most two. For us, that meant two to four weeks.

All teams were involved in this topic:

• iOS, written in Swift
• Android, written in Kotlin
• Driver (iOS and Android), written in React Native
• Backend, written in Ruby

The Android team merged the first client-side changes on July 24th, 2025.
Unfortunately, it would take until early February 2026 to finish this story.

Why this took so long, the problems we faced, and how we solved them are all covered in this blog post.

Note: I write this blog from the perspective of an Android developer and owner of this story, since I was the primary driver behind this feature.

The (Android) pull request looked clean. It mostly involved moving classes around, renaming files, and the decryptor moved to a shared library. With that, we were able to decrypt the encrypted vehicle positions as well as push notifications.

We had tests in place, and two other developers checked not only the code but also ran an end-to-end test. Our QA Engineer tested it as well. Everything seemed to be working. And honestly, why shouldn’t it? We’ve had the same piece of code in production for more than half a decade.

We shipped it.

Thanks to staged rollouts, 30% of our lucky users benefited from…

AEADBadTagException
if this Cipher object is decrypting in an AEAD mode (such as GCM/CCM), 
and the received authentication tag does not match the calculated value

So we got crashes. At first glance, they looked random. We didn’t know where they were coming from and why. Since none of us are cryptography experts, we first had to dig in and understand what was actually going on.

Sidekick: We are using AES/GCM/NoPadding for encryption. This involves an initialization vector (iv), a tag, the encrypted message, and a decryption key. Without going into details on how it works, you need all of these building blocks to decrypt the message.

We immediately stopped the rollout and got in touch with our backend team. Our crash reporting didn’t provide a clear picture of what was wrong or which specific users were affected. The only thing we found was that, for some reason, the tag was incorrect. So it sounded like a backend bug?

For more aggressive testing, we got a huge list of real-world data from the backend team. It covered many different scenarios: different users, thus different encryption keys, different messages, and so on. We tested each and every item in that list. All of them worked. So, not a backend bug?

We decided to disable this feature on the backend in production, keep it running in our staging environment, and tell the other teams (iOS & Driver) to halt the implementation until we find what was wrong.

Date: August 6th.

What sounded like a good solution at first was, in retrospect, not the best idea. Because we, the Android team, developed the feature in a non-backward-compatible way. Our code is expected to receive a push notification that contains a iv and a tag. If those were not present, we omitted the message, thus not posting the notification to the phone’s status bar.

Since we didn’t think about this case at the time, and because our staging system still sent encrypted notifications, we thought everything was fine so far. User got their unencrypted notifications again, we increased the release to 100%, and we could focus on solving the bug.

But this was not correct. Disabling this feature basically disabled all push notifications for users who updated to the latest version.

To find out more about this issue, we implemented a fallback service. This was basically the second solution we discussed, but decided not to go with for several reasons.

This is how it works in a nutshell:

• We receive an encrypted notification
• We try to decrypt it
• If it fails, we read the notification ID
• We use the ID and do an API call to receive the message
• We post the notification to the user

Since the backend disabled the original feature, the previously used single flag supports_encrypted_notifications was ignored.

From now on, two flags (supports_encrypted_notifications and supports_fetching_notifications) need to be sent to inform the backend to send encrypted notifications.

Together with a bunch of client-side logs, we shipped the new version.

Date: August 21st.

At this point, thanks to the working fallback service, our users received encrypted notifications again. With the fallback service kicking in if decryption failed.

So far, so good. Now we could relax a bit, continue observing the problem, and calmly investigate to find the root cause.

Unfortunately, the initial logs we added didn’t help at all. We needed another release to add more meaningful logs.

Date: October 16th.

Two weeks later, on October 29th, we finally found the root cause of this problem. To explain this, I have to go a bit deeper into the world of Kotlin. But no worries, I still try to keep it as simple as possible.

The Firebase Cloud Messaging Receiver
To receive messages via FCM, you have to register a single FirebaseMessagingService in your application.

Since it is only possible to declare one instance of this service class, but for various reasons we need “more than that”, we introduced the concept of FirebaseCloudMessagingReceivers. It’s just an interface that can be implemented to handle incoming notifications. The single FirebaseMessagingService will then broadcast the incoming message to those Receivers.

This is the single FirebaseMessagingService:

internal class FcmMessagingService : FirebaseMessagingService() {

    private val messagingReceivers get() = FcmMessagingReceiverComponent.instance.fcmMessagingReceivers()

    override fun onMessageReceived(message: RemoteMessage) = 
        messagingReceivers.forEach { it.onMessageReceived(message) }
}

These are our FireabseCloudMessagingReceivers:

public interface FirebaseCloudMessagingReceiver {
    public fun onMessageReceived(message: RemoteMessage)
}

public class MyReceiver : FirebaseCloudMessagingReceiver {
    public fun onMessageReceived(message: RemoteMessage) {
        // Decrypt the message and push to the status bar
    }
}

In our pull request, we added Kotlin Coroutine support for these. This was mainly driven by the fact that the decryption part requires the decryption key, which might be fetched on demand from our backend.

The changes looked like this:

internal class FcmMessagingService : FirebaseMessagingService() {

    private val scope = CoroutineScope(Dispatchers.Default + SupervisorJob())

    private val messagingReceivers get() = FcmMessagingReceiverComponent.instance.fcmMessagingReceivers()

    override fun onMessageReceived(message: RemoteMessage) = messagingReceivers.forEach {
        scope.launch { it.onMessageReceived(message) }
    }

    override fun onDestroy() {
        scope.cancel()
        super.onDestroy()
    }
}

Our FirebaseCloudMessagingReceiver simply added the suspend keyword to be able to run jobs concurrently. Our receiver, which can decrypt incoming messages now, looks like this:

public class MyReceiver(
    private val decrypter: Decrypter
) : FirebaseCloudMessagingReceiver {
    public suspend fun onMessageReceived(message: RemoteMessage) {
        // Derypt the message and push to the status bar
    }
}

The Decrypter Class
Our class that decrypts the messages looked like this:

internal class Decrypter {

    val cipher = Cipher.getInstance("AES/GCM/NoPadding")

    override fun invoke(iv: String, tag: String, encryptionKey: String, encrypted: String): String {
        cipher.init(
            Cipher.DECRYPT_MODE,
            makeKey(encryptionKey),
            GCMParameterSpec(TAG_LENGTH_BYTES, iv.base64Decoded),
        )
        val encryptedMessage = cipher.doFinal(encrypted.base64Decoded + tag.base64Decoded)
        return String(encryptedMessage)
    }

To explain all the code above in words, the following happens:

• The FirebaseCloudMessagingService is called by Android
• Our service creates a CoroutineScope and dispatches the receiving message to each of our FirebaseCloudMessagingReceiver
• One of our receivers uses the Decrypter as a parameter to decrypt the incoming message

Sounds solid, doesn’t it?

If you look closely, and are familiar with Kotlin (and read the blog carefully 😉), you might have already spotted the bug.

Revealing the bug
Imagine the FirebaseCloudMessagingService.onMessageReceived method is called twice by Android, for example, because the first message was delayed by the system.

We then call our receiver twice in two different coroutines, which ends up running in two different threads!

In this case, our single receiver, using the same instance of theDecrypter, calls theDecrypter.invoke() twice from different threads.

The Decrypter, however, holds a state of the Cipher! And Cipher is not thread-safe. We are calling two methods on the Cipher init and doFinal.

If the “first thread” calls init with some data, and the “second thread” overrides that data by calling init with obviously different data, we get a crash or exception if the “first thread” calls doFinal. At that point, the Cipher holds the init-data from the “second thread”, which no longer matches the “first thread’s” init-data.

A classical concurrency, aka race condition, bug 🐛!

One solution to fix this bug is to make our Decrypter stateless by moving the Cipher.getInstant() call into the Decrypter.invoke() function.

With this approach, each call — even from two different coroutines or threads — creates a new Cipher instance. Eliminating any shared state.

The Android bug was fixed on October 29th.
We pushed it to production a day later.

Why did it still take until February to finally close the original story?
Initially, we went with a single flag, supports_encrypted_notifications. As soon as a client sent this, the backend knew that it could encrypt the messages for this specific user.

When the Android bug appeared, the other teams (iOS and Driver) halted their implementation because at this point, it was unclear whether the issue was on the backend or Android side.

After we found out that it was an Android bug and finally fixed it, we unfortunately forgot to tell the other teams that it had been resolved, and they could continue working on that feature. For four or five weeks, nothing actually happened.

But this was not the only reason for the delay.

When the Android team introduced the fallback service, the backend added a second flag supports_fetching_notifications. The backend would send encrypted notifications only when both flags were available; otherwise, notifications stayed unencrypted.

However, this flag was only meant as a band-aid to help find the bug. Nobody wanted it in the codebase unless it was truly necessary. As it turned out, it wasn’t!

But the backend couldn’t simply remove it and rely only on the supports_encrypted_notifications flag, because there might still be Android clients running outdated versions that send only this single flag (and not supports_fetching_notifications). Those clients should still receive notifications in clear text, unencrypted!

Remember what I wrote earlier: Those users wouldn’t receive notifications at all. The Android client ignores those notifications because it requires certain properties in the payload. But encrypted ones might crash the app.
So what is worse?

Luckily, we have a force-update behaviour in place. As soon as a user opens the app and their current version is lower than the required one, we show an “update required” screen and don’t let them use the app anymore. We decided that we wanted to make use of that. But not “today”. First, we wanted to give all users a bit of time to update on their own, since the force-update behaviour isn’t the greatest for the user experience.

As it was already early December and the Christmas season was approaching, we decided to postpone this to mid-January 2026.

Around this time, we gave users the chance to either update manually or, if they were lazy, be forced to update the App. They had about two weeks until early February. At that point, the backend was planned to switch back to the original implementation, ignoring supports_fetching_notifications and relying solely on supports_encrypted_notifications.

The other teams, iOS and Driver, finished their implementations of the feature as originally planned, using a single flag, in December. Even if they released an update earlier, users wouldn’t receive encrypted notifications. However, the clients already sent the single flag, and once the backend logic was flipped, all of them would receive encrypted notifications.