mantra-kmp/docs/long-running-sync.md

A long-running chat sync

Today every chat sync in this app is a pull: a screen queues a request row, a pump drains it, the relay answers, the subscription is closed. Nothing arrives between pulls. This document works out what it takes to keep the chat subscriptions open for as long as the app is active, so relays push new gift wraps and group events at us instead of us asking, and what has to change in the existing pipeline before that is even possible.

The target is the two kinds that carry conversation:

• kind 1059, NIP-59 gift wraps p-tagged to us — DMs, and the Marmot Welcome events that make us a member of a new group.
• kind 445, Marmot group events h-tagged with a group id we belong to.

And one hard requirement: the group subscription must widen the moment we join or create a group, without anyone remembering to call a "subscribe" function.

What happens today

Three pumps in SynchronizationViewModel drain three single-row queues — broadcast, REQ, negentropy — each row observed with ... ORDER BY createdAt ASC LIMIT 1. Screens are producers. Nothing else opens a socket.

screen	queues
ChatRoomListViewModel.scheduleSynchronization	negentropy over 1059 #p=[me] limit 50, then negentropy over 445 #h=[every group id], once per DM relay
ChatMessageListViewModel.scheduleSynchronization	for an MLS room: negentropy over 445 #h=[this group]. For a NIP-17 room: gift wraps p-tagged to the peer and authored by us, or the peer's kind-10050 if we lack it

A negentropy exchange that finds ids we are missing turns them into SynchronizeNostrEventRequest rows, which the REQ pump then fetches in chunks of 500 ids. Both subscription types end the same way: at EOSE the flow completes, the finally sends CLOSE, the semaphore permit is released.

That design is good at what it is for — reconciling a set we already have against a set the relay has. It is structurally incapable of delivering a message that arrives one second after EOSE.

Why we cannot simply stop closing

Five properties of the current pipeline each have to be undone deliberately. None of them is wrong; all of them assume a subscription is short.

1. EOSE ends the flow. completeOnSubscriptionEnd is what makes a collector finish at all — the socket's incomingMessages is hot and a filtered view of it never completes on its own. For a live subscription, EOSE is not the end, it is the boundary between "stored history" and "live tail", and CLOSED is the only genuinely terminal message.

2. Every subscription is force-killed at 120 seconds. SUBSCRIPTION_TIMEOUT exists so a relay that goes quiet cannot park a slot forever. A live subscription is exactly the case it is designed to kill.

3. There are four subscription slots, total, across all relays. subscriptionSlots is a Semaphore(4). A permanently-open subscription is a permanently-held permit. Two live subscriptions would leave the backfill queue with two slots and a permanent halving of throughput. Live subscriptions must not draw from this semaphore — they need their own budget, and the semaphore's comment ("well under the ~20-per-connection cap relays commonly enforce") needs re-reading as a combined budget rather than a queue-only one.

4. The queue row is the subscription identity and the provenance record. The subId is the row's UUID, and both saveNostrEvent overloads (DatabaseNostrRepository.kt:483, :515) take a request object, use its relayURL/level, and flip its status to "processed". A live subscription has no row and never finishes. It needs a third entry point that goes straight to NostrDao.storeNostrEvent with an explicit relayURL and level = 0.

5. Nothing reconnects. NostrSocketClientImpl catches a socket failure, calls close(), and fires onSocketConnectionClosed, which only flips a boolean in relayPoolStatus. There is no retry loop anywhere. Today that is invisible: every subscription is short, and the next queued request re-opens the socket through ensureSocketConnectionOrThrow. A live subscription that outlives a socket drop is a subscription that has silently stopped, on a socket whose status flag says so and which nobody reads.

There is a sixth, quieter problem. _incomingMessages is a MutableSharedFlow() with no replay and no buffer, and each frame is emitted from its own launch (processIncomingMessage). A message emitted while nothing is collecting is dropped, and messages can be delivered out of order relative to the wire. For request/response that is harmless — the collector is attached before the REQ goes out. For a firehose whose consumer does MLS decryption and SQLite writes, it is not: give the router a buffered channel and process in arrival order per relay.

What Wisp does

Wisp (/home/sigidli/Documents/development/nostr/wisp) keeps its subscriptions open for the life of the foreground app. Five mechanisms are worth copying:

• Stable, meaningful subscription ids. "dms", "notif", "grp-<id>". Not UUIDs. Re-sending a REQ with the same id replaces the filter server-side, which makes "widen the filter" a send rather than a close-and-reopen.
• The REQ text is retained per relay. activeSubscriptions: Map<relayUrl, Map<subId, reqJson>>, and resyncSubscriptions(relay) re-sends all of them when that relay reconnects (relay/RelayPool.kt).
• A per-relay subscription budget. SubscriptionTracker enforces a soft cap of 20, with a priority prefix list ("dms", "notif", "grp-", …) that bypasses the cap so a transient feed subscription can never crowd out DMs.
• Explicit reconnect policy. RelayLifecycleManager observes connectivity and app pause/resume, debounces, and drives reconnectAll(); on reconnect only the long-lived prefixes are retained and everything transient is dropped.
• One router, not one collector per request. viewmodel/EventRouter.kt dispatches by subscription id.

And one detail that is a direct warning to us: Wisp deliberately does not put a since on its kind-1059 filter, because NIP-59 randomizes created_at into the past.

The shape to build

A live tier beside the queue, not inside it

A LiveSubscriptionManager owned at the same level as SynchronizationViewModel (the application scope, not a screen), holding one piece of state:

// what we want to be subscribed to, per relay
data class LiveSubscription(val subId: String, val relayUrl: String, val filters: List<Filter>)

and doing one thing: reconcile desired state against what has actually been sent to each socket. Recompute the desired set, diff it against the sent set, send a REQ for anything added or changed, send CLOSE for anything removed. Every trigger — a new group, a relay reconnect, an app resume, a relay-list change — becomes a call to the same reconcile(). That is the property that makes this maintainable: there is no "subscribe" path and "resubscribe" path that can drift.

The sent-set belongs in RelayPool, next to socketClients, because that is where a reconnect is observed. Mirror Wisp: keep the serialized REQ per (relayUrl, subId) and re-send on connect.

The two filters, and their timestamps

// live-giftwrap
Filter(kinds = listOf(GiftWrapEvent.KIND), tags = mapOf("p" to listOf(myPubkey)))

// live-groups-<n>
Filter(kinds = listOf(GroupEvent.KIND), tags = mapOf("h" to groupIdChunk))

The two kinds need opposite treatment on since, and this is not a matter of taste — it is visible in our own outbound code.

Gift wraps are stamped with TimeUtils.randomWithTwoDays(), so a wrap published now can carry a created_at up to two days in the past. A live filter with since = now would silently drop a large fraction of genuinely new messages, and the failure mode is "some DMs just don't arrive" — the worst kind to debug. Either omit since entirely (gift wraps are low volume and storeNostrEvent already no-ops on a known id) or floor it at now - 2 days and accept the redelivery.

Group events use TimeUtils.now(). A watermark is safe there: since = (newest 445 we hold for these groups) - slack. Keep the slack generous (minutes, not seconds) — relay clocks drift, and the cost of re-receiving a handful of 445s is a few no-op storeNostrEvent calls.

#h has to be chunked. One subscription per group is the simple option and what Wisp does for NIP-29, but Marmot groups here are all on the same DM relay set, so one subscription carrying N group ids is far cheaper in sockets and slots. Chunk at a conservative width (~100 ids) across live-groups-0, live-groups-1, … and accept that a chunk boundary shift re-sends one REQ.

Routing

One collector per socket, attached for the life of the socket, dispatching on subscription id prefix — the EventRouter shape. That collector is also what fixes problem six above: it is always attached, so nothing is dropped for want of a subscriber, and it can impose ordering and back-pressure in one place.

Events off a live subscription go to the new request-free saveNostrEvent, and from there through the same storeNostrEvent → MarmotInboundManager path as everything else. No inbound processing changes. A 1059 carrying a Welcome still lands in the branch at NostrDao.kt:720; a 445 still reaches processGroupEvent. Only the delivery mechanism is new.

Keeping the group filter current

This is the requirement that decides the design, and it is the easy part — provided we derive the filter from the database instead of from the join sites.

chatRepository.observeChatRoomListByPublicKey(publicKey)
    .map { rooms -> rooms.filter { it.chatRoom.mlsGroupState != null && it.chatRoom.leftGroupAt == null }
                         .map { it.chatRoom.id }.sorted() }
    .distinctUntilChanged()
    .debounce(500.milliseconds)
    .collect { groupIds -> live.setGroupIds(groupIds); live.reconcile() }

ChatRoomListViewModel already collects this exact Flow to render the list. The difference is where it is collected: at application scope, so it keeps running when no chat screen is open.

Driving it off the DB rather than off the join sites matters because there are several ways a group id appears, and they are not all in view models:

• we create a group (MarmotOutboundDao),
• we are added to one — the Welcome arrives inside a gift wrap and is processed deep inside NostrDao.storeNostrEvent, on the inbound path,
• membership shifts under us via a commit (MarmotInboundManager.processGroupMembershipChanges),
• and we leave one (leftGroupAt).

Only the first is a place a developer would naturally think to call subscribeToGroup(). Observing the room list catches all four, and it closes the loop cleanly: a Welcome arrives on the live gift wrap subscription → a ChatRoom row is written → the Flow re-emits → the group filter widens → the first message in that new group arrives without anyone opening the chat.

Debounce is not optional. Joining a group writes the room, its participants, and placeholder profiles in quick succession; without it we would re-send the group REQ several times per join.

Lifecycle

Mantra has no app lifecycle observer at all today — MainActivity only calls setContent. Long-running subscriptions make one necessary:

• background: stop the live subscriptions (CLOSE, or just drop the sockets). Holding a socket open behind a doze window achieves nothing except battery.
• foreground: reconnect and reconcile. Assume the server-side subscription is gone — Wisp's forceReconnectAll exists precisely because a socket that survives an OS sleep reports isConnected == true while being functionally dead — and queue a bounded catch-up negentropy pass for the gap.
• connectivity change: reconnect, debounced, with the suppression window Wisp uses to avoid a resume and a network-change event double-reconnecting.

The foreground catch-up is what preserves correctness across the gap. Live subscriptions cover the online window; negentropy covers everything else.

What the screens stop doing, and what they keep

Once the live tier exists:

• ChatRoomListViewModel.scheduleSynchronization — drop both queues on open. Keep the code path behind an explicit pull-to-refresh.
• ChatMessageListViewModel.scheduleSynchronization — drop the mlsMessages branch entirely; it is a strict subset of live-groups-*. Keep the ChatMessageRelayListEvent lookup: that is discovery (which relay does this peer read from), not message sync, and it still has to happen on open.
• Everything not chat — profiles, follows, feeds, key packages — is untouched.

One thing to fix while in here, since it is one of the per-chat syncs we are proposing to stop firing: the "sent-messages" filter (kinds=[1059], authors=[me], #p=[peer], at ChatMessageListViewModel.kt:113 and NostrDao.kt:643) cannot match anything. Gift wraps are signed with a fresh throwaway key (val wrapperKeyPair = KeyPair()), so authors = [myPubkey] matches no wrap this app has ever published. It also turns out to be unnecessary: createNip17ChatRoom puts the user in their own participant list (NostrNip17Dao.kt:50), so we wrap a copy to ourselves and the account-wide #p=[me] subscription picks our own sent messages up on every device. Delete the filter rather than port it.

Negentropy's remaining job

Live subscriptions replace polling, not reconciliation. Negentropy is still the right tool for:

• first login / restore, where the local set is empty,
• the foreground catch-up after a background gap,
• "load older messages" in a chat,
• and repairing a relay that was unreachable while we were online.

The distinction to hold onto: a live REQ answers "what is new since I connected"; negentropy answers "what do you have that I don't". Neither subsumes the other, and the queue and its three pumps stay exactly as they are.

Traps, in the order they will bite

1. Reusing the subscriptionSlots semaphore. Two permanent permits out of four halves backfill throughput forever. Give the live tier its own budget and read the two together against the relay's real cap.
2. completeOnSubscriptionEnd on a live flow. It will complete at the first EOSE and the subscription will look established while delivering nothing. The live variant must terminate on CLOSED only.
3. since = now on kind 1059. Randomized wrap timestamps mean silent, partial message loss. See above.
4. Treating CLOSED as "retry now". isBackPressure already exists in SynchronizationViewModel and should be reused: a rate-limited CLOSED on a live subscription needs exponential backoff, an auth-required CLOSED needs the AUTH flow (which the app does not implement yet — worth knowing before pointing this at a relay that requires it), and only an unsupported-filter CLOSED should fall back to a queued REQ.
5. A NOTICE has no subscription id. It is already admitted to every collector by filterBySubscriptionId; with a permanent collector attached, every notice reaches it forever. Handle it once in the router, not per subscription.
6. Duplicate delivery. storeNostrEvent no-ops on a known id, so duplicate 1059s are cheap. Duplicate 445s are not automatically cheap — MarmotInboundManager maintains epoch state and a commit tracker; confirm the re-delivery path is idempotent before widening the since slack.
7. Relay list changes. RelaysSocketManager.updateRelayPools calls relayPool.changeRelays, which closes sockets. Live subscriptions must be reconciled after that, or they quietly vanish on any relay edit.
8. The DM relay set is one relay. Relays.DefaultDMRelayList is [wss://ephemeral.mantra.press]. Every point above about per-relay budgets is currently a budget of one socket — which makes this cheap to build and makes a single relay outage total. Worth widening the set at the same time.

Suggested order

1. Reconnect first. A supervised reconnect loop in NostrSocketClientImpl plus per-relay REQ retention and resync in RelayPool. This is a strict improvement on its own, before any live subscription exists.
2. The router. One permanent collector per socket, dispatching by subId prefix; the request-free saveNostrEvent. Existing pumps keep working.
3. live-giftwrap. One subscription, one relay, no since. Verifiable by sending yourself a DM from another client with the chat list closed.
4. live-groups-*, driven off the room-list Flow. Verify by having a second device add this one to a group and watching messages land with no chat screen ever opened.
5. Lifecycle + foreground catch-up.
6. Then, and only then, delete scheduleSynchronization from the two chat view models.