Files
giza-kmp/docs/marmot-v2.md
T
Kgothatso NgakoandClaude Opus 5.5 b75d4b427f Rebrand Curare to Bennu under com.machankura.bennu
Move the packages to com.machankura.bennu.{compose,android,desktop}, the
Android applicationId to com.machankura.bennu.android and the iOS bundle
id to com.machankura.bennu.ios. The root project, launcher label, iOS
product, desktop window, user-facing strings, client tags and the
Curare* classes all say Bennu now.

Left alone on purpose: the mantra/ derivation tags, the `curated` desktop
data directory, the mantra.press relay and the curare.to references.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 13:53:45 +02:00

59 KiB

Marmot v2: bringing the group layer onto the current profile

quartz 1.15.2 implements the restructured Marmot protocol -- what its own comments call the current profile, against the legacy MIP-era profile this app speaks -- and bumping the version gives the app almost none of it. The pipeline that decides how a group event is read, how a fork is settled, when a commit is published and where the state is persisted is the app's own, not the library's, and the bump left every one of those decisions exactly where it was. This is what would have to change for the app to behave like a v2 client, in the order it can safely change, and the two places where v2 and this app's own design genuinely disagree.

Read marmot-membership.md and shared-key-derivation.md first. The first describes the invite race the current pipeline has, which v2's publish rule is the real fix for. The second explains why a room's id is a key -- and that is the thing v2's routing rule is about to collide with.

Built, phases 0-8, one commit each and three follow-ups where a phase's commit had left a line of its own phase unbuilt; Phase 9 is the rollout and is process rather than code, and where it stands is noted at its end. The phases are kept as written because they are the reasoning, and the code reads better against the argument it came from than against a summary of itself. It is deliberately ordered so that the phases that buy interoperability with other v2 clients come before the phases that make the app conform -- a member can talk to White Noise before its forks settle the way White Noise's do. Where the implementation chose differently the table below says so, and the first of its rows is the one to read before any of the code:

what the plan said what it turned out to be
Phase 5: the library's engine replaces the app's pipeline -- MarmotInboundManager shrinks to the membership helpers, MlsGroupCache and the vendored CommitOrdering are deleted a room's profile picks its pipeline. A legacy room keeps the app's pipeline exactly as it was, comparator and cache included; a current-profile room goes through MarmotEngine, and NostrDao.indexMarmotGroupEvent branches on ChatRoom.marmotProfile(). The library's processor has no bare-wrap carve-out, so under it every legacy direct message would have stopped decrypting, and the legacy pipeline is frozen anyway: it retires with the profile, not before
Phase 1: the accessors, "in a small extension file beside MlsGroupCache" MlsGroupProfile.kt, and six of them rather than three, because Phase 4 made the room's id a thing a group has to say about itself: roomId() reads the "Room id:" line MarmotRoomIdentity puts in the profile's description, then 0x8004, then the MLS group id; routingId() is the address without the first step
Phase 2: the kind guard in mip03Rejection "goes", and MarmotMip03CarveOutTest inverts into the test that it is gone it stays, for legacy rooms: mip03Rejection(..., legacyRoom). A current-profile room refuses the bare wrap as a conformant peer would; a legacy room admits it, because every build in one still sends it. The test holds both halves
Phase 2: the envelope's kind is "the next free number above 30329" 30330, MarmotDirectMessage.ENVELOPE_KIND; wrap is unchanged and envelope goes around it, wrapInside takes it off
Phase 2: bring MAX_PAGE_BYTES to 48 KiB and re-size MAX_PAGE_EVENTS 48 KiB and 96 -- measured escaped. A page's JSON travels as a string inside the payload's content, so the bound that matters is ChronicleEvent.escapedByteSize, and a dense page that measured 48 KiB raw escaped to 70 KiB
Phase 3: createKeyPackage(signer, lastResort = true) not last-resort. The notary republishes a fresh package for whichever profile is left without one when a Welcome consumes a bundle; a last-resort leaf is for a client that cannot
Phase 3: run isValid and isCryptographicallyValid at ingest the second alone, which runs the first
Phase 3: schema 20 -> 21 20 -> 23: the two profile columns (Phase 3), MarmotRouting (Phase 4), MarmotPublishObligation and MarmotOutboundGate (Phase 6), each an AutoMigration
Phase 4: the two creation sites call CurrentProfileGroupFactory.createGroup directly MarmotGroupFactory.create, one wrapper for both profiles and three sites -- the plan missed createMlsDirectMessageChatRoom, which makes the room behind a direct message. On the current profile it mints the routing id, and the description carries the derivation sentence and a "Room id: " line, because a member joining from a Welcome has nothing but the group's own state to learn the room's id from
Phase 4: "extend Outcome.BlockedOn to a package that cannot be used" MarmotKeyPackageProfileException at the add step, whose message the confirmation screen shows. BlockedOn still means missing, because the picker -- getMarmotKeyPackageForPublicKey(publicKey, roomProfile) -- refuses the wrong profile before creation ever asks, so there a wrong-profile package is a missing one
Phase 5: four stores one, RoomMlsGroupStateStore, over the state blob, MarmotRetainedEpochSecret and MarmotRouting. No KeyPackageBundleStore and no snapshot table: the KeyPackageRotationManager is constructed because the processor's constructor demands one and is never consulted, and bundles stay the app's rows with the notary's per-profile republish over them. No MarmotIngestDedupStore: 1.15.2's processor takes no such store, so there is nothing to mark into
Phase 5: MarmotGroupEvent gains a marmotMessageId column not added. The processor dedups by message id in memory for the life of the process; the reindex sweep still dedups by event id. The follow-up if a relay ever redelivers under a second id
Phase 5: one engine "held by the identity's scope" BennuDatabase.marmotEngine, one per database, restored on first use. The DAOs that need it hold the database and not a view model's scope, and a switch of identity is a restart
Phase 5: the timer "beside LiveSubscriptionManager"; an outbound row "stays on the queue" while a pass is open MarmotEngine.settleLater, armed by the ingest whenever a pass opens and ticking on the engine's own scope; and awaitSettled on the outbound side -- a wait of up to three ticks rather than a refusal, because the outbound queue re-emits on a table change and a pass settling is not one
Phase 6: "a terminal rejection resolves FAILED" built in a follow-up, once "terminal" had a definition: a NIP-01 OK reason a relay will keep giving (blocked:, invalid:, pow:, restricted:) is a new "rejected" status that the launch-time requeue leaves alone, rate-limited: and error: are "failed" like a timeout, and duplicate: is "published". The obligation goes only when every relay in scope has rejected -- BroadcastStatus, MarmotEngine.discardRejected -- and the room gets a line in the relay's words
Phase 6: gate.satisfyEmptyObligation for the group that is only its creator installState straight after stageAddMember, keyed off the isOneMemberInitialGroupCreation the app already computed
Phase 7: leaving is leaveGroup behind a LEAVING gate; an admin commits the proposal "on receipt, or behind a confirmation" on receipt (commitStagedProposals from ProposalStaged), and an admin is refused before anything is raised or dropped: the library refuses an admin's SelfRemove, so the app says "hand over admin first", or, for the only admin, "make another member an admin, or disband the group"
Phase 7: captions from MarmotGroupSnapshot against a marmotSnapshot column from the diff processGroupMembershipChanges already made against the participants table, on every device that applies a commit, and without a TYPE_MEMBER_REMOVED: the tree does not say whether a member left or was removed, so TYPE_MEMBER_LEFT covers both. No snapshot column
Phase 7: rename is stageAppDataUpdate and "the button is small" not built. There is no rename for either profile today; the subject on the detail screen is display only
Phase 7: a received 1210 or 1009 gets no bubble, "and neither does any other unknown kind" those two only, in a follow-up. The bubble stays for the rest: the kind:1 arm reaches it on purpose, and it is the only trace an unrecognised payload leaves
Phase 8: four scenarios in one test spread over four: CurrentProfileInteropJvmTest (the join, both directions, envelope included), GroupLifecycleJvmTest.roomWithMember (the app's room, the stand-in joining, routing id not room id), ConvergenceForkJvmTest (the fork) and PublishBeforeApplyJvmTest (the restart). The wire-vector test is MarmotAppPayloadTest and MarmotPayloadBoundsTest in commonTest
Phase 8: the fork -- "whichever branch won, they agree" proven for the case the app depends on and pinned for the case it cannot yet have. Two other members fork; the app and a second observer are handed both commits in opposite orders and settle on the same state byte for byte. A fork this device authored does not: see below

One thing found on the way that is the library's and not this app's. quartz 1.15.2's convergence rebuilds each candidate branch by replaying its commits against the retained parent state, and a member cannot replay a commit it authored -- processFramedCommit on its own commit fails with "UpdatePath at common ancestor carries no ciphertext for us", since a committer holds no key for its own copath. So a fork this device is half of is scored with its own branch stale, and two committers swap branches rather than agree; every device that only observed still agrees, which the first fork test holds. The second fork test pins the swap exactly as observed and says in its last assertion what to do the day it fails: the state the missing replay would produce is already retained, and using it is the fix. Until then the practical exposure is small -- a device that never applies before publishing rarely authors one side of a fork -- and the follow-up in Phase 5 (marking a losing branch's rows withdrawn) waits on the same fix.

Written against the code as it stood at 2bbe1fc9.

Where "v2" comes from, and what it asks

The Marmot spec was reorganised from a list of MIPs into four layers -- foundation/, protocol-core/, transports/, app-components/ -- and quartz 1.15.2 mirrors that under marmot/foundation, marmot/protocolCore, marmot/appComponents and marmot/mls/components. Groups on the old layout are what the library calls legacy; groups on the new one are the current profile. The reference engine (MDK, which White Noise is built on) has already cut over: its comments, quoted in quartz, say it "refuses to add a member to a legacy group at all", and it rejects a Welcome into a group that requires neither identity-proof component. Amethyst, on quartz, still reads both. So the set of clients this app can reach is decided by which profile its groups and its KeyPackages are on, and today that is legacy for both.

Nine rules reach this app. Each row says what v2 requires, what the app does now, and the phase that closes the gap.

v2 says the app today phase
an application payload is an unsigned Nostr-shaped JSON object with exactly six members; a decoder rejects any sig member, any unknown member, and any id that does not rehash a NIP-59 rumor serialised with "sig":""; direct messages as a kind:1059 wrap signed by a throwaway key 2
group state is a dictionary of versioned components -- profile 0x8001, admin policy 0x8003, routing 0x8004, retention 0x8005, lifecycle 0x800c -- and every member leaf carries an account identity proof (0x8009) signed by the account's Nostr key one monolithic 0xF2EE extension read in seventeen places; a leaf with no proof, which a v2 group refuses at add time 1 (read), 3 (leaf), 4 (groups)
nostr_group_id is a delivery address: random, and "MUST NOT be derived from any ... public key" the room's id, which is a public key derived from the group's threshold key, doubles as the h tag 4
duplicates are recognised by the Marmot message id -- SHA-256 over the MLS bytes -- never the Nostr event id, because every relay copy has a different one event id 5
competing commits are settled by convergence: retained states, a bounded pass, a six-step deterministic comparison; the created_at-then-event-id rule is gone because both are sender-chosen and unauthenticated that rule, vendored into the app when the library deleted it 5
a local commit is published before it is applied, as a durable obligation that survives restart, and outbound work is held while a group is unsettled apply, persist, then broadcast; the Welcome waits for a relay OK but the state already moved 6
leaving is a SelfRemove proposal an admin commits; a group can be disbanded; membership captions are derived from state, never received a local leftGroupAt flag and a "Left group" chat line 7
KeyPackages go to the NIP-65 write set; kind:10051 is a legacy hint; a Welcome is matched to a bundle by KeyPackageRef, not its e tag DefaultDMRelayList and 10051; bundle by e tag 3
no encoding tag on any event shape one on every application kind:445 2

Two things the bump did do are worth knowing before reading on. MlsGroupState's on-disk blob went from version 2 to version 4 -- 1.15.2 reads both, 1.14.0 refuses v4 -- so every room touched under the new library is unreadable by a build on the old one. And Event.fromJsonOrNull defaults a missing sig to "", so the build already in the field can read a canonical v2 payload; it just cannot produce one. Both facts shape the rollout in Phase 9.

The constraint

Two independent facts, and the phases are shaped around both.

The app does not use the library's pipeline. MarmotInboundManager, MarmotOutboundDao and DatabaseMarmotRepository drive MlsGroup directly, with Room as the store: their own outer-layer decryption, their own retained-epoch window (in memory -- the table is written by the outbound path and never read for a decrypt, so a restart forgets every epoch the inbound path retained), their own commit conflict handling, their own persistence in ChatRoom.mlsGroupState. The inbound manager is a copy of the library's as it stood in July, when the ingestion was written -- DatabaseMarmotRepository says as much in its header -- and every fix the library has made since has had to be carried across by hand, or missed. Everything in the v2 column above lives in MlsGroupManager, MarmotInboundProcessor, MarmotConvergenceEngine and MarmotPublishGate, and those are built around four store interfaces -- MlsGroupStateStore, KeyPackageBundleStore, MarmotIngestDedupStore, MarmotPublishObligationStore -- the app has never implemented.

So the choice is between adopting the library's managers over Room, or re-implementing v2 inside the app's pipeline. This plan adopts. The convergence engine alone is two thousand lines with a spec behind it, and its whole value is determinism: two clients that resolve the same batch must pick the same branch, and a second implementation that disagrees with the first in any corner forks the group in exactly the situation convergence exists to prevent. Adopting also holds live MlsGroup instances in the manager, which is what MlsGroupCache exists to do by hand, and makes retained epochs durable for free. The cost is Phase 5, which is the biggest phase, and the fact that the four stores must be written over tables that were shaped for the old pipeline.

The room's id is a key. SharedKeyDerivation.marmotGroupId derives a room's id from the group's threshold key so that every member can compute it unaided and the room can sign as itself -- the derivation note calls that "one function because they are one key", and the chronicle note builds its entire verification story on it. v2's routing component says the opposite of its nostr_group_id: random, never derived, because a derivable address lets relays link a group's traffic to the identity behind it. Both are right about their own value. The resolution in Phase 4 is that they are two values: the room keeps its derived id as its identity and the key it signs with, and gets a random routing id as the address its kind:445s carry. That costs a table and a lookup on every inbound path, and it is the one design decision in here that cannot be reversed once a v2 room exists.


Phase 0 -- the bump

Done, in a worktree. Half a day.

quartz = "1.15.2" in the version catalogue, and two compile fixes:

  • CommitOrdering was deleted from the library. It is vendored, unchanged, as managers/CommitOrdering.kt with a header saying why the library dropped it. It goes again in Phase 5.
  • GroupEventResult gained AppMessageOnCandidateBranch and RefusedByLifecycle, and the exhaustive when in ChatMessage.fromGroupEventResult now returns null for both. Neither is a delivery, and until Phase 5 the app never constructs either.

Nothing else moved: all 1,128 JVM tests pass, the rewritten Curve25519 arithmetic matches the RFC 8032 and RFC 7748 vectors, and the app's own KeyPackages sit inside the new 84-day lifetime ceiling at 30. The one-way door is the state blob: a room re-persisted under 1.15.2 cannot be opened by 1.14.0 again.


Phase 1 -- read both profiles

Two days. No wire change.

After this phase the app can be a member of a current-profile group and can invite a current-profile peer into one of its own legacy rooms -- which is where interop with Amethyst users starts, since Amethyst still joins legacy groups. Nothing it sends changes.

One accessor per field

Seventeen call sites read currentMarmotData(), which is the 0xF2EE extension and is null on a v2 group. Ten of them read nostrGroupId, and the Welcome path in NostrDao.kt:701 throws MarmotMissingNostrGroupDataExtension when it gets null -- so today a v2 Welcome is not a room the app cannot read but an exception it cannot get past.

The library already reads both profiles: MlsGroup.currentNostrGroupId(), currentAdminIdentities(), currentMemberIdentities(), isLeafAdmin(), and currentGroupState() for the profile, routing, retention and lifecycle components. Write the fallbacks once, in a small extension file beside MlsGroupCache:

fun MlsGroup.roomName(): String? =
    currentGroupState().profile?.name?.ifBlank { null }
        ?: currentMarmotData()?.name?.ifBlank { null }

fun MlsGroup.roomRelays(): List<String> =
    currentGroupState().routing?.relays
        ?: currentMarmotData()?.relays
        ?: emptyList()

fun MlsGroup.retentionSeconds(): Long? =
    currentGroupState().retention?.takeIf { it.isEnabled }?.disappearingMessageSecs?.toLong()
        ?: currentMarmotData()?.disappearingMessageSecs?.toLong()

and point the seventeen sites at those, currentNostrGroupId() and currentAdminIdentities(). The two that matter most are SubgroupManager.kt:123, which decides who may open a subgroup ceremony, and MarmotInboundManager.processGroupMembershipChanges, which stamps adminAt. On a v2 group both currently see no admins at all, so a member who is an admin is shown as none, and a subgroup cannot be started. MarmotInnerEvent.disappearingExpiration gets the same treatment; the library's own outbound processor had precisely this bug -- it read only 0xF2EE and disappearing messages silently did not disappear in v2 groups -- and fixed it the same way.

KeyPackages that arrive framed

A current-profile kind:30443's content is not a bare KeyPackage but an MLSMessage{KEY_PACKAGE} around one. MarmotOutboundDao.inviteMember does MlsKeyPackage.decodeTls(TlsReader(bytes)) on whatever the ingest stored, which reads the framing's version and wire-format words as the KeyPackage's own and fails. KeyPackageUtils.decodeKeyPackage(bytes) sniffs the first four bytes and handles both; use it there.

While in the ingest at NostrDao.kt:429: nothing validates a KeyPackage today. Run KeyPackageUtils.isValid(event) and isCryptographicallyValid(event) before storing one. The second now checks the leaf's lifetime and, for a current-profile package, the identity proof -- which is the check that makes a proof worth carrying at all. A package that fails is not stored, and the invite screen's "no key package" state already says the right thing for that.

Payloads without a signature

In MarmotInboundManager.processPrivateMessage the author check reads Event.fromJsonOrNull(innerJson). That accepts a canonical payload by accident -- the deserialiser defaults sig -- but it also accepts payloads the spec says to refuse: duplicate keys, unknown members, an id that does not rehash. Decode the strict way first and fall back only for the legacy shape:

val author = MarmotAppEvent.decodeOrNull(innerJson)?.pubKey
    ?: Event.fromJsonOrNull(innerJson)?.pubKey

The ApplicationMessage result keeps carrying the JSON as received; every consumer downstream goes through Event.fromJsonOrNull and copes with either shape. The fallback is what lets a room full of old builds keep working through Phase 2, and it is the same fallback the library keeps for its own history.

The Welcome, by reference

The Welcome's e tag names the kind:30443 the inviter thinks it used. MDK stamps the id of the copy it cached, so a peer that rotated its KeyPackage while an inviter kept inviting from cache cannot be joined by the tag alone. RFC 9420 says a joiner takes the first EncryptedGroupSecrets whose new_member ref it holds keys for, and the library now does exactly that before falling back to the tag. Do the same in the Welcome branch: decode the Welcome, walk secrets, and match newMember against MlsKeyPackage.decodeTls(bundle.tlsEncodedMarmotKeyPackage).reference() over the unconsumed rows in MarmotKeyPackageBundle. There are at most a handful of rows, so this is a scan rather than a column, and no schema moves.

Tests. RumorIdAgreementTest in commonTest grows two cases: a canonical payload and a legacy rumor with the same members decode to the same id, and a payload with a sig member is refused by the strict decoder and accepted by the fallback. In jvmTest, MarmotOutboundDaoJvmTest gains an invite of a current-profile KeyPackage -- built with CurrentProfileGroupFactory.createKeyPackage(NostrSignerInternal(KeyPair())) and published through KeyPackageEvent.buildCurrentProfileFrom -- into a legacy room, asserting the member lands. That is the Amethyst-user case, end to end.


Phase 2 -- the canonical payload

Two days.

What goes out

In encryptAndSendMarmotInnerEvent the plaintext is innerEvent.toJson().encodeToByteArray(), which carries "sig":"". The library's outbound processor is explicit about why that is refused by a conformant peer rather than merely untidy: a payload with a real signature would be a valid standalone relay event, so one leaked plaintext could be republished as a signed public statement by its author. The fix is one line --

val plainTextBytes = MarmotAppEvent.fromEvent(innerEvent).encodeToPayload()

-- and the id does not move, because NIP-01 never hashed the signature. Every row already filed under an inner event id stays valid. Drop EncodingTag.assemble() from the kind:445 template in the same function; transports/nostr.md forbids it on every shape and the library's GroupEvent.build no longer adds one.

Old builds read this shape (see the note under the table), and Amethyst reads it, so the switch can be made everywhere at once. It is the only wire change in this plan that needs no gating.

The two limits

A conformant decoder rejects a payload over 64 KiB, deeper than 16 levels, or with 64 or more elements in any one collection -- and the tags array is a collection. The app has payloads near both edges. ChronicleEvent.MAX_PAGE_BYTES is 64 KiB of content, so a page plus its envelope is over; bring it under with room to spare (48 KiB is safe and costs nothing but a few more pages), and re-size MAX_PAGE_EVENTS with it, as the comment on the two caps insists. Any kind that carries one tag per member -- the contributor lists are the ones to audit -- must stay under 64 tags, which is a cap to enforce where those events are built, not a limit to discover from a dropped message. Within this app's own rooms the library's fallback would still read an oversized payload; an MDK peer drops it without a word. Add a commonTest that builds one of every app kind at its maximum and asserts MarmotAppEvent.decode accepts it.

Direct messages get an envelope

A kind:1059 wrap is signed -- by a throwaway key, but signed -- and its pubkey is that key, not the sender's. Under v2 it is refused twice over: for the sig member and for the author mismatch. The carve-out in mip03Rejection that admits it is exactly what other clients do not have, and the direct-message note already says those clients drop these as impersonation.

The wrap does not need to be the payload; it needs to be in one. The new shape is an unsigned app event whose pubkey is the sender's account, whose kind is a new app kind (the next free number in the app's range above 30329), whose single tag is ["p", recipient], and whose content is the wrap's JSON. Read the direct-message note's Why a throwaway key before objecting that the envelope names the sender: the MLS frame already did, and the property the throwaway key buys -- a wrap lifted out of the group is not attributable -- survives, because the envelope is unsigned and so is a claim rather than a proof, and the wrap inside it is still a stock NIP-59 wrap signed by nobody. MarmotDirectMessage.wrap returns the envelope; classify and open take one; the seal-to-leaf binding in classify stays, since it is the wrap's own authentication and nothing above it replaces it. The kind guard in mip03Rejection goes, and MarmotMip03CarveOutTest inverts into the test that it is gone.

This one is gated, because an old build renders the envelope as a raw-JSON bubble. Legacy rooms keep sending the bare wrap until Phase 9 retires it; current-profile rooms send the envelope from the day they exist. That is safe by construction rather than by policy: a current-profile room admits only leaves that carry an identity proof, and only a Phase 3 build can make one. The room's profile is the feature flag, and it is the same flag for every gated change after this.

Tests. MarmotDirectMessageTest round-trips the envelope through MarmotAppEvent.decode and opens the wrap inside; MarmotDirectMessageDeliveryTest asserts the bare wrap is still what a legacy room sends.


Phase 3 -- identity proofs and the current-profile KeyPackage

Three days. One schema change.

After this phase White Noise and Amethyst users can add this app's users to their current-profile groups, and the app can join them.

What a v2 leaf carries

A current-profile leaf has an app_data_dictionary extension holding three components: the list of component ids it supports, an explicitly empty safe_aad list, and the 104-byte account identity proof -- the account's Nostr key signing a kind:450 event (never published) that names this leaf's Ed25519 signature key under this ciphersuite. The proof is why a leaf cannot be forged: MLS never checked that the account named in the credential agreed to the signature key beside it.

The proof needs a NostrSigner, and the order is fixed -- generate the leaf keypair, have the account sign its public half, then build the leaf around both -- because a proof cannot be attached to a finished leaf. Identity yields a KeyPair for every signing kind, so NostrSignerInternal(KeyPair(privKey)) is the signer; a NostrPublic identity has nothing to sign with and could never publish a kind:30443 anyway -- the notary returns before signing anything for it, so the bundle is never attempted, as the npub sign-in note records.

Replace DatabaseMarmotRepository.generateKeyPackage and buildKeyPackageLeafNode with CurrentProfileGroupFactory.createKeyPackage(signer, lastResort = true), and the event with KeyPackageEvent.buildCurrentProfileFrom(bundle.keyPackage, dTagSlot, clientName = "curare"). The event's tags change shape -- mls_extensions names 0x0006, mls_proposals names 0x0008 and 0x000a, an app_components tag lists the components, and there is no encoding and no relays tag -- and KeyPackageUtils.isValid checks all of it, so the jvmTest below runs the validator a peer runs.

What the leaf promises

The factory advertises the library's SUPPORTED_COMPONENTS: profile, both avatar carriers, admin policy, routing, retention, the identity proof, encrypted media v2, lifecycle -- and the agent-text-stream component 0x8006, together with its receive role capability. The library's comment on that last one is the reason to copy the list rather than narrow it: the reference client puts 0x8006 into the required set of every group it creates and refuses any leaf that omits it, so a narrower list makes this app's users un-addable to every White Noise group.

A component in that list is a promise to understand it, not to have a screen for it. What this obliges the app to is: decode every one of them when reading group state (Phase 1's accessors do, via MarmotGroupState), apply an AppDataUpdate proposal that changes one (the library does, inside processCommit), and never render a payload of a kind it does not know as if it were chat. Avatars, media and streams are state the app reads and never edits; the appendix says why that is enough.

Two slots, for now

KeyPackageUtils.MAX_SLOTS is ten. Publish the current-profile package in PRIMARY_SLOT and a legacy package in a second slot for the length of the transition, because an old build of this app decodes kind:30443 content as a bare KeyPackage and would otherwise be unable to invite anyone who upgraded. The current-profile leaf also advertises 0xF2EE, so a new build can be invited into a legacy room with either package; the second slot exists purely for old inviters and is retired in Phase 9.

One wrinkle decides the publish order. MarmotKeyPackageDao hands an inviter the peer's newest package by created_at, and an old build does exactly that and then fails to decode a framed one. So the legacy package is stamped one second after the current-profile package, every time both are published, and stays the newest until the slot is retired. A new build stops choosing by age: in a current-profile room it takes the peer's current-profile package or reports none, and in a legacy room it takes whichever decodes.

MarmotKeyPackageBundle gains a nullable profile column (schema 20 -> 21, an AutoMigration; existing rows are legacy) so rotation republishes the right shape into the right slot when a bundle is consumed, and MarmotKeyPackage gains the same column, filled at ingest from KeyPackageEvent.isCurrentProfile(), so the picker above is a query rather than a decode.

Where they go, and where to look

transports/nostr.md publishes KeyPackages to the account's NIP-65 write set, and the library's fetcher looks for a peer's in their NIP-65 write set unioned with the legacy kind:10051 hint and our own outbox. The app publishes to Relays.DefaultDMRelayList and reads 10051. Union rather than replace, in both directions: publish to the write set plus the default list, and add kind:10002 to the per-member sync request in processGroupMembershipChanges, which today asks for kinds 0, 10050, 30443 and 10051.

Tests. In jvmTest: the built kind:30443 passes KeyPackageUtils.isValid and isCryptographicallyValid, and the leaf's proof passes AccountIdentityProofV2.isValid against the credential identity; a group created by the library's own CurrentProfileGroupFactory.createGroup (standing in for Amethyst) adds the app's KeyPackage and the app processes the resulting Welcome through the NostrDao path into a room whose id is the routing id from 0x8004 and whose admin is the creator. That test is the one that fails first whenever the leaf shape drifts.


Phase 4 -- current-profile rooms, and the routing id

Four days. One schema change.

Splitting the id

A MarmotRouting table -- routingId (primary key), chatRoomId (foreign key to ChatRoom), activeFromEpoch, retiredAtEpoch (nullable) -- and every room gets a row. Rooms that already exist, and every room created by another client, get one whose routingId equals chatRoomId, so nothing about them moves. A room this app creates on the current profile gets a fresh RandomInstance.bytes(32) as its routing id and keeps SharedKeyDerivation.marmotGroupId as its ChatRoom.id.

The lookups that change are the ones keyed by the h tag: NostrDao.kt:441, where an inbound kind:445's groupId() becomes the room via the table; getMarmotGroupNostrEventsByChatRoomId, whose LIKE on the tag narrows by the room's routing ids; the #h filter LiveSubscriptionManager keeps open, which lists routing ids rather than room ids; and the outbound templates in MarmotOutboundDao, which put the routing id in GroupEvent.build. ChatRoom.id remains the key of every other table, the FROST cache, the chronicle verifier and the derivation. retiredAtEpoch is unused until routing rotation exists; it is in the schema now because v2 says a member "has to be able to map several routing ids to one group" and a second migration later would be the same column.

The alternative -- keep the derived id as the routing id and accept the spec's MUST NOT -- is rejected in the appendix, and not on purity grounds: the app broadcasts the entries a room's group signs, authored by the room's id, and a relay that also carries the room's kind:445s under that same id can link the group's private traffic to its public work with one string comparison.

Creating one

MarmotGroupCreation.create and SelectChatRoomTypeViewModel.createMarmotChatRoom both build a MarmotGroupData and hand it to MlsGroup.create. Both become

CurrentProfileGroupFactory.createGroup(
    signer = signer,
    nostrGroupId = routingId,
    relays = Relays.DefaultDMRelayList.map { it.url },
    profile = GroupProfileV1(name = groupName, description = description),
    additionalAdmins = adminPublicKeys.filterNot { it == userPublicKey }.map { it.hexToByteArray() },
)

which writes epoch 0 with required_capabilities naming extension 0x0006 and proposal 0x0008, a GroupContext dictionary holding the required-component list, the admin policy, the routing and the profile, and the creator's leaf with its proof. The description keeps carrying the derivation path exactly as the derivation note records it; GroupProfileV1.description allows 4096 bytes and byte equality, which is more than the sentence needs.

No 0xF2EE goes into a current-profile group. RFC 9420 requires every member to support every GroupContext extension, and an MDK leaf does not advertise 0xF2EE, so carrying both would make the room un-joinable by the clients the profile exists to reach. A current-profile room is pure, and Phase 1's accessors are what make that invisible to the rest of the app.

Which profile a new room gets

One constant, MarmotProfile.DEFAULT_FOR_NEW_ROOMS, legacy until Phase 9 flips it, and read only by the two creation sites. A current-profile room can only ever hold Phase 3 builds, so flipping the constant is the moment this app's users stop being able to make rooms for members who have not upgraded -- which is why it is the last act of the rollout and not part of this phase.

Refusals at the add step

MlsGroup.addMember refuses a leaf whose capabilities do not cover the group's required_capabilities, naming what is missing. In a current-profile room that is what adding an old build's KeyPackage does: its leaf advertises neither extension 0x0006 nor proposal 0x0008. (MDK goes further and also checks the component list in the leaf's dictionary; quartz checks only the MLS-level requirements, which is why Phase 3 copies the library's supported list rather than trimming it.) MarmotGroupCreation already reports Outcome.BlockedOn for a peer with no package; extend it to a package that cannot be used, and let the add-member confirmation screen say "their key package is from an older Marmot profile" rather than the generic failure, because the member can fix that and cannot fix a generic failure.

Tests. MarmotGroupCreationJvmTest gets a current-profile variant: the room's state decodes to a group whose currentGroupState().isCurrentProfile is true, whose routing id is not the room id, whose admin policy lists the admins; a kind:445 h-tagged with the routing id lands in the room; SubgroupManagerJvmTest still derives the child's id from the parent's key. A legacy room created by the same code with the constant unflipped is byte-for-byte what it makes today.


Phase 5 -- the library's engine over Room

A week. Two schema changes. The phase everything after depends on.

The four stores

  • MlsGroupStateStore over ChatRoom.mlsGroupState, keyed by routing id and resolved through MarmotRouting, since the manager's map is keyed by the nostr_group_id a group carries. saveRetainedEpochs / loadRetainedEpochs over MarmotRetainedEpochSecret, which is finally read back: today the inbound path's retainedEpochs is a map in an object, and every epoch it retained is gone at the next launch.
  • KeyPackageBundleStore is one opaque snapshot, so it is one row in a new MarmotKeyPackageBundleSnapshot table. On the first run the existing MarmotKeyPackageBundle rows are fed in through KeyPackageRotationManager.installBundle and the old table becomes what the key-package screen reads, written back from the manager's state.
  • MarmotIngestDedupStore over a new MarmotIngestDecision(eventId) table. Nothing in the library calls it -- it is the app's to mark terminal decisions with: a Welcome joined from, a Welcome naming a bundle this device never held. Never an event that is merely unreadable right now, which the library's comment on the interface is emphatic about.
  • MarmotPublishObligationStore is Phase 6.

Encryption at rest is unchanged by this: the library's interfaces say MUST, the app's TODO(security) H1 says the same, and moving the blobs behind an interface neither fixes nor worsens it. It does make the eventual fix one class.

One engine per identity

MlsGroupManager, KeyPackageRotationManager, MarmotInboundProcessor, MarmotOutboundProcessor and the MarmotPublishGate, constructed together and held by the identity's scope -- the same place the notary and the sync pumps get their KeyPair. restoreAll() and restoreFromStore() at startup, then trackGroup for every restored group, because convergence cannot see a fork in a group whose current state it never retained.

Inbound

NostrDao.indexMarmotGroupEvent replaces its MlsGroupCache.withGroup { MarmotInboundManager.processGroupEvent(...) } with engine.inbound.processGroupEvent(groupEvent). Everything after the result -- ChatMessage.fromGroupEventResult, the FROST and DKG dispatch, membership changes on CommitProcessed -- stays, which is why Phase 0's two when branches were written to return null rather than to throw. MarmotInboundManager shrinks to processGroupMembershipChanges and the payload helpers; MlsGroupCache and the vendored CommitOrdering are deleted.

Two things the app did by hand that the library does differently:

  • Dedup. The processor recognises a redelivered event by SHA-256 over the MLS bytes, computed after the outer layer opens. MarmotGroupEvent gains a nullable marmotMessageId column so the reindex sweep and the transcript can use the same identity. The Nostr event id stays the row's key -- it is still what the relay gives back.
  • Order. reindexMarmotGroupEvents sorts by the vendored comparator so that "the first one replayed is the one every other member also picked". Under convergence no replay order decides anything; the retained window and the six-step comparison do. The sort goes, and forgetPendingCommits becomes resolveConvergence(groupId) before the replay.

The retained-epoch decrypt path's security note -- a late application message under a prior epoch is decrypted without verifying the sender's signature -- survives this phase, because the library's tryDecryptWithRetainedEpoch has the same gap. Say so in the new code where the old warning was, rather than letting the rewrite look like it closed it.

Driving convergence

Inbound traffic is only an opportunistic tick. After every processGroupEvent, if openConvergencePasses() is non-empty, arm a timer that calls settleDueConvergence() at the quiescence window (one second) and keeps calling it until the map is empty; a group that goes quiet mid-pass has nothing else to drive it. LiveSubscriptionManager already owns a loop that outlives screens, and this belongs beside it.

A ConvergenceResolution with rewound = true means the state that was canonical lost. The first cut logs the outcomes and withdraws nothing, and the plan is honest about why: a rewind needs two members committing against the same epoch before either has seen the other's commit, the lines it leaves behind are membership captions filed under the losing commit, and every payload the losing branch delivered was still an authenticated member's words. The follow-up, once MarmotInnerEvent records the epoch each message arrived under (the result already carries it), is to mark rows from a losing branch withdrawn rather than delete them, so a transcript shows what the group contradicted instead of silently rewriting itself.

Welcome and outbound

The Welcome branch at NostrDao.kt:701 becomes engine.inbound.processWelcome(welcomeEvent, hint). WelcomeResult.Joined carries the routing id and whether a rotation is due; the room, its MarmotRouting row, its participants and the consumed-bundle bookkeeping are built from the manager's group as they are today from the local one. AlreadyJoined and a missing bundle are the two decisions to mark in the dedup store.

In encryptAndSendMarmotInnerEvent the middle -- MLS encrypt, outer layer, template, ephemeral signer -- becomes engine.outbound.buildAppEvent(routingId, appEvent); persistence and broadcast stay the app's. Before either, read convergenceStatus(groupId).allowsOutboundWork: while a pass is open the row stays on the outbound queue, which already retries, rather than being encrypted against a state that may lose.

Tests. The DAO tests keep their shape; their MlsGroup.create seeds become engine.groupManager.adoptGroup. MarmotRetainedEpochSecretDaoJvmTest gains the case the old code could not pass: a message from the prior epoch decrypts after the engine is rebuilt over the same database. MarmotReindexSweepTest gains a case that a replay of a duplicate is a Duplicate by message id even though every relay copy has a different event id, and the sort in reindexMarmotGroupEvents goes with the comparator it used.


Phase 6 -- publish before apply

Three days. One schema change.

Why not apply and roll back

The library's gate explains it better than a paraphrase: between applying a local commit and undoing it on failure there is a window in which this client's canonical state is an epoch no peer has, and a crash inside the window makes the fork permanent. So the commit is prepared on a clone, the live group never advances, and the pending state becomes canonical only on a confirmed accept. What counts as confirmed is at least one acknowledged accept from a relay in the obligation's scope; queued, sent, or "no error yet" are explicitly not.

The flow

MarmotPublishObligationStore over a new MarmotPublishObligation(obligationId, chatRoomId, bytes) table plus a MarmotOutboundGate(chatRoomId, gate) table, the second because a Disbanding or Leaving gate must survive a restart. MarmotCommitResult stays as the Welcome half; it already holds the framed commit, the Welcome bytes and the pre-commit exporter secret.

inviteMember becomes: stageAddMembers on the manager (a clone; the live group keeps its pending proposals) -> buildCommitEvent(routingId, staged.framedCommitBytes, staged.preCommitExporterSecret) -> gate.prepare(groupId, staged, outboundBytes, recipientScope = relays), which is durable before anything is broadcast -> the existing rows -> the existing broadcast requests. recordLocalCommit(groupId, staged.framedCommitBytes, sourceEpoch, staged.priorState) goes to the inbound processor at the same time, because the gate does not tell convergence about our own commits and a fork we are half of would otherwise look like an orphan.

DatabaseNostrRepository.broadcastProcessed is already the place a relay OK arrives and already turns it into the deferred Welcome. It gains gate.resolve(obligationId, CONFIRMED) first, which installs the pending state, and only then delivers the Welcome -- which is the MIP-02 ordering rule the Welcome sender's comment calls critical, enforced by sequence rather than by hope. A terminal rejection resolves FAILED, discarding the pending state with the live group already correct. A timeout resolves nothing: the obligation stays PENDING_PUBLISH, and the broadcast queue's requeue re-sends the same signed event, which is the byte-identical retry the rule demands -- a replacement commit would be a second commit at the same epoch, a fork this client caused itself.

The group that is only its creator keeps its exception. gate.satisfyEmptyObligation is the library's name for what isOneMemberInitialGroupCreation decides today, and its comment draws the boundary in the same place: creation and its founding Add, never a later commit.

What it closes

While an obligation is pending, canPrepareLocalCommit is false and a second invite waits in the queue instead of forking. The membership note's batching closed the race inside one invite; this closes it between invites, and between an invite and any other commit -- a rename, a removal -- that lands while the first is unacknowledged.

Tests. In jvmTest: an invite whose relay never answers leaves the live group at its old epoch and an obligation on disk; a new engine over the same database resumes it and a late OK installs the state; a rejection discards it and the room can invite again; an OK installs the state and then queues the Welcome, which is the existing test an invite into an established group is in the room before its welcome goes out with the order it asserts made strict.


Phase 7 -- lifecycle: leaving, disbanding, system rows

Three days.

Leaving

leaveChatRoom sets leftGroupAt and sends the text "Left group". The group does not learn of it: the leaf stays in the tree, holding the group's keys. v2 leaving is a SelfRemove proposal published as a PublicMessage kind:445, which the app already stages when it receives one from a White Noise member, and which an admin folds into their next commit. engine.groupManager.leaveGroup(routingId) produces the framed proposal and the exporter key to wrap it with, and drops the local state in the same call; the gate is raised to LEAVING first, which blocks further outbound from this device, and the leaver's row is marked as it is now. On the admin side, ProposalStaged sets hasPendingProposals, and an admin's device commits it through Phase 6 -- on receipt, as the library's comment expects, or behind a confirmation on the room detail screen, which is a UI choice and not a protocol one. The MIP-03 gate in the library bounds what a non-admin may commit to their own removal or their own key update, on both the sending and the receiving side.

Removal and disbanding

stageRemoveMember from the member sheet on the room detail screen, admins only, with the library's admin-depletion guard refusing a commit that would leave no admin in the tree. stageEnableDisbanding then stageDisband for a group that has run its course; after that every inbound is RefusedByLifecycle(DISBANDED), the room renders as closed, and the outbound queue refuses it. Disbanding is absorbing and there is no rejoin; the UI should say that before the tap.

Captions derived, never received

The app already synthesises its membership lines -- processGroupMembershipChanges diffs the participants table against the tree. v2 formalises the same idea as kind:1210 rows derived from a MarmotGroupSnapshot of canonical state, and it is explicit that a 1210 which arrives over the wire is one member's assertion, not a fact. Keep the app's TYPE_MEMBER_* line types and change their source: after each applied commit, MarmotGroupSnapshot.of(state, memberAccounts) against the last snapshot (a marmotSnapshot text column on ChatRoom, which is what the library's MarmotMessageStore.recordGroupSnapshot is for) yields the rows. A received kind:1210 or kind:1009 gets no bubble, and neither does any other unknown kind -- which retires the raw-JSON bubble the chronicle note complained about, for every kind at once.

Rename is stageAppDataUpdate with a new GroupProfileV1; the room detail screen already has the subject, so the button is small. Receive-side retention -- pinning a message's expiry to the retention of the epoch that carried it, and deleting rather than hiding -- is deferred to the list at the end; the app honours retention on send today and that does not change.

Tests. A leave produces a SelfRemove the library's stand-in processor stages; the admin's commit removes the leaf and the leaver's later kind:445 is refused; a disband is absorbing across a rebuilt engine; the snapshot diff yields exactly one member_added row per added member and none on a commit that changed nothing.


Phase 8 -- the test that actually proves it

Two days.

Two implementations in one group, in jvmTest: the app's engine over a BennuDatabase, and a bare library stand-in -- MlsGroupManager and MarmotInboundProcessor over the in-memory stores -- playing Amethyst. The existing subgroup tests already run two databases in one process; this is the same shape with a foreign member.

  1. Join. The stand-in creates a current-profile group and invites the app's KeyPackage; the app joins from the Welcome, and both sides exchange canonical payloads including a direct-message envelope. Assert each side's MlsGroupState agrees on epoch, tree hash and members.
  2. Invite. The app creates the room; the stand-in joins; same assertions, plus the routing id in the stand-in's 0x8004 is the app's MarmotRouting row and not its ChatRoom.id.
  3. The fork. Both sides add a different third member at the same epoch and each is handed the other's commit. Drive settleDueConvergence on both with the pass window elapsed. Assert both report SETTLED, the same canonical epoch and the same member set -- whichever branch won, they agree, and the test does not care which.
  4. The restart. The app prepares an obligation, is torn down, is rebuilt over the same database, and receives the OK. Assert the state installed once and the Welcome went out after it.

Plus a wire-vector test in commonTest: the bytes the app's outbound produces for each of its kinds decode with MarmotAppEvent.decode, and the id equals the one RumorAssembler computed before the switch.


Phase 9 -- rollout

No code, and three one-way doors.

The order is the phase order, with two holds in it.

Phases 0 to 3 ship together or in sequence; none of them changes what an existing room can do and each is readable by the build already in the field. Phase 2's chat payload switch is safe everywhere; its envelope is confined to current-profile rooms, of which there are none yet. Phase 3 publishes both KeyPackage slots.

Phase 4 ships built and unflipped. Phases 5 to 7 ship as they land; none of them is visible on the wire except that forks now settle and commits now wait for an OK.

The flip. MarmotProfile.DEFAULT_FOR_NEW_ROOMS goes to the current profile when every member this app's users are likely to invite runs a Phase 3 build. There is no negotiation for this and adding one is not worth it: a current-profile room refuses an old build's KeyPackage with a clear message at the add step, which is ugly rather than dangerous, and it stops as soon as they upgrade. Existing legacy rooms are unaffected forever; a room that wants to be on the current profile is a new room, and the members who join it late are handed its history by a chronicle, which is the mechanism that already exists for exactly that gap.

The retirement. One KeyPackage lifetime -- thirty days -- after the flip, the legacy slot stops being republished, and KeyPackageEvent.build has no caller left.

The three doors: the state blob (Phase 0; a downgrade loses every room touched), the first current-profile room a user creates (Phase 4 flipped; it can never hold an old build), and the envelope (Phase 2; a legacy room that ever sends one has a member who cannot read it, which is why it never does).

Where it stands. Everything above ships built and unflipped: MarmotProfile.DEFAULT_FOR_NEW_ROOMS is LEGACY, both KeyPackage slots are published and kept, and no room this app has made is on the current profile unless a caller asked for one by name -- the tests do; the creation screens do not. The first door is behind us the moment a build with the bump runs. The other two wait on the flip, which waits on the people this app's users invite.


What this does not do

  • Media, avatars, streams. Components 0x8002, 0x8007, 0x800b and 0x8006 are advertised and decoded, never edited or rendered. The app has no in-group media today and nothing here adds it.
  • Push. mip05PushNotifications was rewritten in 1.15.2; the app has no push path to attach it to.
  • External join by GroupInfo, and routing rotation. The retiredAtEpoch column is ready for the second; nothing writes it.
  • Message edits (kind:1009) and receive-side retention. Both are MarmotMessageStore features the app's own tables would carry; both are small and neither is needed to interoperate.
  • Encryption at rest. Unchanged, and now behind one interface.
  • Two library gaps the rewrite does not close and should not be read as closing: the retained-epoch decrypt without sender verification, and the secret tree's skipped keys not surviving saveState (mls-skipped-keys.md). The engine holding live groups covers the second within a process, exactly as MlsGroupCache did.
  • Migrating a legacy room in place. See the appendix.
  • iOS. Nothing here was compiled for it; the JSON deserialiser's sig default that Phase 2 leans on is the JVM and Android actual.

Appendix -- what was considered and rejected

Re-implementing v2 inside the app's pipeline. The app's MarmotInboundManager is already a fork of the library's manager as it stood in July, carrying every fix made since by hand and missing the ones it did not notice -- the framed KeyPackage, the Welcome by reference, the retention component. Convergence is two thousand lines whose only property that matters is agreeing with every other implementation; a second copy that agrees today drifts tomorrow. Adopting costs a week once; re-implementing costs the same week and then every release.

Keeping the derived id as the routing id. It reads as the smaller change: no table, no lookup, no second id. It violates a MUST NOT, and the reason for the MUST NOT applies to this app more than to most: the entries a room's group signs are published under the room's id, so the routing id would be the one string that ties a group's encrypted traffic to its public output. The split costs a table.

Adding 0xF2EE to current-profile groups so old builds could join. A GroupContext extension must be supported by every member's leaf; an MDK leaf does not advertise 0xF2EE; the room would then refuse exactly the clients the profile exists to reach. The old build's problem is solved by the old build upgrading.

Migrating a legacy room in place. The identity proof lives in the leaf and cannot be added afterwards, so migrating means every member re-keying their leaf by Update proposal and an admin committing a GroupContext change on top -- a ceremony across every device in the room, during which any member on an old build breaks the commit. The library's comment on the proof says it in four words: "no fallback and no in-place migration". A new room and a chronicle is what the app already knows how to do.

Sending the bare wrap into current-profile rooms. Every conformant peer drops it, and the app's own carve-out that admits it is the one place a member could render another member's words as their own if it were ever loosened. The envelope keeps every property the wrap was chosen for and removes the carve-out.

Signing the envelope. It would make the direct message attributable outside the group, which is the exact property the throwaway key was chosen to avoid, and a conformant decoder refuses it anyway.

Applying a commit and rolling it back on failure. Equivalent-looking and not equivalent; the gate's comment on the crash window is the whole argument, and the membership note's race is what that window looks like in practice.

One KeyPackage slot from the start. A current-profile package is framed; an old build of this app cannot decode it and so cannot invite anyone who upgraded. The second slot costs one more kind:30443 per rotation for one lifetime.