Phase 4 settled the blob and left the database alone, because there was nothing to settle: the state is opaque to Room and its version is inside it. Phase 8 added a table, and that turns "does an existing install still open?" from a question about MLS into a question about Room. They fail differently. A blob the engine cannot read costs one room; a migration Room cannot find costs the app, on every device that has ever had it, on the first launch after the update. That second one is also the easiest here to ship by accident, because a fresh install never migrates anything. Measured rather than assumed: with `AutoMigration(from = 36, to = 37)` commented out, 32 database tests ran and 4 failed -- all four of them the new ones. ShippedProfileUpgradeJvmTest builds a v36 database on disk from the exported v36 schema, down to its indices, its room_master_table identity hash and its user_version, puts a profile and two rooms in it with the Phase 4 blob in one, and opens it with the app's own getRoomDatabase. Four claims: it reaches v37 with Room's own identity hash stamped and every table's columns and indices equal to 37.json's; the legacy room's bytes come back unchanged and MarmotEngine.groupFor restores a group with its 0xF2EE metadata and a 32-byte kind:445 key; opening the profile rewrites nothing, which is asserted only after checking the restore held the room; and the new table takes a row keyed to a ChatRoom that predates it. Room opens lazily, so a test that builds a database and closes it migrates nothing. The first version of this did that and failed on user_version still reading 36. The blob itself moves to OldEngineBlob in commonTest, read by both tests. The only thing those bytes are worth is that they are old ones, and a blob living in two files can be regenerated in one of them. 1171 tests across 157 classes, 0 failures; the four audits unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
55 KiB
Marmot comes from marmot-kmp, not from quartz
quartz 2.3.0 took the Marmot engine out of quartz. What this app imports from
com.vitorpamplona.quartz.marmot — the MLS engine, the three processors, the
convergence engine, the publish gate, the app components, the KeyPackage managers — is
not in quartz any more at any version above 2.2.0, and it is not coming back. It lives
in marmot-kmp (the Marmot protocol and a
headless client) over mls-kmp (the RFC 9420
engine) over crypto-kmp (the primitives).
quartz keeps the Marmot event classes and nothing else.
So the app is not merely behind on a version. It is on the last quartz that still has an engine inside it, and every fix the engine has had since — including the one mls-skipped-keys.md is about — has landed in a library the app does not depend on. This is the plan to move onto that library.
Read marmot-v2.md first, in particular its Phase 5 and the first row of its table of where the build chose differently: a room's profile picks its pipeline, and everything here happens on the current-profile side of that branch. mls-skipped-keys.md second — it is the gap this move closes, and the only one it closes for free.
Built, phases 0 to 9, one commit each but for 5 and 6, which share one because the
fork probe produced no code. Phases 8 and 9 were not in the plan as written: 8 is the
piece Phase 7 deferred, and 9 is the schema change 8 made necessary, checked. Written against the code as it stood at 8b066795,
against marmot-kmp 2.2.1, mls-kmp 1.3.0, crypto-kmp 2.1.0 and quartz 2.7.3. The
phases are kept as they were written, with what each one turned out to be recorded under
it, because the reasoning is the point and the code reads better against the argument it
came from.
Two things it does not claim. The manual upgrade has not been done — Phase 4 proves a real version-4 blob opens and Phase 9 proves a v36 database carrying one opens at v37, but a profile with rooms that have real histories in them is still the last check before a release. And the authored fork is still not settled — Phase 6 probed it and the gap is the library's, unchanged.
| phase | what it was |
|---|---|
| 0 | the rule table, as a reversible script: 187 lines move, 42 stay |
| 1 | the swap — and the policy, which is the one behavioural change in it |
| 2 | the six renames quartz 2.x kept deprecated typealiases for |
| 3 | a second stand-in, on the library's own client, and the three tests it unlocks |
| 4 | C1 measured against a real blob from the old engine, six ways |
| 5 | the skipped keys: the format carries them, the app's row does not yet |
| 6 | the authored fork, probed: still open, and the fix now visible |
| 7 | the guard, so the engine cannot come back from quartz |
| 8 | the sender ratchet as its own row, and the receive-side write beside it |
| 9 | the schema change that followed, opened from the version the app ships |
Where the engine went
| library | coordinate | holds | the app today |
|---|---|---|---|
| quartz | com.vitorpamplona.quartz:quartz |
nostr: events, tags, signers, relay client — and the four Marmot event classes (KeyPackageEvent, KeyPackageRelayListEvent, WelcomeEvent, GroupEvent) plus the MIP-05 push events |
1.15.2, where the engine is still inside it |
| marmot-kmp | com.vitorpamplona.marmot:marmot |
MIP-00..05, MarmotInboundProcessor, MarmotOutboundProcessor, MlsGroupManager, MlsGroupStateStore, MarmotPublishGate, the convergence engine, appComponents, foundation/appEvents, KeyPackageRotationManager, and MarmotManager — a headless client the app does not have to use |
not depended on |
| mls-kmp | com.vitorpamplona.mls:mls |
RFC 9420: MlsGroup, MlsGroupState, the TLS codec, the ratchet tree, the key schedule, framing |
not depended on |
| crypto-kmp | com.vitorpamplona.crypto:crypto |
SHA-256, HKDF, HMAC, AES, ChaCha20-Poly1305, X25519, Ed25519, RandomInstance |
arrives transitively under quartz 2.x; not declared |
All four are on Maven Central, as is com.vitorpamplona.negentropy:kmp-negentropy, which
quartz pulls in — so no new repository is needed. marmot-kmp publishes jvm, android,
iosArm64, iosSimulatorArm64, macosArm64 and linuxX64: every target this app
declares. MIT, like quartz.
marmot-quic is a second artifact for agent text-stream previews over raw QUIC, JVM and
Android only. Nothing here wants it.
The move, symbol by symbol
209 import lines across 57 files name com.vitorpamplona.quartz.marmot — 32 files under
commonMain, 25 under test. Every one of them resolves, and the rule is almost entirely
positional:
| from | to | lines |
|---|---|---|
quartz.marmot.mls.{codec,components,crypto,framing,group,messages,schedule,tree}.X |
mls.{same}.X |
80 |
quartz.marmot.mls.crypto.{Ed25519,X25519} |
crypto.curve25519.{same} |
8 |
quartz.marmot.mls.group.{MlsGroupManager,MlsGroupStateStore} |
marmot.groups.{same} |
5 |
quartz.marmot.{appComponents,foundation,protocolCore,mip00KeyPackages,mip01Groups,mip03GroupMessages}.X, quartz.marmot.{GroupEventResult,MarmotInboundProcessor,MarmotOutboundProcessor} |
marmot.{same} |
74 |
quartz.marmot.mip00KeyPackages.{KeyPackageEvent,KeyPackageRelayListEvent}, mip02Welcome.WelcomeEvent, mip03GroupMessages.GroupEvent |
unchanged | 42 |
Two symbol pairs that the compiler will catch but the rule above does not: the MLS group
manager and store are marmot-kmp's, in marmot.groups, while everything else called
MlsGroup* is mls-kmp's, in mls.group — one letter apart, and the only place the
positional rule breaks. And RetainedEpochSecrets sits in mls-kmp beside MlsGroupState,
which is where RoomMlsGroupStateStore reads it from.
Outside the Marmot tree, 19 more lines move to crypto-kmp because quartz 2.x no longer
declares them: quartz.utils.RandomInstance → crypto.random.RandomInstance (17 lines,
same bytes(n)/int()/randomChars() API) and quartz.utils.sha256.sha256 →
crypto.sha2.sha256 (2 lines; the package is gone from quartz entirely).
And then the part that is not a rename: 854 import lines name quartz outside
quartz.marmot, and all of them ride a 1.15.2 → 2.7.3 major bump. Resolving every one of
those 854 against a 2.15.1 checkout found the packages intact — nip01Core.core,
nip01Core.crypto, nip01Core.signers, nip01Core.relay, the nip-numbered event
classes, utils.TimeUtils, utils.ensure, utils.EventFactory are all where they were,
eventTemplate and EventTemplate included — but "the import resolves" is not "the
signature is the same", and reading cannot tell the difference. Phase 1 is where the
compiler tells us.
Two constraints
C1 — a group on a device today must still open tomorrow. Group state is persisted
as MlsGroupState.encodeTls() hex in ChatRoom.mlsGroupState, and retained epochs as
MarmotRetainedEpochSecret rows. If the new engine cannot read what the old one wrote,
every existing room loses its membership and the only repair is a new group and a
re-invite — LegacyGroupContractTest in marmot-kmp is explicit that there is no in-place
migration from a legacy group. Reading mls-kmp says this holds: MlsGroupState is
versioned, STATE_VERSION is 10, decodeTls accepts 1..STATE_VERSION, and the comments
say each version only appends ("optional (empty for STATE_VERSION 1 blobs) so older
persisted state…"). RetainedEpochSecrets carries the same five fields the app's rows do
and tolerates an absent exporterSecret. That is an argument, not a measurement. Phase 4
measures it.
C2 — the fork still merges from Bennu. giza-rebrand.md keeps cherry-picking from upstream, and upstream is on quartz 1.15.2 with the engine inside it. Every file this plan rewrites is a file a future cherry-pick will conflict in. That argues for one mechanical commit with a message that says exactly what the rule was, so the conflicts are resolvable by re-applying the rule rather than by reading the diff.
ForkCompatibilityTest pins the npub and the profile id a recovery phrase derives, and
the mantra/… derivation tags are hash inputs. Nothing here touches either, and both are
in the gate at every phase boundary anyway.
Decisions
D1 — take :marmot, keep MarmotEngine. marmot-kmp ships MarmotManager, a headless
client that owns the stores, the publisher, the receiver and the dedup store. The app has
MarmotEngine over the same three processors, wired into GizaDatabase, the outbound
queue, the broadcast status machine and NostrDao.indexMarmotGroupEvent. Swapping engines
and libraries in one move would make a failure impossible to attribute. The constructors
line up exactly as the app already calls them — MlsGroupManager(store),
MarmotInboundProcessor(groupManager, KeyPackageRotationManager()),
MarmotOutboundProcessor(groupManager), MarmotPublishGate(groupManager, store) — so the
seam holds with no redesign. MarmotManager is Phase 7's question, not this plan's.
D2 — pin quartz to 2.7.3, the version marmot 2.2.1 was built against. 2.15.1 is
published and marmot-kmp has not been compiled against it; a quartz that moved an API
marmot-kmp calls gives a NoSuchMethodError at runtime rather than a compile error.
Gradle resolves to the highest, so the catalog has to name 2.7.3 deliberately and the app
has to stay off any quartz API added after it. Moving both forward together is its own
change, with marmot-kmp's own CI as the evidence.
D3 — crypto-kmp is declared, not inherited. It arrives transitively under both quartz
2.x and mls-kmp, but after this move the app imports RandomInstance, sha256, Ed25519
and X25519 from it directly in 27 places. A dependency the source names is a dependency
the build names.
D4 — the legacy pipeline is touched only by the rename. MarmotInboundManager,
MlsGroupCache and the app's own CommitOrdering keep their imports pointed at the same
classes, now in mls-kmp, and keep their behaviour. The reasoning is marmot-v2.md's and
unchanged: the library's processor has no carve-out for the bare gift wrap a legacy room
carries a direct message in (marmot-direct-messages.md),
and the legacy path retires with the profile.
D5 — no new behaviour inside the swap. The sender-ratchet split, the durable ingest dedup and the skipped-key persistence are all things the new engine makes possible. Each is a change in what the app does, each is separately testable, and none of them belongs in the commit that changes where the classes come from.
D6 — the stand-in moves with the library. MarmotStandIn is "another Marmot client,
played by quartz's own managers" — the other side of every two-implementation test in
managers. It must be played by marmot-kmp's managers, or the interop tests prove the new
engine agrees with itself.
Phase 0 — the inventory, before anything moves
No code. Produce the three lists the later phases consume, and check them into
docs/scripts/ if any of them turns out to be worth re-running:
- the 228 import lines and their destinations, by the rule in the table above — a
sedscript per row, not a hand edit; - the five places the app implements or extends a library type, which are where the
compiler will have the most to say:
RoomMlsGroupStateStore(nowmarmot.groups, and the interface has gained defaultedsaveSenderRatchet/loadSenderRatchet),RoomPublishObligationStore(all six methods it overrides are still there, gate trio included),MlsGroupExtensions.exporterSecret,MlsGroupProfile's six accessors, andMarmotGroupFactory; - the app's own copies of what the library now has, for Phase 7 to delete rather than
carry:
CommitOrdering, the in-memory dedup inMarmotInboundProcessor's caller, and whateverMarmotIngestDedupStoreturns out to replace.
Check: the lists exist and the sed scripts are reversible.
Built. The rule table is
docs/scripts/marmot-kmp-imports.sh, which takes
--apply, --revert and --check, and the destinations in it were derived by listing
what quartz 2.15.1 still has under marmot/ against what marmot-kmp 2.2.1 and mls-kmp
1.3.0 declare rather than by reading the packages' names. Applied and reverted on a clean
tree it moves 187 lines and puts all 187 back: 79 onto marmot-kmp (74 by package plus the
manager and the store), 80 onto mls-kmp, 27 onto crypto-kmp, and one KDoc reference in
CommitOrdering. 42 quartz.marmot imports are left standing, which is the count the
table above predicts for the event classes.
Three things the split turned out to be finer than the table says, all of them inside a package quartz keeps:
| stays in quartz | moved to marmot-kmp |
|---|---|
mip00KeyPackages: the events, PublishedKeyPackage, the ten tag classes and the tag accessors (encoding, keyPackageRef, mlsCiphersuite, mlsExtensions, mlsProposals, mlsProtocolVersion) — 16 files |
KeyPackageBundleStore, KeyPackageEventFromMls, KeyPackageFetcher, KeyPackageRotationManager, KeyPackageUtils |
mip02Welcome: WelcomeEvent and its three tags — 6 files |
WelcomeGiftWrap |
mip03GroupMessages: GroupEvent, GroupIdTag, the two tag extensions — 4 files |
GroupEventEncryption |
mip05PushNotifications: all 13 event and record files |
MarmotPushStateStore, PushRecordStore, TokenEncryption |
marmot-kmp imports the quartz side of that table itself, which is the clearest statement
that the halves are meant to sit together. And mip01Groups and mip04EncryptedMedia
have no quartz side at all.
The five places the app implements or extends a library type:
RoomMlsGroupStateStore, RoomPublishObligationStore (whose six overrides all still
exist, gate trio included), MlsGroupExtensions.exporterSecret, MlsGroupProfile's six
accessors and MarmotGroupFactory. The app's own copies of what the library now has, for
Phase 7: CommitOrdering, and MarmotEngine's reliance on the processor's in-memory
dedup.
One thing Phase 0 found that is not about imports at all: the lightning-kmp-app
submodule, and the four nested ones under it, have to be checked out before the build can
configure — Project with path ':library' not found in build ':lightning-kmp-app' is what
a fresh worktree says. git submodule update --init --recursive.
Phase 1 — the swap, in one commit
The version catalog gains marmot, mls and crypto, quartz goes to 2.7.3,
composeApp's commonMain declares the three, and the Phase 0 scripts run. Nothing else.
quartz = "2.7.3" # was 1.15.2 -- what marmot 2.2.1 was compiled against (D2)
marmot = "2.2.1"
mls = "1.3.0"
cryptoKmp = "2.1.0"
Expect the compiler to have plenty to say, and expect almost all of it to be in the 854
non-Marmot lines rather than in the renamed ones — quartz 2.0.0 is a repository split as
well as a major version, and the 1.x line was published from Amethyst's own tree, so there
is no upstream changelog that covers the gap. Three things are already known to be in it:
RandomInstance and sha256 move (Phase 0's third script), quartz 2.x drops
whyoleg.cryptography, andreypfau.kotlinx-crypto-*, charlietap.cachemap and
jackson-module-kotlin in favour of crypto-kmp, and it brings kotlin-stdlib 2.4.21 and
kotlinx-serialization 1.12.0 against the app's Kotlin 2.4.10 and serialization 1.11.0.
The stdlib skew is not new — quartz 1.15.2 already brings 2.4.20 — but a klib built by
2.4.21 read by a 2.4.10 compiler is, and it will only show up on the native targets, on a
mac. Bumping kotlin to 2.4.21 is the cheaper answer if it does.
Check: ./gradlew :composeApp:compileDebugKotlinAndroid :composeApp:jvmTest, then
./gradlew check for the three audits. On a mac, the ios targets too — they are the only
place a klib skew can appear.
Built, and the fear was misplaced. The 854 non-Marmot quartz imports came through the
1.15.2 → 2.7.3 bump without a single edit. Every compile error — 45 of them — was in the
engine seam, and 44 of the 45 were one thing: five accessors that were members of
MlsGroup in quartz's engine are extension functions in marmot-kmp
(currentGroupState, currentMarmotData, currentNostrGroupId,
currentAdminIdentities, isLocalAdmin, all in marmot.groups), so the 14 files that
call one now import it. The 45th was KeyPackageEvent.Companion.buildCurrentProfileFrom,
an extension too. Nothing in the API the app drives had changed shape: every constructor
the plan quoted in D1 took its arguments unaltered, and GroupEventResult's nine variants
are still nine — MarmotEngine's exhaustive when over them compiles, which is the
compiler proving there is no tenth to drop silently.
Two things the bump brings that are not errors. quartz 2.x keeps a deprecated typealias
for each class it renamed, so six the app uses still compile and warn:
SealedRumorEvent → SealEvent (22 sites), ChatMessageRelayListEvent →
DmRelayListEvent (19), RelayFeedsListEvent → FavoriteRelayListEvent (5),
LnZapEvent → ZapReceiptEvent (5), LongTextNoteEvent → LongFormContentEvent (1),
ChannelListEvent → PublicChatListEvent (1). And quartz 2.x drops
whyoleg.cryptography, andreypfau.kotlinx-crypto-*, charlietap.cachemap and
jackson-module-kotlin from the dependency graph in favour of crypto-kmp. The
kotlin-stdlib skew the plan worried about was already there — 1.15.2 brought 2.4.20
against the app's Kotlin 2.4.10 — and 2.4.21 behaves no differently.
And one thing that is not a rename at all: the group policy. MlsGroupPolicy is new
in mls-kmp. quartz's engine hardcoded Marmot's rules; mls-kmp made them an argument, and
MarmotGroupPolicy (marmot-kmp, marmot.groups) is the object that carries them. Every
factory — create, restore, processWelcome, externalJoin, createKeyPackage —
defaults it to MlsGroupPolicy.Permissive, so the eleven places the app builds a group
itself were silently building a non-Marmot group. What that costs, in the order it bites:
MarmotGroupPolicy |
Permissive |
|---|---|
commitExporter = MLS-Exporter("marmot", "group-event", 32) |
null, so CommitResult.preCommitExporterSecret comes back empty and GroupEventEncryption.encrypt refuses it: "Group key must be 32 bytes" |
greaseCapabilities = false |
true — GREASE values in the leaf's capabilities, which MDK "neither sends nor has been tested against" |
knownExtensionTypes = 0xF2EE and the app-data dictionary |
empty, so RFC 9420 §13.4's all-members-support check refuses a GroupContextExtensions proposal in a group this client is happily a member of |
defaultLeafCapabilities / defaultRequiredCapabilities = Marmot's |
empty / none |
authorizeCommit, authorizeProposal, authorizeSelfRemove, validateJoin |
no-ops: no authorized-proposal-set check, no admin-depletion check |
Only the first of those five is loud, and it is what the single failing test found — the
second invite into an established legacy room, MarmotOutboundDaoJvmTest. The other four
are silent, and the GREASE line is wire-visible: a KeyPackage this app published would
have carried values the reference client has never been tested against. So the policy goes
in at every construction site, which is this phase's one behavioural change and the reason
the swap is not purely mechanical. Four in commonMain — ChatRoom.toMlsGroup,
MarmotEngine.groupFor, MarmotGroupFactory's legacy branch, NostrDao's Welcome path —
and seven in the fixtures. The current-profile paths needed none: they go through
marmot-kmp's own CurrentProfileGroupFactory, which passes the policy itself, and so does
every MlsGroupManager entry point.
With the policy in, the suite is green: 1149 tests, 0 failures, and compileDebugKotlinAndroid
clean. One housekeeping note: a rewritten import keeps its old alphabetical slot, so the
touched files' import blocks were re-sorted — but only the touched ones. Sorting every
block in composeApp would have moved lines in 253 further files whose imports were
already out of order before any of this, and C2 is the reason not to: each of those is a
line a cherry-pick would then conflict in for no reason.
Phase 2 — the drift in the seam
Phase 1 ends when it compiles. This phase is for what compiles and means something else.
The engine moved from quartz 1.15.2 to a library that has had five releases of its own
since, so the behavioural questions are concentrated in MarmotEngine and its callers:
GroupEventResult's variants:MarmotInboundProcessordeclares them in its own file and the app branches on them inNostrDao.indexMarmotGroupEventandDatabaseMarmotRepository. A new variant the app does not handle is a silent drop.ConvergenceStatus,ConvergenceDispositionandGroupLifecycleState— the processor now exposessettleDueConvergence(),openConvergencePasses()andgroupLifecycle(), which is whatMarmotEngine.settleLaterandawaitSettledare built on.MlsGroupManager's staging API (stageAddMember,stageCommit,installState,StagedCommit) against whatMarmotPublishGate.prepare/resolveexpects, which is the publish-before-apply path marmot-v2.md's Phase 6 built.KeyPackageUtils,CurrentProfileGroupFactoryandMarmotGroupDatainMarmotGroupFactoryandDatabaseMarmotRepository— the group-creation shapes.
Check: the existing suites are the specification here. GroupLifecycleJvmTest,
ConvergenceForkJvmTest, PublishBeforeApplyJvmTest, CurrentProfileInteropJvmTest,
MarmotGroupCreationJvmTest, RobustRoomKeyCeremonyTest,
SubgroupCeremonyInParentRoomTest. A test that needed editing to pass is a finding;
record it in this note's table rather than in the diff.
Built, and most of this phase turned out to be already answered by Phase 1 rather than work of its own:
GroupEventResulthas nine variants, the same nine. Not argued from reading:MarmotEngine.inRoomis awhenover the sealed class used as an expression, so the compiler requires it to be exhaustive. It compiles, which is the compiler stating there is no tenth variant for the app to drop silently.- The convergence and lifecycle surface is unchanged.
convergenceStatus,resolveConvergence,settleDueConvergence,openConvergencePassesandgroupLifecycleall take and return whatMarmotEngine.settleLaterandawaitSettledalready expected, down toConvergenceStatus.allowsOutboundWork. - So is the staging API against the gate.
stageAddMember→StagedCommit→gate.prepare→installStateneeded no edit, andPublishBeforeApplyJvmTestandConvergenceForkJvmTestpass with their assertions untouched. - No test's assertions had to move. The fixtures changed, in that seven of them now
construct their group with
MarmotGroupPolicylike the app does — a construction-site change, not a changed expectation. Nothing in this migration has yet been accommodated by weakening a test.
What was left is the six renames quartz 2.x made and kept deprecated typealiases for, now adopted: 104 references across 25 files. Same packages, so each is an identifier rename, and none of them reaches a string the app persists — the diff touches code and log lines only, no Room column, no string resource, no kind number.
| quartz 1.15.2 | quartz 2.x | why |
|---|---|---|
SealedRumorEvent |
SealEvent |
NIP-59 names kind 13 the seal |
ChatMessageRelayListEvent |
DmRelayListEvent |
NIP-17 kind 10050 is the relay list to receive DMs |
RelayFeedsListEvent |
FavoriteRelayListEvent |
NIP-51 names kind 10012 the favorite relays list |
LnZapEvent |
ZapReceiptEvent |
NIP-57 names kind 9735 the zap receipt |
LongTextNoteEvent |
LongFormContentEvent |
NIP-23 names kind 30023 long-form content |
ChannelListEvent |
PublicChatListEvent |
NIP-51 names kind 10005 the public chats list |
Phase 3 — the stand-in, and what interop now means
MarmotStandIn goes over to marmot-kmp's managers, and then there is a choice worth
making here rather than later: the stand-in is the one place MarmotManager can be tried
with nothing depending on the answer. If the stand-in is a MarmotManager, every
two-implementation test in managers becomes the app's engine against the client Amethyst
actually ships, which is a stronger claim than the one those tests make today.
Check: ./gradlew :composeApp:jvmTest. ForkCompatibilityTest and
SignedGroupKeyStateTest are in the same run and are the fork's own pins.
Built, and both stand-ins are kept. MarmotStandIn moved to marmot-kmp's managers in
Phase 1 with nothing but its imports changing, and replacing it with a MarmotManager
would have cost the convergence and lifecycle tests the thing they need, which is a
processor they can reach into: recordLocalCommit with a state snapshot of their own,
buildAppEvent with a payload they wrote. So the second one is additive —
MarmotClientStandIn, a MarmotManager over InMemoryMlsGroupStateStore and a publisher
that says yes, with messageStore and keyPackageStore left null because both are
nullable and a client that persists nothing is all these tests want — and
MarmotClientInteropJvmTest is the three tests it unlocks.
They are not redundant, and the difference is worth stating because it is the difference
between the two questions this migration has to answer. In
CurrentProfileInteropJvmTest the test's own code decides how the Welcome is sealed and
what an application payload looks like, and asks whether two engines agree. In the new one
the library decides: addMemberInvites picks the commit's shape and MarmotWelcomeSender
wraps the Welcome, and the question is whether the app's own orchestration —
MarmotEngine, the DAOs, the outbound queue — reaches the same group as the orchestration
Amethyst drives. It does: the app joins a group MarmotManager created and invited it to,
under the routing id the client minted; it files and attributes a buildTextMessage from
it; and the client decrypts and decodes what the app's outbound path sends as a canonical
payload. Three tests, green on the first run, which is the strongest statement available
that the move did not change what goes on the wire.
One carve-out rides along unchanged: the app's rumor still carries "sig":"", so the
assertion strips it before decoding, exactly as the older interop test does.
docs/marmot-v2.md's Phase 2 is where that is recorded as unfinished, and this note does
not change it.
Phase 4 — C1, measured
The phase that decides whether this ships. Everything before it is a build; this is the evidence that an upgrade does not cost a user their rooms.
- A real blob, decoded. Take
ChatRoom.mlsGroupStateout of a database written by the current build — the first four hex characters are itsSTATE_VERSION— and decode it with mls-kmp 1.3.0. Assert the version is in1..10and that the decoded state's epoch, group id and leaf index are what the old engine reported. This belongs injvmTestwith the blob as a checked-in fixture, which is also how it stays true: it becomes the test that a future mls-kmp bump has to pass. - A round trip through the store.
RoomMlsGroupStateStoreJvmTestalready covers save/load and the retained-epoch rows; extend it to load an old blob, re-save it, and assert the group still decrypts a message encrypted before the upgrade. - The window. Retained epochs are re-encoded field by field by
RoomMlsGroupStateStore.loadRetainedEpochs, so a field added toRetainedEpochSecretswould silently truncate them. Pin the five fields in the same test.
If any of the three fails, this is where the plan stops and a migration gets designed. If all three pass, no schema change is needed at all — the blob is opaque to Room and the version is inside it, so schema 36 stands.
Check: the three tests, and one manual upgrade on a device or desktop profile with real rooms in it. The automated evidence is necessary and not sufficient; a room that has been through several epochs and several members is not something a fixture reproduces.
Built, and C1 holds — on real bytes rather than on the argument. The argument turned
out to be checkable before any test was written: quartz 1.15.2's own sources, in the
sources jar beside the artifact in the gradle cache, declare STATE_VERSION = 4. mls-kmp
1.3.0 declares 10 and accepts 1..10. So every blob in the wild is a version-4 blob and
the window covers it with six versions to spare.
Then the measurement. docs/tools/old-state-blob/ is a project whose only dependency is
com.vitorpamplona.quartz:quartz:1.15.2 — the engine the app shipped — and it prints a
real MlsGroupState blob: 1033 bytes, a legacy MIP-01 group at epoch 0, which is the
shape every room in the wild started as. It is checked in for provenance, because once it
is gone there is no way to tell by reading the hex whether the fixture came from the old
engine or the new one, and a fixture written by the new engine would make the test pass
while testing nothing. MlsGroupStateCompatibilityTest pins it, six ways:
| the fixture is still a version-4 blob | asserted on its own, so a fixture that quietly became version 10 cannot make the rest pass vacuously |
| mls-kmp decodes it | epoch 0, leaf 0, a 32-byte group id, an encryption secret |
| the group restores from it | key schedule, secret tree and ratchet tree rebuilt under MarmotGroupPolicy, 0xF2EE intact, same nostr_group_id, same name, and it still reads as legacy rather than current-profile |
| the kind:445 outer key is derivable from it | 32 bytes — Phase 1's policy finding pinned against real pre-migration bytes rather than a fixture the new engine built |
| a re-save moves it forward and still decodes | version 10, same group id, same epoch, same leaf: the upgrade a room gets the first time the manager persists it |
| the retained-epoch record still has five fields | round-tripped, because RoomMlsGroupStateStore decodes one into five columns and re-encodes it from them, so a sixth field would be silently truncated on every load |
No schema change was needed, as the plan guessed: the blob is opaque to Room, the version is inside it, and schema 36 stands.
What is not done, and is not automatable here: the manual upgrade. A fixture is a group at epoch 0 with one member. The rooms that matter have been through several epochs, several members and a retained-epoch window, and the only way to know they open is to open one. That check needs a device or desktop profile with real rooms in it, and it is the last thing between this work and a release. Phase 9 takes half of it: the database a v36 install has, opened at v37 with this blob inside it, is a test now. The histories in a real profile still are not.
Phase 5 — the skipped keys, closed
mls-skipped-keys.md describes a conformance gap in quartz's MLS
implementation: SecretTree caches the keys for generations that arrive early, exactly as
RFC 9420 §9.1 requires, and then never persists them — so two messages published in the
same second reliably lose one across a restart, and the loss is silent. mls-kmp's
MlsGroupState persists them: STATE_VERSION 5 is "skipped-generation secrets" and 7 is
"the secret tree's unexpanded node secrets". The app's partial mitigation — the live group
held in MlsGroupManager for the life of the process — is no longer all there is.
So: write the test the note's symptom section describes, from the receiver's side, across
a save/restore boundary. Two messages in the same epoch delivered in reverse order
after a restart; both decrypt. Then update that note to say which engine closed it and
leave its mitigation section as the history of how it was lived with.
Check: the new test, and MarmotEngine's own doc comment, whose last paragraph
currently names this gap as open.
Built, and half of it is closed — not the half the plan assumed. The engine-level gap
is gone: MlsGroupState carries skippedApplicationSecrets and
skippedHandshakeSecrets from STATE_VERSION 5, saveState exports them from the secret
tree and restore imports them, and because they arrived at version 5 while the decoder
accepts 1..10, a version-4 blob reads them as an empty map. That is steps 1 to 3 of the
fix mls-skipped-keys.md proposed, in the library the app now
depends on. Step 4 is not: consumedGenerations, the guard against a replayed message
re-using a cached key, is still one SecretTree instance's field and is in no version of
the state.
The app does not benefit yet, and the first draft of this test said it did.
MlsGroupManager.decrypt persists only when the message was a commit that advanced the
epoch; an application message is decrypted and nothing is written. So the secret reaches
the live group's state and never reaches ChatRoom.mlsGroupState.
That matters for how the test had to be written, and the first attempt is worth recording
because it passed. It delivered generation 1, rebuilt the engine with MarmotEngine(db),
delivered generation 0 and asserted it decrypted — and it did, for the wrong reason
entirely. The rebuilt engine restores a blob whose sender ratchet never moved, so
generation 0 is simply the next generation it expects and no skipped key is involved. A
green test proving nothing, and what caught it was the one assertion that looked redundant
beside it: that the secret is in the stored state. It was not.
So SkippedGenerationJvmTest is three tests that say what is true:
| the secret is in the state the engine would save | MlsGroupManager.snapshot after generation 1 arrives first — exactly one skipped secret, where the old engine had none |
| a group rebuilt from those bytes reads the late message | the full round trip through encodeTls/decodeTls/restore, which is what a restart is |
| the stored state does not carry it | pinned, in the style of ConvergenceForkJvmTest's authored fork, with the KDoc saying what to invert the day the receive path starts persisting |
The fix for the third is one call — the app owns the store, so
store.save(routingId, groupManager.snapshot(routingId)!!.encodeTls()) after an
application message would do it — and it is deliberately not made here, because it is a
write per inbound message that re-encodes the ratchet tree, the group context and every
epoch secret. MlsGroupStateStore.saveSenderRatchet is the library's answer to exactly
that cost on the send side; the receive side wants the same shape, and that is Phase 7's
business rather than this one's. mls-skipped-keys.md has a new
last section saying all of this, and its mitigation section stands word for word with a
new cause.
Phase 6 — the authored fork
marmot-v2.md ends on a gap that is the library's and not the app's: a member cannot replay
a commit it authored, so a fork this device is half of is scored with its own branch stale,
and two committers swap branches rather than agree. ConvergenceForkJvmTest holds both
halves — the case the app depends on, proven, and the swap, pinned, with its last
assertion saying what to do the day it fails.
Reading marmot-kmp's MlsCandidateStateEngine and MarmotConvergenceEngine does not
settle whether that is still true; they talk about holding an authored commit and a
received one "to one rule rather than two", which is suggestive and not evidence. The test
is the probe. If the pinned swap now fails, the gap closed in the library and the test
inverts into the assertion it predicts. If it still passes, nothing has changed and
Phase 5's follow-up — marking a losing branch's rows withdrawn — still waits on the same
fix.
Check: ConvergenceForkJvmTest, and whichever way it goes, this note and
marmot-v2.md's closing paragraph both say so.
Probed: the gap is still open, and the test still pins the swap. Both fork tests pass unchanged — the one that proves two observers of somebody else's fork settle on the same state byte for byte, and the one whose last assertion is
assertNotEquals(ours.contextBytes().toHexKey(), theirsAfter.contextBytes().toHexKey(),
"the two sides agree -- the library has been fixed; rewrite this test as the agreement")
That message is the probe, and it did not fire. So marmot-kmp 2.2.1 scores a fork this
device is half of the same way quartz 1.15.2 did: MlsCandidateStateEngine rebuilds each
candidate branch with MlsGroup.processFramedCommit, a member cannot replay a commit it
authored, and two committers swap branches rather than agree. Reading says the same thing
the test does — the suggestive line in MlsCandidateStateEngine about holding "an inbound
commit and one we authored to one rule rather than two" is about authorization, not
replay.
What did change is that the fix marmot-v2.md predicts is now visibly available. Its
closing note says "the state the missing replay would produce is already retained, and
using it is the fix", and mls-kmp has made exactly that state a first-class thing:
stageCommit/mergePendingCommit, with the staged commit persisted in MlsGroupState
from its STATE_VERSION 9. The app already holds the same state twice over, as
MarmotPublishObligation.pendingState. So the fix is a change to the candidate engine's
choice of how to get a branch's state — replay it, or take the one we kept — and it
belongs in marmot-kmp rather than here.
Unchanged, therefore: the practical exposure is small, because a device that never applies before publishing rarely authors one side of a fork, and the Phase 5 follow-up in marmot-v2.md (marking a losing branch's rows withdrawn) still waits on the same fix. No code in this phase.
Phase 7 — what the library now offers, and the guard
Four things, each its own commit, none of them required by the move:
- The sender ratchet record.
MlsGroupStateStore.saveSenderRatchet/loadSenderRatchetdefault tofalse/null, and the manager then falls back to a fullsave()— correct, and it re-encrypts the ratchet tree, the group context and every epoch secret to persist a counter, per message, growing with the group. Implementing the pair over a small table is the fix. The interface's own words: the write MUST be durable before it returns, because a lost position re-emits a generation already used, which reuses an AEAD key and nonce pair. - Durable ingest dedup.
MarmotIngestDedupStoreis what marmot-v2.md's Phase 5 follow-up asked for — the processor's dedup is in memory for the life of the process, and gift wraps are backdated up to two days, so a caught-upsincecursor re-delivers the whole band on every sync. One catch: the store is taken byMarmotManager, not byMarmotInboundProcessor, so this one is only available by adopting the client — which makes it the honest trigger for that decision rather than a change on its own. - The app's own copies. Phase 0's third list, deleted.
- A guard.
docs/scripts/already holds three audits that./gradlew checkruns. A fourth, or a line in an existing one, that fails onimport com.vitorpamplona.quartz.marmot.followed by anything but the four event classes — so the engine cannot come back from quartz by way of a cherry-pick from upstream, which is exactly the shape C2 predicts.
Check: ./gradlew check, with the new guard in it.
The guard is built; the other three are named rather than done, and here is why each.
marmotAudit runs docs/scripts/marmot-kmp-imports.sh --check from check, beside the
M3, brand and wallet audits. Verified both ways: silent on a clean tree, and on a tree
where one import is put back it names the file and the line and fails the build with a
pointer to this note. That is the whole point of C2 — a cherry-pick from a Bennu still on
quartz 1.15.2 brings the old spelling back, and the resolution is to re-run the rule, which
is advice until a build failure makes it a step.
The app's own copies: nothing to delete. CommitOrdering is still live, in
NostrDao's reindex sweep and in MarmotInboundManager's epoch tracker, and marmot-kmp
has no CommitOrdering at all — the convergence engine replaced the idea rather than the
class. Under D4 the legacy pipeline keeps its behaviour, so the comparator stays until the
legacy profile retires with it. The KDoc that used to cite
com.vitorpamplona.quartz.marmot.mls.group.MlsGroup now cites mls-kmp's, and the sentence
about quartz 1.14.0's own CommitOrdering stays as the historical note it always was.
The sender ratchet record: a schema change, and it should be its own. The store's
saveSenderRatchet/loadSenderRatchet default to false/null, and the manager then
falls back to a full save() — which is correct, just expensive: it re-encodes the
ratchet tree, the group context and every epoch secret to persist a counter, per message,
growing with the group. Implementing the pair means a new table, so schema 36 → 37 and an
AutoMigration, and the interface's own words are that the write MUST be durable before
it returns or a lost position re-emits a generation already used and reuses an AEAD key
and nonce pair. That is a change to a per-message write path with a correctness hazard
attached, and it does not belong in the tail of a migration whose whole discipline was D5:
no new behaviour inside the swap. It is the obvious next task, and it is the same shape as
the receive-side write Phase 5 pinned — worth doing as one piece of work that decides how
often this app persists a group, for both directions at once.
That became Phase 8, as its own piece of work and its own commit, and the reason to do both halves together turned out to be stronger than this paragraph guessed: the send-side record is what stops the row being written, so the receive-side write is what keeps it current afterwards. See below.
Durable ingest dedup: not reachable without MarmotManager.
MarmotIngestDedupStore is taken by the client, not by MarmotInboundProcessor, so the
follow-up marmot-v2.md's Phase 5 asked for is only available by adopting the client — the
question D1 deferred. Phase 3 is where that stopped being hypothetical: a MarmotManager
now runs in this repo's tests and the app's own orchestration agrees with it. What is left
is a decision about what owns the stores and the publisher in the app, not a question
about whether the library's client works.
Phase 8 — how often this app persists a group
The piece Phase 7 deferred, as one change in both directions, because they are one question: a group's state is a few kilobytes of ratchet tree, group context and epoch secrets, and the app was rewriting all of it to record things that moved a counter.
What moves, and what has to be written for it.
| what happened | what actually changed | what the app writes now |
|---|---|---|
| an epoch change — a commit applied, a Welcome processed, a convergence rewind | nearly everything | the whole state, by the manager, as it applies it (unchanged) |
| this device sent an application message | this leaf's own entry in the secret tree (RFC 9420 §9) | a MarmotSenderRatchet row, ~100 bytes |
| this device received an application message | the sender's receive ratchet, and any generations it skipped past | the whole state, by MarmotEngine.processGroupEvent |
| a standalone proposal staged | the pending-proposal pool | the whole state, by the manager (unchanged) |
The send side is the library's own split — MlsGroupStateStore.saveSenderRatchet, in the
spirit of OpenMLS keeping message_secrets apart from group_state. The receive side has
no equivalent and cannot have the same one: the skipped keys are (leafIndex, generation)
pairs for other members' ratchets, they live in MlsGroupState and nowhere else, and
OwnSenderRatchet is this leaf's position only.
The table is MarmotSenderRatchet: one row per room, the record opaque. chatRoomId
as the primary key, and the FK to ChatRoom with CASCADE the way
MarmotRetainedEpochSecret has it. The bytes are stored as the library wrote them, which
is the opposite of what the retained-epoch store does — it decodes RetainedEpochSecrets
into five columns — and the precedent followed instead is MarmotPublishObligation, whose
KDoc says the bytes are "opaque here, because the gate decodes them". The reason to choose
that way round here: the retained-epoch rows already existed, written by the outbound path
for another purpose, and RetainedEpochSecrets is RFC-shaped. OwnSenderRatchet is the
library's own invention, it carries both the handshake and the application chain because
it already grew once, and a record whose entire job is to never go backwards is safest
when nothing in this app re-encodes it. epoch, leafIndex and applicationGeneration
are kept beside it, read off the same bytes as they are written: they cannot drift from the
blob, and "where does this device think its ratchet is" is the first question a lost
message asks.
Schema 36 → 37, one AutoMigration. A new table, so Room generates it. A room that
existed before the upgrade has no row, which reads as "has not sent since its last full
save" — true of every one of them at that moment, because the build before this one saved
the whole state on every send.
The ordering hazard, which is the real content of this phase. MlsGroupManager
applies the record itself in restoreAll. The app has a second way in — a room joined
after the engine restored is loaded from its row the first time it is asked for, which
is the common way round in a running app — and MarmotEngine.groupFor is that path. It
has to apply the record before groupManager.adoptGroup, because adopting persists,
and persisting rewrites the ratchet row from the group being adopted. Apply it afterwards
and the row has already been overwritten with the restored state's older position: the
group in memory looks right for the rest of the process, and the next restart re-emits
generations that have already been published, under AEAD keys and nonces that have already
been used.
SenderRatchetPersistenceJvmTest has it both ways round, and finding the test that could
tell them apart took two goes. The first version asserted the row after a lazy load and
passed with the order inverted — because a state saved before this leaf ever sent has no
ratchet entry at all, exportOwnSenderRatchet returns null for it, and the adopt writes
nothing. The order cannot matter until the stored state holds a position of its own that is
behind the record's, and what puts one there is a receive: it is a full save, so it
captures wherever the sender ratchet had got to, and the sends after it move the record
past that. With one received message in the middle the inverted order fails with
expected:<2> but was:<1>, which is the rewind, in the row, in one line.
The same file has the A/B that says the record is load-bearing at all: with the row, a restarted engine's next message is read by the peer; with the row deleted, the same message is refused, because the peer's ratchet for this leaf has consumed that generation and RFC 9420 §9 does not allow it twice.
A correction to Phase 5. That phase pinned the receive-side gap and said a restart
between two messages still lost the late one. That is not what an unwritten row costs
here, and the difference is worth stating because it is the whole reason the original bug
was a bug. quartz's engine saved the ratchet positions and dropped the skipped keys, so
the saved state said the ratchet had passed generation N while carrying nothing for it —
incoherent, and a message lost. mls-kmp writes both or neither, so an unwritten row is
simply an older coherent snapshot, and a snapshot that has not passed the late generation
derives it forward. What the gap actually cost was the deriving — every generation since
the last full save, re-derived after each restart, against SecretTree's thousand-step
forward cap — plus replay protection, which consumedGenerations does not persist in any
case.
So the receive-side write is not a lost-message fix, and this phase does not claim it is.
Its reason is the send side: until this change every send wrote the whole state, so the row
could not fall far behind inside an epoch. Once sends write a ratchet record instead,
nothing updates the row between epoch changes, and a long-lived room could drift an
unbounded number of generations from what this device has actually read. That is what
MarmotEngine.processGroupEvent closes, and it is why the two halves are one change.
One more seam. NostrDao now reads inbound events through MarmotEngine.processGroupEvent
rather than engine.inbound.processGroupEvent, so the write cannot be forgotten by a
second caller; the engine's KDoc says so, and the one test that used the processor directly
was moved to the same door.
Built. 1167 tests across 156 classes, 0 failures; Android and JVM compile clean; the
four audits unchanged. Six new tests — five in SenderRatchetPersistenceJvmTest, one more
in SkippedGenerationJvmTest, whose third test is now inverted from what Phase 5 left and
whose KDoc carries the correction above. IdentitySwitchJvmTest's profile test failed once
during this phase on a 5-second DataStore timeout and did not reproduce in four later runs,
including the full gate; it passes in isolation on this commit and on the one before it,
and its own KDoc says it depends on a DataStore cache that is per-process and per-test-order.
What this phase does not do: encryption at rest. The record holds two 32-byte chain
secrets, so it is exactly as sensitive as the state beside it, and it carries the same
TODO(security) the rest of the store does. Phase 8 moves those bytes into one more table
and does not change what guards them.
Phase 9 — the upgrade, made runnable
Phase 4 settled the blob and left the database alone, because there was no schema change
to worry about. Phase 8 added a table, and that turns "does an existing install still
open?" from a question about MLS into a question about Room. The two fail differently. A
blob the engine cannot read costs one room. A migration Room cannot find costs the app:
every device that has ever had it, on the first launch after the update, with an
IllegalStateException and no rooms at all.
That second failure is also the easiest one in this repository to ship by accident. An
entity added to the @Database list without an AutoMigration beside it compiles, and
it installs perfectly on a phone that has never had the app — a fresh install creates
every table at the current version and never migrates anything. Only a database that is
already there asks for the path that is missing. It was worth knowing how much of the
suite would notice, so the AutoMigration(from = 36, to = 37) line was commented out and
the database tests run: 32 tests, 4 failures, and all four of them the new ones. Every
other migration test and every test over an in-memory database passed with the migration
gone.
So, ShippedProfileUpgradeJvmTest. A v36 database, built on disk from the exported v36
schema — its tables, its indices, its room_master_table identity hash and its
user_version, which together are what make Room read the file as a v36 install rather
than as a corrupt one — with a profile in it, two rooms, and the Phase 4 blob in one of
them. Then the app's own getRoomDatabase over that file, and four claims:
| it opens at v37, with the table this branch adds | user_version 37, the v37 identity hash stamped by Room, and every table's columns and indices equal to 37.json's |
| the legacy room survives and the engine opens it | the blob comes back out of the migrated file byte for byte, MarmotEngine.groupFor restores it, 0xF2EE intact and a 32-byte kind:445 key, while the room beside it with no state comes back as no group |
| opening the profile rewrites nothing | the restore picks the room up and leaves its bytes exactly as they were |
| the added table takes a row keyed to a room that predates it | the one direction an added table's foreign key can be wrong in: it points at ChatRoom, which was already there with rows in it |
The third one is the one worth arguing for, because it asserts an absence. Restoring a group re-derives its key schedule and could just as well save the result at the current state version — and the first thing that persists each room will do exactly that, which is what Phase 4's re-save test pins. Doing it to every room at once, on the first launch after an upgrade, is a different thing: it is the one moment when every room's only copy is being rewritten, and a crash in the middle of it has nothing underneath. The absence is the claim, so the test first asserts the restore actually held the room — otherwise it would be the absence of a write by an engine holding nothing, which is not the same sentence.
One gotcha, which cost the first run. Room.databaseBuilder(...).build() does not
touch the file. Room opens lazily and the migration runs on the first query, so a test
that builds a database and closes it has migrated nothing; the helper has to ask it
something. The first version of this test did exactly that and failed on user_version
still reading 36.
And the fixture moved. The Phase 4 blob is now OldEngineBlob in commonTest, used
by both tests rather than copied into the second. The point of those bytes is that they
are old ones, and a blob living in two files can be regenerated in one of them — after
which the copy that was left alone is the only evidence, with nothing on either saying
which is which.
Built. 1171 tests across 157 classes, 0 failures; the four audits unchanged. The tool
in docs/tools/old-state-blob/ is untouched: its job was always provenance for those
bytes, and this phase gives them a second reader rather than a new source.
What is still manual. A fixture is a profile with two rooms in it, one of them a group
at epoch 0 with a single member. A real profile has rooms that have been through several
epochs, several members and a retained-epoch window, and rows in forty-odd other tables
that this plants placeholders in. The schema path is now covered on every launch of
./gradlew check; the data is not, and opening one real profile is still the last check
before a release.
Out of scope
marmot-quic and agent text streams. MIP-04 encrypted media and the Blossom image
components, which marmot-kmp has and the app has no screen for. MarmotManager as the
app's engine — Phase 3 tries it in a test and Phase 7 names the reason to want it, and
that is as far as this plan goes. Moving quartz past 2.7.3, which is D2's other half and
needs marmot-kmp to move first.
Where the numbers come from
Counted on the worktree at 8b066795:
grep -rl 'import com.vitorpamplona.quartz.marmot' --include='*.kt' composeApp/src | wc -l # 57 files
grep -rh 'import com.vitorpamplona.quartz.marmot' --include='*.kt' composeApp/src | wc -l # 209 lines
grep -rh 'import com.vitorpamplona' --include='*.kt' composeApp/src | grep -v quartz.marmot | wc -l # 854
The destinations were resolved by matching each of the 57 distinct symbols against
checkouts of marmot-kmp e47856e, mls-kmp 9702762 and quartz v2.15.1, and the
published coordinates and their transitive dependencies against the poms on Maven Central.