feat: the sender ratchet as its own row, and the receive side beside it (marmot-kmp, Phase 8)

The piece Phase 7 deferred, in both directions, because they are one question:
a group's state is a few kilobytes of ratchet tree, group context and epoch
secrets, and the app was rewriting all of it to record things that moved a
counter.

Sending an application message advances exactly one thing -- this leaf's entry
in the secret tree (RFC 9420 §9) -- so the library carries the position as its
own record, `MlsGroupStateStore.saveSenderRatchet`, in the spirit of OpenMLS
keeping `message_secrets` apart from `group_state`. Giza defaulted that to
false, which sent the manager back to a full save per message. It is now a
`MarmotSenderRatchet` row, one per room, schema 36 -> 37 with an AutoMigration.
The bytes are stored as the library wrote them -- the `MarmotPublishObligation`
precedent, not the retained-epoch one -- because a record whose whole job is to
never go backwards is safest when nothing here re-encodes it; the epoch, leaf
and generation sit beside it, read off those same bytes, so they cannot drift.

The hazard this introduces is an ordering, and it is in the app's code, not the
library's. `MlsGroupManager.restoreAll` applies the record itself. A room joined
after the engine restored is loaded from its row by `MarmotEngine.groupFor`
instead, and that has to apply the record BEFORE `adoptGroup`, because adopting
persists and persisting rewrites the row from the group being adopted. The other
way round, the next restart re-emits generations already published, under AEAD
keys and nonces already used.

Finding the test that could tell the two orders apart took two goes. The first
asserted the row after a lazy load and passed with the order inverted: a state
saved before this leaf ever sent has no ratchet entry, `exportOwnSenderRatchet`
returns null, and the adopt writes nothing. The order cannot matter until the
stored state holds a position of its own behind the record's, and a received
message is what puts one there. With one receive in the middle the inverted
order fails with expected:<2> but was:<1>.

The receive side is a full save, from `MarmotEngine.processGroupEvent`, which
`NostrDao` now reads inbound events through so a second caller cannot forget it.
And a correction to Phase 5, which said an unwritten row still lost the late
message. It does not: mls-kmp writes the skipped keys beside the positions they
belong to, so an unwritten row is an older COHERENT state, and a position that
has not passed the late generation derives it forward. quartz's engine saved the
positions and dropped the keys -- that pair is what lost messages. The cost of
an unwritten row is the re-deriving, against SecretTree's thousand-step cap. The
reason the write lands now is the send side: until this change every send wrote
the whole state, so the row could not fall far behind inside an epoch.

1167 tests across 156 classes, 0 failures; Android and JVM compile clean; the
four audits unchanged. docs/mls-skipped-keys.md loses its mitigation claim and
gains the correction; docs/marmot-kmp.md gains Phase 8.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kgothatso NgakoandClaude Opus 5 committed 2026-11-19 09:55:31 +02:00
1 parent 7fb56b8f8f
commit e07bb68014
13 files changed
+5263 -93

No files matched your search

@@ -33,6 +33,7 @@ import mobi.giza.compose.database.dao.MarmotRoutingDao
import mobi.giza.compose.database.dao.MarmotPublishObligationDao
import mobi.giza.compose.database.dao.MarmotKeyPackageDao
import mobi.giza.compose.database.dao.MarmotRetainedEpochSecretDao
import mobi.giza.compose.database.dao.MarmotSenderRatchetDao
import mobi.giza.compose.database.dao.MentionDao
import mobi.giza.compose.database.dao.NegentropySynchronizeRequestDao
import mobi.giza.compose.database.dao.NegentropySynchronizeResultDao
@@ -76,6 +77,7 @@ import mobi.giza.compose.database.model.MarmotPublishObligation
import mobi.giza.compose.database.model.MarmotOutboundGate
import mobi.giza.compose.managers.MarmotEngine
import mobi.giza.compose.database.model.MarmotRetainedEpochSecret
import mobi.giza.compose.database.model.MarmotSenderRatchet
import mobi.giza.compose.database.model.Mention
import mobi.giza.compose.database.model.NegentropySynchronizeRequest
import mobi.giza.compose.database.model.NegentropySynchronizeResult
@@ -132,6 +134,7 @@ val GENESIS_AT = Instant.fromEpochMilliseconds(1231006505000L)
MarmotPublishObligation::class,
MarmotOutboundGate::class,
MarmotRetainedEpochSecret::class,
MarmotSenderRatchet::class,
Mention::class,
NegentropySynchronizeRequest::class,
NegentropySynchronizeResult::class,
@@ -150,7 +153,7 @@ val GENESIS_AT = Instant.fromEpochMilliseconds(1231006505000L)
UnsignedNostrEvent::class,
Zap::class
],
version = 36,
version = 37,
autoMigrations = [
// v2 only adds the DkgSession/DkgParticipantMessage tables, so Room can
// generate the migration itself — nothing existing changes shape.
@@ -385,6 +388,14 @@ val GENESIS_AT = Instant.fromEpochMilliseconds(1231006505000L)
// v36 removes the wallets (docs/giza-wallet-removal.md, Phase 5): sixteen tables, three
// columns and the chat lines and signing sessions they left. Manual, in
// PlatformDatabaseBuilder -- see RemoveWalletsMigration.
//
// v37 adds the MarmotSenderRatchet table: where this device's own sender ratchet has
// got to in a current-profile room, as its own row instead of a whole group state
// rewritten per sent message (docs/marmot-kmp.md, Phase 8). A new table, so Room
// migrates it itself. A room that existed before this has no row, which reads as "has
// not sent since its last full save" -- true of every one of them at the moment of the
// upgrade, because the build before this one saved the whole state on every send.
AutoMigration(from = 36, to = 37),
]
)
@ConstructedBy(GizaDatabaseConstructor::class)
@@ -452,6 +463,8 @@ abstract class GizaDatabase: RoomDatabase() {
abstract fun marmotRetainedEpochSecretDao(): MarmotRetainedEpochSecretDao
abstract fun marmotSenderRatchetDao(): MarmotSenderRatchetDao
abstract fun mentionDao(): MentionDao
abstract fun negentropySynchronizeRequestDao(): NegentropySynchronizeRequestDao
@@ -0,0 +1,23 @@
package mobi.giza.compose.database.dao
import androidx.room3.Dao
import androidx.room3.Query
import androidx.room3.Upsert
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import mobi.giza.compose.database.model.MarmotSenderRatchet
@Dao
interface MarmotSenderRatchetDao {
@Query("SELECT * FROM MarmotSenderRatchet WHERE chatRoomId = :chatRoomId")
suspend fun forChatRoomId(chatRoomId: HexKey): MarmotSenderRatchet?
/**
* Upsert rather than insert: a send overwrites the position its predecessor
* wrote, and there is one row per room for the life of the room.
*/
@Upsert
suspend fun upsert(marmotSenderRatchet: MarmotSenderRatchet)
@Query("DELETE FROM MarmotSenderRatchet WHERE chatRoomId = :chatRoomId")
suspend fun deleteForChatRoomId(chatRoomId: HexKey)
}
@@ -1402,9 +1402,11 @@ abstract class NostrDao(
upsertMarmotGroupEvent(groupEvent, chatRoomId, activeKeyPair)
// The processor names the group by its address; every row below is keyed by
// the room, so the result is re-addressed once.
val groupEventResult = engine.inbound.processGroupEvent(groupEvent).inRoom(chatRoomId)
// Through the engine, not the processor: an application message is the one
// result that moves state the library writes nothing for, and the engine is
// where that write is. The processor names the group by its address; every
// row below is keyed by the room, so the result is re-addressed once.
val groupEventResult = engine.processGroupEvent(groupEvent).inRoom(chatRoomId)
logger.d("groupEventResult: $groupEventResult")
fileGroupEventResult(
@@ -0,0 +1,76 @@
package mobi.giza.compose.database.model
import androidx.room3.Entity
import androidx.room3.ForeignKey
import androidx.room3.PrimaryKey
import com.vitorpamplona.quartz.nip01Core.core.HexKey
/**
* Where this device's own sender ratchet has got to in a current-profile room.
*
* Sending an application message advances exactly one thing in an MLS group: this
* leaf's entry in the secret tree (RFC 9420 §9). The ratchet tree, the group
* context, the epoch secrets and the path keys are all byte for byte what they
* were. So the library carries the position as its own small record rather than
* rewriting the group to hold it -- `MlsGroupStateStore.saveSenderRatchet`, in the
* spirit of OpenMLS keeping `message_secrets` apart from `group_state` -- and this
* is where Giza puts it.
*
* It is a safety record, not a cache. A position that is lost re-emits a generation
* this leaf has already published, which reuses an AEAD key and nonce pair; that is
* the one failure the record exists to prevent, and it is why the write has to
* happen before the ciphertext it produced is broadcast. The library makes applying
* one safe from the other direction: a record is accepted only for the epoch, leaf
* and secret tree it was taken from, and then only to move a chain *forward*.
*
* [record] is the library's own `OwnSenderRatchet.encodeTls()` and is the truth
* here, opaque the way [MarmotPublishObligation.bytes] is: nothing in this app
* re-encodes it, so a field the library adds cannot be dropped on the way through.
* [epoch], [leafIndex] and [applicationGeneration] are that record's own numbers,
* read off it as it is written and kept beside it to be read -- "where does this
* device think its ratchet is" is the first question a lost message asks, and a blob
* answers nothing.
*
* One row per room: the manager keeps one position per group, and writes it under
* the lock it encrypts under, so the row only ever moves forward.
*
* See docs/marmot-kmp.md, Phase 8, and docs/mls-skipped-keys.md.
*/
@Entity(
foreignKeys = [
ForeignKey(
entity = ChatRoom::class,
parentColumns = ["id"],
childColumns = ["chatRoomId"],
onDelete = ForeignKey.CASCADE,
),
],
)
data class MarmotSenderRatchet(
@PrimaryKey
val chatRoomId: HexKey,
val epoch: Long,
val leafIndex: Int,
val applicationGeneration: Int,
val record: ByteArray, // TODO: Encrypt this...
) {
override fun equals(other: Any?): Boolean =
this === other ||
(
other is MarmotSenderRatchet &&
chatRoomId == other.chatRoomId &&
epoch == other.epoch &&
leafIndex == other.leafIndex &&
applicationGeneration == other.applicationGeneration &&
record.contentEquals(other.record)
)
override fun hashCode(): Int {
var result = chatRoomId.hashCode()
result = 31 * result + epoch.hashCode()
result = 31 * result + leafIndex
result = 31 * result + applicationGeneration
result = 31 * result + record.contentHashCode()
return result
}
}
@@ -11,6 +11,8 @@ import com.vitorpamplona.marmot.protocolCore.MarmotPublishGate
import com.vitorpamplona.marmot.protocolCore.PublishOutcome
import com.vitorpamplona.mls.group.MlsGroup
import com.vitorpamplona.mls.group.MlsGroupState
import com.vitorpamplona.mls.group.OwnSenderRatchet
import com.vitorpamplona.quartz.marmot.mip03GroupMessages.GroupEvent
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
@@ -51,15 +53,23 @@ import mobi.giza.compose.exceptions.MarmotConvergencePendingException
* and matched by reference since Phase 1, and the processor reads the rotation
* manager only on its own Welcome path, which the app does not use.
*
* Two gaps the app's own pipeline had are not closed by moving to this one, and
* should not be read as closed. A late application message decrypted under a
* retained prior epoch is decrypted without the sender's leaf signature being
* verified -- the library's `tryDecryptWithRetainedEpoch` has the same gap the
* app's had. And the secret tree's skipped keys reach the state the manager would
* save -- mls-kmp carries them, which quartz's engine did not -- but an application
* message is decrypted without anything being written, so they still do not reach
* the row (docs/mls-skipped-keys.md). The manager holding the live group covers
* that within a process, exactly as `MlsGroupCache` did, and no further.
* One gap the app's own pipeline had is not closed by moving to this one, and should
* not be read as closed: a late application message decrypted under a retained prior
* epoch is decrypted without the sender's leaf signature being verified -- the
* library's `tryDecryptWithRetainedEpoch` has the same gap the app's had.
*
* The other one is closed. The secret tree's skipped keys reach the state the
* manager would save -- mls-kmp carries them, which quartz's engine did not -- and
* [processGroupEvent] writes that state after an application message, so they reach
* the row as well (docs/mls-skipped-keys.md). Read inbound events through that
* rather than through [inbound] directly, or the write is lost.
*
* Which state is written where is now split by what moved. An epoch change is a full
* save, by the manager, as it applies. An application message received is a full
* save, here. An application message *sent* moves only this leaf's place in the
* secret tree, and that is a `MarmotSenderRatchet` row rather than the group --
* which is why [groupFor] has to put the record back before the manager adopts a
* group, and says there what goes wrong if it does not.
*/
class MarmotEngine(
private val database: GizaDatabase,
@@ -113,11 +123,82 @@ class MarmotEngine(
groupManager.getGroup(routingId)?.let { return it }
val bytes = store.load(routingId) ?: return null
val group = MlsGroup.restore(MlsGroupState.decodeTls(bytes), MarmotGroupPolicy)
applySenderRatchetRecord(routingId, group)
groupManager.adoptGroup(routingId, group)
inbound.trackGroup(routingId)
return group
}
/**
* Move [group] up to the sender ratchet position its own row records, as
* `MlsGroupManager.restoreAll` does for the groups it loads.
*
* **Before `adoptGroup`, not after.** Adopting persists, and persisting rewrites
* the ratchet row from the group it was handed -- so a group adopted at the
* restored state's older position overwrites the record with that position, and
* the generations between the two are published a second time under keys and
* nonces already used. This is the half of the split the library cannot do for
* the app: `restoreAll` is its own path and applies the record itself, while this
* one is the app's, for a room first touched after that restore.
*
* The library decides whether the record applies at all: it is accepted only for
* the epoch, the leaf and the exact secret tree it was taken from, and then only
* where it moves a chain forward. So a record that does not belong to this state
* is simply not applied.
*/
private suspend fun applySenderRatchetRecord(routingId: HexKey, group: MlsGroup) {
val bytes = store.loadSenderRatchet(routingId) ?: return
val record = OwnSenderRatchet.decodeTlsOrNull(bytes) ?: return
if (group.restoreOwnSenderRatchet(record)) {
logger.d("${routingId.take(8)}… resumes at application generation ${record.applicationGeneration}")
}
}
/**
* Read one kind:445 against this database's groups, and write what the library
* leaves in memory.
*
* The processor's own persistence is the manager's: a commit that advanced the
* epoch is saved as it is applied, a staged proposal is saved as it is staged.
* An application message is the one result that moves state and writes nothing,
* because the manager has no small record for the receive side the way it has for
* the send side -- decrypting generation N+1 before N derives N's key and keeps it
* (RFC 9420 §9.1), and those keys are `(leafIndex, generation)` pairs for other
* members' ratchets, which live in the group state and nowhere else.
*
* Not writing them does not lose the late message, and it is worth being exact
* about that, because the opposite is what the original bug was.
* `MlsGroupState` carries the skipped keys beside the ratchet positions they
* belong to, so a state saved without a receive is an older *coherent* state, and
* a position that has not passed the late generation derives it forward. quartz's
* engine saved the positions and dropped the keys, which is the pair that lost
* messages. What an unwritten row costs is the deriving -- every generation since
* the last full save, re-derived on each restart, against `SecretTree`'s
* thousand-step forward cap.
*
* The reason the write is here is the send side. Until the sender ratchet became
* its own row, every sent message wrote the whole state, so the row could not
* fall far behind inside an epoch; now nothing else would update it between epoch
* changes, and a long-lived room would drift. It is a full save, which is the cost
* the legacy pipeline has always paid -- `MlsGroupCache` saves the whole state
* after every inbound event -- and it is paid on the receive path, where the
* device is already doing an AEAD open and a database write for the event itself.
*
* The snapshot is taken through `readGroup`, under the lock the manager applies
* commits under, so it cannot catch a commit half applied.
*
* See docs/mls-skipped-keys.md and docs/marmot-kmp.md, Phase 8.
*/
suspend fun processGroupEvent(groupEvent: GroupEvent): GroupEventResult {
ensureRestored()
val result = inbound.processGroupEvent(groupEvent)
if (result is GroupEventResult.ApplicationMessage) {
val encoded = groupManager.readGroup(result.groupId) { it.saveState().encodeTls() }
if (encoded != null) store.save(result.groupId, encoded)
}
return result
}
/**
* A relay accepted the commit behind [obligationId]: the pending state becomes
* canonical, and convergence learns of the commit we authored -- without which a
@@ -4,10 +4,12 @@ import co.touchlab.kermit.Logger
import com.vitorpamplona.marmot.groups.MlsGroupStateStore
import com.vitorpamplona.mls.codec.TlsReader
import com.vitorpamplona.mls.codec.TlsWriter
import com.vitorpamplona.mls.group.OwnSenderRatchet
import com.vitorpamplona.mls.group.RetainedEpochSecrets
import com.vitorpamplona.quartz.nip01Core.core.hexToByteArray
import mobi.giza.compose.database.GizaDatabase
import mobi.giza.compose.database.model.MarmotRetainedEpochSecret
import mobi.giza.compose.database.model.MarmotSenderRatchet
import mobi.giza.compose.extensions.toHex
/**
@@ -20,6 +22,8 @@ import mobi.giza.compose.extensions.toHex
* retained epochs are `MarmotRetainedEpochSecret` rows, one per epoch, which until
* this store were written by the outbound path and read back by nothing -- every
* epoch the inbound path retained was a map in an `object`, gone at the next launch.
* The sender ratchet is a `MarmotSenderRatchet` row, one per room, and is the one
* thing here that is written on the send path instead of the full state.
*
* A save for a room that does not exist yet is dropped with a log rather than
* creating one: the room row is what carries the state when a room is made, and
@@ -55,6 +59,10 @@ class RoomMlsGroupStateStore(
val chatRoomId = chatRoomIdFor(nostrGroupId)
database.chatRoomDao().updateMlsGroupState(chatRoomId, null)
database.marmotRetainedEpochSecretDao().deleteForChatRoomId(chatRoomId)
// A position is only meaningful inside the secret tree that derived it, and
// that tree is what just went. The record would be refused on its binder
// anyway; going now means no restore has to reason about it.
database.marmotSenderRatchetDao().deleteForChatRoomId(chatRoomId)
}
override suspend fun listGroups(): List<String> =
@@ -98,4 +106,59 @@ class RoomMlsGroupStateStore(
).encodeTls(writer)
writer.toByteArray()
}
/**
* The sender ratchet position as its own row, which is the whole point: a send
* advances this leaf's entry in the secret tree and nothing else, so the
* alternative is re-encoding the ratchet tree, the group context and every epoch
* secret per message, growing with the group.
*
* Returns false for a room with no row, which sends the manager back to a full
* [save] -- the one place that can create the state, and it drops a stateless
* room with a log for the same reason. Returning false for an unreadable record
* does the same: the interface's fallback is correct-but-slower, never
* correct-but-lossy, so anything this cannot store honestly is declined rather
* than reported saved.
*
* Durability is Room's, which is what [save] has always had: the suspend returns
* when the write is in the write-ahead log. The interface asks for the write to
* be durable before it returns because the ciphertext it describes is about to be
* published, and that is the guarantee this inherits -- no stronger.
*/
override suspend fun saveSenderRatchet(nostrGroupId: String, state: ByteArray): Boolean {
val chatRoomId = chatRoomIdFor(nostrGroupId)
if (!database.chatRoomDao().exists(chatRoomId)) {
logger.w("No room for $nostrGroupId; its sender ratchet is not saved")
return false
}
val record = OwnSenderRatchet.decodeTlsOrNull(state)
if (record == null) {
logger.w("The sender ratchet record for $nostrGroupId did not decode; falling back to a full save")
return false
}
database.marmotSenderRatchetDao().upsert(
MarmotSenderRatchet(
chatRoomId = chatRoomId,
epoch = record.epoch,
leafIndex = record.leafIndex,
applicationGeneration = record.applicationGeneration,
record = state,
)
)
return true
}
/**
* The bytes as they were written. Null for a room that has not sent since its
* last full save, which is every room the moment this table arrives.
*
* The manager applies what comes back on top of the restored state, and the
* library accepts it only for the epoch, leaf and secret tree it was taken from,
* and then only to move a chain forward. So a stale row is inert rather than
* harmful -- but see `MarmotEngine.groupFor`, which has to apply it before the
* manager adopts the group, because adopting persists and would otherwise write
* the restored state's older position over this one.
*/
override suspend fun loadSenderRatchet(nostrGroupId: String): ByteArray? =
database.marmotSenderRatchetDao().forChatRoomId(chatRoomIdFor(nostrGroupId))?.record
}