mantra/mantra-kmp
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat: rebuild the group's signed record out of the rows it left behind

Browse Source 
Phase 3 of docs/member-archive.md. `ArchiveManager.assemble` walks a room's rows,
rebuilds each into the event the group signed, drops anything it cannot prove,
and cuts the rest into pages. Nothing sends one yet.

**The gate found a real bug, which is why it was the gate.** Signed events are
not stored as events -- `FrostSigningManager.complete` applies one and what
survives is a `Mantra*` row -- so an archive has to rebuild them with `toXEvent()`
and stands or falls on that being byte-identical to what was signed. Every
`toXEvent()` in the codebase turned out to be unused in production, written for
exactly this and never called, so the "tag order matches build so the event id
round-trips" comments on them were claims nothing had ever checked.

One was wrong. `MantraArtifact.toArtifactEvent` put the alt tag last where
`ArtifactEvent.build` puts it first, and left out the version metadata tag
altogether -- because that tag is not on the artifact row at all.
`fromArtifactEvent` reads the artifact's own fields and drops the version label,
which `applyInnerEvent` has by then turned into the artifact's first
`MantraArtifactVersion`. So the label is now a parameter, read off the initial
version: the one whose `createdAt` is the artifact's, since `initialVersionOf`
derives it from the same event.

Neither fault would have surfaced as an error. Both produce a well-formed
artifact whose id no longer matches its fields, which every receiver drops as a
forgery, silently, one kind at a time. `ArchiveRoundTripTest` now signs each
archivable kind with a real quorum, files it as a row, rebuilds it and asserts
the signature still covers what comes out -- plus the negative case, that
rebuilding with the wrong version label fails as a forgery rather than as a
mistake, which is why the assembler reads the label rather than defaulting it.

**The allowlist narrows from nine kinds to six, and this is the finding to read.**
Only six of the thirteen nip30303 kinds ever reach a signing session; the rest
travel as member rumors, vouched for by the MLS frame they arrived in and by
nothing that survives leaving it. An artifact version is derived rather than
signed -- which is fine, because applying the archived artifact derives it again
and the chapters hanging off it keep their foreign key. Nothing builds a
`TranslationEvent` at all. The contributor lists have no arm in `applyInnerEvent`
that writes a row.

And `TranslationChunkEvent` -- **the translated text itself** -- is submitted by
`MantraDao.saveTranslation` as its author's rumor, because a translation is one
member's work rather than a group decision. So an archive restores everything a
translation hangs on and not the translation: a new member gets the dialects, the
artifacts, the chapters, the source chunks, which translations exist and their
chapter scaffolding, and none of the prose. That is a real limit rather than a
detail, so it is written into the allowlist's own doc comment, into the plan's
"what this does not do", and into a test named after it -- with the three ways
out sketched and none of them taken here, because the cheapest gives up the
property the rest of this rests on and the best is a product decision about
whether translating is an act of the group or of a member.

**Nothing unverifiable leaves.** Every rebuilt event is checked with
`isSignedByRoom` against the same room id the recipient will use. Not politeness
-- the receiver checks anyway -- but so the page count says what will actually
arrive: a row from a member's rumor is dropped here rather than by the recipient.

**Walked down the tree, not queried per kind.** Only dialects and artifacts have
a by-room query and the rest hang off a parent, and the walk is also what puts an
artifact's version label within reach. Order is settled afterwards by
`inApplyOrder` rather than by the walk, since the walk groups by artifact and the
foreign keys are by kind.

**Paging is greedy against both caps**, because they bind different archives: a
room of one-line dialects hits the count first and a room of chapters hits the
bytes. An event too large for a page of its own is dropped with a log rather than
failing the archive -- a chapter nobody can archive is a hole, a member who gets
nothing is a bigger one.

Assembling only; queueing moved to Phase 5, where the thing that decides when to
send lives. That keeps this testable against a real database with no outbound
path in the way.

Seven tests over a real in-memory database seeded through `applyInnerEvent`
itself, so what is archived is what a member's device really holds rather than
rows built to suit the test: every payload verifies, all six kinds appear exactly
as often as they were signed, the whole archive is in dependency order end to
end, a member's unsigned dialect sitting in the same room is left out, an empty
room archives nothing without failing, and two archives of identical rows do not
share an id -- which is what stops two members answering one request from having
their pages counted towards each other's total.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kgothatso Ngako
2026-09-06 14:12:29 +02:00
parent 6e04f6c7af
commit acff66a22e
7 changed files with 1107 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
composeApp/src/commonMain/kotlin/press/mantra/compose/database/model/MantraArtifact.kt
View File

@@ -11,6 +11,7 @@ import com.vitorpamplona.quartz.nip31Alts.AltTag
import press.mantra.compose.database.model.traits.OptionalNostrEventEntity
import press.mantra.compose.database.model.traits.TimestampedEntity
import press.mantra.compose.nostr.nip30303.ArtifactEvent
import press.mantra.compose.nostr.nip30303.tags.ArtifactVersionMetadataTag
import press.mantra.compose.nostr.nip30303.tags.DialectIdTag
import press.mantra.compose.nostr.nip30303.tags.LicenseTag
import press.mantra.compose.nostr.nip30303.tags.UrlTag
@@ -65,12 +66,34 @@ data class MantraArtifact(
override val createdAt: Instant = Clock.System.now(),
override val updatedAt: Instant = createdAt
): OptionalNostrEventEntity, TimestampedEntity {
fun toArtifactEvent(): ArtifactEvent {
/**
* The event the group signed, rebuilt from this row.
*
* [versionLabel] is a parameter because it is not on this row and cannot be.
* `fromArtifactEvent` reads the artifact's own fields and drops the version
* metadata, which `ChatMessage.applyInnerEvent` has by then turned into the
* artifact's first `MantraArtifactVersion`. It was still part of what the
* group put its signature to, so a rebuild without it hashes to a different
* id and produces an event the signature does not cover -- which a receiver
* reads as a forgery, silently. Recover it from the initial version: the one
* whose `createdAt` is this artifact's, since `initialVersionOf` derives it
* from the same event.
*
* Tag order matches [ArtifactEvent.build] exactly -- alt first -- for the
* same reason, and `ArchiveRoundTripTest` is what says so. Both of those were
* wrong here until an archive needed to rebuild an artifact and nothing had
* ever called this.
*/
fun toArtifactEvent(versionLabel: String): ArtifactEvent {
return ArtifactEvent(
id = id,
pubKey = publicKey,
createdAt = createdAt.epochSeconds,
// Tag order matches ArtifactEvent.build so the event id round-trips.
tags = TagArrayBuilder<ArtifactEvent>()
.addUnique(
AltTag.assemble(ArtifactEvent.ALT_DESCRIPTION)
)
.addUnique(UrlTag.assemble(url))
.addUnique(VisibilityTag.assemble(visibility))
.addUnique(LicenseTag.assemble(license))
@@ -78,7 +101,7 @@ data class MantraArtifact(
DialectIdTag.assemble(dialectId)
)
.addUnique(
AltTag.assemble(ArtifactEvent.ALT_DESCRIPTION)
ArtifactVersionMetadataTag.assemble(versionLabel)
)
.build(),
content = name,

212
composeApp/src/commonMain/kotlin/press/mantra/compose/managers/ArchiveManager.kt Normal file
View File

@@ -0,0 +1,212 @@
package press.mantra.compose.managers
import co.touchlab.kermit.Logger
import com.vitorpamplona.quartz.nip01Core.core.Event
import com.vitorpamplona.quartz.nip01Core.core.HexKey
import com.vitorpamplona.quartz.nip01Core.signers.EventTemplate
import com.vitorpamplona.quartz.utils.RandomInstance
import com.vitorpamplona.quartz.utils.TimeUtils
import press.mantra.compose.database.MantraDatabase
import press.mantra.compose.extensions.toHex
import press.mantra.compose.nostr.archive.ArchiveEvent
import press.mantra.compose.nostr.frost.GroupKeyStateEvent
/**
* Building the group's signed record into pages a member who lacks it can apply.
*
* See docs/member-archive.md. The short of it: a member added after the work was
* done has none of it and never will, because a group-signed event is applied
* locally by each device that took part and never goes on the wire. This is how
* it gets to them.
*
* ### The rows are the archive
*
* Signed events are not stored as events. `FrostSigningManager.complete` applies
* one and what survives is a `Mantra*` row, so every payload here is rebuilt with
* `toXEvent()` and stands or falls on that rebuild being byte-identical to what
* was signed. `ArchiveRoundTripTest` is what says it is, per kind, against a real
* quorum -- and it found two faults in the artifact's rebuild the first time it
* ran, both of which would have shipped payloads that every receiver drops as
* forgeries without a word.
*
* ### Nothing unverifiable leaves
*
* Every rebuilt event is checked with [GroupKeyStateEvent.isSignedByRoom] before
* it is packed, against the same room id the recipient will check it with. That
* is not politeness towards the receiver, who checks anyway. It is what keeps an
* archive honest about its own size: a row that came from a member's rumor
* cannot be archived, and dropping it here rather than letting the recipient
* drop it means the page count says what will actually arrive.
*/
object ArchiveManager {
private const val TAG = "ArchiveManager"
private val logger = Logger.withTag(TAG)
/**
* Every group-signed event this device holds for [chatRoomId], paged and
* addressed to [recipient].
*
* Empty when there is nothing to send -- a room with no shared key, a room
* whose work is all member rumors, or a device that is itself behind. An
* empty result is not an error and the caller should not report one: the
* honest answer to "send them the history" in a room with no signed history
* is nothing.
*/
suspend fun assemble(
database: MantraDatabase,
chatRoomId: String,
recipient: HexKey,
archiveId: String = RandomInstance.bytes(32).toHex(),
createdAt: Long = TimeUtils.now(),
): List<EventTemplate<ArchiveEvent>> {
val rebuilt = signedEventsOf(database, chatRoomId)
val verified = rebuilt.filter { GroupKeyStateEvent.isSignedByRoom(it, chatRoomId) }
if (verified.size != rebuilt.size) {
// Expected rather than alarming: an artifact version is derived
// rather than signed, and a translation is its author's rumor. What
// would be worth looking at is this dropping something the group
// really did sign, which reads as a broken `toXEvent` rather than as
// a missing signature.
logger.d(
"Leaving ${rebuilt.size - verified.size} of ${rebuilt.size} row(s) out of " +
"$chatRoomId's archive: nothing verifiably signed by the room"
)
}
val pages = paginate(ArchiveEvent.inApplyOrder(verified))
if (pages.isEmpty()) {
logger.i("Nothing signed to archive for room $chatRoomId")
return emptyList()
}
logger.i(
"Archiving ${verified.size} event(s) for $chatRoomId as $archiveId, " +
"${pages.size} page(s) for ${recipient.take(8)}"
)
return pages.mapIndexed { index, payloads ->
ArchiveEvent.build(
payloads = payloads,
archiveId = archiveId,
index = index,
count = pages.size,
recipient = recipient,
createdAt = createdAt,
)
}
}
/**
* The room's rows, rebuilt into the events they came from.
*
* Walked down the tree rather than queried per kind, because only dialects
* and artifacts have a by-room query and the rest hang off a parent. The walk
* is also what makes an artifact's version label reachable: it is not on the
* artifact row -- see `MantraArtifact.toArtifactEvent` -- and the version it
* went into is one step away here.
*
* Order does not matter at this point; [ArchiveEvent.inApplyOrder] settles it
* afterwards. What matters is that nothing is missed, so this returns
* everything and the verify filter above decides what can travel.
*/
private suspend fun signedEventsOf(
database: MantraDatabase,
chatRoomId: String,
): List<Event> = buildList {
database.mantraDialectDao().getDialectsByChatRoomId(chatRoomId).forEach {
add(it.toDialectEvent())
}
database.mantraArtifactDao().getArtifactsByChatRoomId(chatRoomId).forEach { artifact ->
val versions = database.mantraArtifactVersionDao()
.getArtifactVersionsByArtifactId(artifact.id)
// The version an artifact starts life with carries the label the
// artifact was signed with, and `initialVersionOf` gives it the
// artifact's own timestamp. A row with none of those is an artifact
// this device cannot rebuild, which is a gap in the archive rather
// than a reason to abandon it.
val versionLabel = versions
.firstOrNull { it.createdAt == artifact.createdAt }
?.versionLabel
if (versionLabel == null) {
logger.w("Artifact ${artifact.id} has no initial version; leaving it out")
} else {
add(artifact.toArtifactEvent(versionLabel = versionLabel))
}
versions.forEach { version ->
database.mantraChapterDao()
.getChaptersByArtifactVersionId(version.id)
.forEach { chapter ->
add(chapter.toChapterEvent())
database.mantraChunkDao()
.getChunksByChapterId(chapter.id)
.forEach { add(it.toChunkEvent()) }
}
database.mantraTranslationArtifactVersionDao()
.getTranslationsByArtifactVersionId(version.id)
.forEach { translationVersion ->
add(translationVersion.toTranslationArtifactVersionEvent())
database.mantraTranslationChapterDao()
.getTranslationChaptersByTranslationArtifactVersionId(translationVersion.id)
.forEach { add(it.toTranslationChapterEvent()) }
}
}
}
}
/**
* [events] cut into pages that fit, keeping the order they arrive in.
*
* Greedy: fill a page until the next event would cross either cap. Both are
* checked because they bind different archives -- a room of one-line dialects
* hits the count first and a room of chapters hits the bytes.
*
* An event too large to share a page with anything is given one of its own.
* One too large for even that is dropped with a log rather than failing the
* archive: a chapter nobody can archive is a hole, and a member who gets
* nothing at all is a bigger one.
*/
private fun paginate(events: List<Event>): List<List<Event>> {
val pages = mutableListOf<List<Event>>()
var page = mutableListOf<Event>()
// The brackets an empty page already costs.
var bytes = 2
events.forEach { event ->
// Plus the comma this event needs if it is not first on its page.
val size = event.toJson().encodeToByteArray().size + 1
if (2 + size > ArchiveEvent.MAX_PAGE_BYTES) {
logger.w(
"Event ${event.id} is $size bytes and will not fit a " +
"${ArchiveEvent.MAX_PAGE_BYTES}-byte page; leaving it out of the archive"
)
return@forEach
}
val full = page.isNotEmpty() &&
(bytes + size > ArchiveEvent.MAX_PAGE_BYTES || page.size >= ArchiveEvent.MAX_PAGE_EVENTS)
if (full) {
pages.add(page)
page = mutableListOf()
bytes = 2
}
page.add(event)
bytes += size
}
if (page.isNotEmpty()) pages.add(page)
return pages
}
}

40
composeApp/src/commonMain/kotlin/press/mantra/compose/nostr/archive/ArchiveEvent.kt
View File

@@ -15,14 +15,11 @@ import press.mantra.compose.nostr.archive.tags.ArchiveIdTag
import press.mantra.compose.nostr.archive.tags.ArchivePageTag
import press.mantra.compose.nostr.archive.tags.ArchiveRecipientTag
import press.mantra.compose.nostr.nip30303.ArtifactEvent
import press.mantra.compose.nostr.nip30303.ArtifactVersionEvent
import press.mantra.compose.nostr.nip30303.ChapterEvent
import press.mantra.compose.nostr.nip30303.ChunkEvent
import press.mantra.compose.nostr.nip30303.DialectEvent
import press.mantra.compose.nostr.nip30303.TranslationArtifactVersionEvent
import press.mantra.compose.nostr.nip30303.TranslationChapterEvent
import press.mantra.compose.nostr.nip30303.TranslationChunkEvent
import press.mantra.compose.nostr.nip30303.TranslationEvent
/**
* 30327
@@ -165,22 +162,43 @@ class ArchiveEvent(
* what the room signs with, replayed by whoever kept a copy. Nothing but
* this list stops it.
*
* Absent on purpose: the contributor-list kinds (30305, 30307, 30310).
* `ChatMessage.applyInnerEvent` has no arm that writes a row for any of
* them, so archiving them would cost bytes and restore nothing. They
* belong here on the day that changes, and the test that this list is a
* subset of what `applyInnerEvent` handles is what should catch it.
* ### What is missing from it, and why
*
* **Only kinds the group actually signs can be here**, because an archive
* that cannot be verified is one that has to be believed. Six of the
* thirteen nip30303 kinds reach a signing session; the rest travel as
* rumors -- empty signature, member author, vouched for by the MLS frame
* they arrived in and by nothing that survives leaving it.
*
* Left out for that reason, and this is a real limit rather than an
* oversight:
*
* - `ArtifactVersionEvent` (30301). Never signed. The version an
* artifact starts life with is *derived* from the signed artifact by
* `MantraArtifactVersion.initialVersionOf`, which the artifact's own
* arm in `applyInnerEvent` runs -- so archiving the artifact brings its
* version along and the chapters hanging off it keep their foreign
* key. Later versions go through `MantraDao.addArtifactVersion` as
* rumors, and no screen calls it today.
* - `TranslationChunkEvent` (30309) -- **the translated text itself**.
* `MantraDao.saveTranslation` submits it as a member's rumor, because
* a translation is one member's work rather than a group decision.
* So an archive restores everything a translation hangs on and not the
* translation. See docs/member-archive.md for what closing that would
* take; it is not a line in this list.
* - `TranslationEvent` (30311). Nothing builds one; the inbound arm
* exists and no producer does.
* - The contributor lists (30305, 30307, 30310). `applyInnerEvent` has
* no arm that writes a row for any of them, so archiving them would
* cost bytes and restore nothing.
*/
private val APPLY_ORDER: List<Kind> = listOf(
DialectEvent.KIND,
ArtifactEvent.KIND,
ArtifactVersionEvent.KIND,
ChapterEvent.KIND,
ChunkEvent.KIND,
TranslationArtifactVersionEvent.KIND,
TranslationChapterEvent.KIND,
TranslationChunkEvent.KIND,
TranslationEvent.KIND,
)
/** Every kind an archive may carry. */