From 3cdce6a9080728d194999eee50feab0dcdab5a7a Mon Sep 17 00:00:00 2001 From: Kgothatso Ngako Date: Sun, 13 Sep 2026 22:32:30 +0200 Subject: [PATCH] feat(marmot): send the canonical app payload, and an envelope for direct messages (v2 Phase 2) What goes into MLS is now the Marmot app event the spec defines: six members and no `sig`. The rumor serialised as an Event carried `"sig":""`, which a conformant decoder refuses outright -- a payload with a signature would be a standalone relay event, republishable as a signed statement by its author. The id does not move, because NIP-01 never hashed the signature, so every row filed under an inner event's id stays valid; and the build already in the field reads the new shape, because Event.fromJsonOrNull defaults a missing signature. This switch alone needs no gating and is made everywhere at once. The `encoding` tag comes off the application kind:445 while here -- transports/nostr.md forbids it on every shape. docs/marmot-v2.md, Phase 2. A direct message could not make that switch as it was. Its payload is a NIP-59 wrap signed by a throwaway key, which a conformant peer refuses twice over: for the sig member and for an author that is not the MLS sender. In a current-profile room the wrap now travels inside an envelope -- an unsigned app event of kind 30330 authored by the sender, one `p` tag for the recipient, the wrap's JSON as content. The envelope passes the author check like any payload, so the carve-out in mip03Rejection that admitted bare wraps is confined to legacy rooms by a new `legacyRoom` flag: in a current-profile room a bare kind:1059 is refused as every other conformant client refuses it. Every property the throwaway key was chosen for survives -- the envelope is unsigned and so a claim rather than a proof, the wrap inside is still signed by nobody, and the seal-to-leaf binding in classify still decides who wrote the words. The line for a message this device cannot read now hangs off the row the payload was filed under (the envelope's id in a current-profile room), which ChatMessage's foreign key requires. The envelope is gated, and the gate is the room's profile: an old build renders it as a raw-JSON bubble, so a legacy room keeps sending the bare wrap, and a current-profile room -- which admits only leaves carrying an identity proof, which only a build past Phase 3 can make -- never holds an old build. That is the flag for every wire change after this one. A conformant decoder bounds the whole app payload at 64 KiB, and a chronicle page rides inside one as JSON-escaped content. The new bounds test caught what the arithmetic missed: dense JSON payloads escape to half again their size, so a 48 KiB page became a 70 KiB app event. The packer now measures a page by ChronicleEvent.escapedByteSize -- the bytes it will occupy inside the envelope -- against MAX_PAGE_BYTES, lowered from 64 to 48 KiB, and MAX_PAGE_EVENTS drops from 128 to 96 so the count cap still binds under it, as the comment on the two caps insists. The receiver's raw-bytes check is unchanged and now holds by construction. Tests: MarmotMip03CarveOutTest becomes the test that the exemption is confined to legacy rooms and that an envelope passes the ordinary check; MarmotDirectMessageTest round-trips the envelope through the strict decoder and opens the wrap inside; MarmotPayloadBoundsTest asserts the largest chronicle pages and an envelope are conformant app events, including the dense-content page measured the way the packer measures it. The ingest test's tamper case flips a decoded byte rather than the base64 tail, which could change nothing but padding. Full JVM suite: 1,148 tests pass. The direct-message and chronicle notes record the changes. Co-Authored-By: Claude Opus 5 --- .../compose/database/dao/MarmotOutboundDao.kt | 74 +++++++--- .../compose/database/model/ChatMessage.kt | 35 ++++- .../compose/managers/ChronicleManager.kt | 8 +- .../compose/managers/MarmotInboundManager.kt | 41 ++++-- .../compose/nostr/MarmotDirectMessage.kt | 66 ++++++++- .../compose/nostr/chronicle/ChronicleEvent.kt | 46 +++++- .../managers/MarmotMip03CarveOutTest.kt | 70 +++++++--- .../compose/nostr/MarmotDirectMessageTest.kt | 52 +++++++ .../compose/nostr/MarmotPayloadBoundsTest.kt | 131 ++++++++++++++++++ .../dao/MarmotKeyPackageIngestJvmTest.kt | 10 +- docs/marmot-direct-messages.md | 22 +++ docs/member-chronicle.md | 7 + 12 files changed, 495 insertions(+), 67 deletions(-) create mode 100644 composeApp/src/commonTest/kotlin/to/curare/compose/nostr/MarmotPayloadBoundsTest.kt diff --git a/composeApp/src/commonMain/kotlin/to/curare/compose/database/dao/MarmotOutboundDao.kt b/composeApp/src/commonMain/kotlin/to/curare/compose/database/dao/MarmotOutboundDao.kt index 7bf46692..e5d427ec 100644 --- a/composeApp/src/commonMain/kotlin/to/curare/compose/database/dao/MarmotOutboundDao.kt +++ b/composeApp/src/commonMain/kotlin/to/curare/compose/database/dao/MarmotOutboundDao.kt @@ -28,8 +28,8 @@ import to.curare.compose.nostr.MarmotDelivery import to.curare.compose.nostr.MarmotDirectMessage import to.curare.compose.nostr.Relays import co.touchlab.kermit.Logger +import com.vitorpamplona.quartz.marmot.foundation.appEvents.MarmotAppEvent import com.vitorpamplona.quartz.marmot.mip00KeyPackages.KeyPackageUtils -import com.vitorpamplona.quartz.marmot.mip00KeyPackages.tags.EncodingTag import com.vitorpamplona.quartz.marmot.mip01Groups.MarmotGroupData import com.vitorpamplona.quartz.marmot.mip02Welcome.WelcomeEvent import com.vitorpamplona.quartz.marmot.mip03GroupMessages.GroupEvent @@ -741,12 +741,20 @@ abstract class MarmotOutboundDao( marmotInnerEvent: MarmotInnerEvent, nostrSignerSync: NostrSignerSync ): GroupEvent { - // A direct message is gift wrapped for one member before it goes into MLS, so - // what the group carries is a kind:1059 whose id is not this row's. Everything - // downstream -- MLS encrypt, outer ChaCha20, kind:445, persistence, broadcast -- - // is the same either way. See MarmotDirectMessage. - val innerEvent: Event = marmotInnerEvent.directMessageRecipientPublicKey?.let { recipient -> - MarmotDirectMessage.wrap( + // The room's profile decides the shape of what goes into MLS. See + // docs/marmot-v2.md, Phase 2: it is the feature flag for every wire change + // an old build cannot read, because a current-profile room admits only + // leaves that carry an identity proof, and only a build past Phase 3 can + // make one. + val isCurrentProfileRoom = mlsGroup.currentGroupState().isCurrentProfile + + val plainTextBytes: ByteArray = marmotInnerEvent.directMessageRecipientPublicKey?.let { recipient -> + // A direct message is gift wrapped for one member before it goes into + // MLS, so what the group carries is a kind:1059 whose id is not this + // row's. Everything downstream -- MLS encrypt, outer ChaCha20, kind:445, + // persistence, broadcast -- is the same either way. See + // MarmotDirectMessage. + val wrap = MarmotDirectMessage.wrap( signer = nostrSignerSync, recipientPublicKey = recipient, kind = marmotInnerEvent.kind, @@ -754,19 +762,44 @@ abstract class MarmotOutboundDao( tags = marmotInnerEvent.tags, content = marmotInnerEvent.content ) - } ?: RumorAssembler.assembleRumor( - pubKey = nostrSignerSync.pubKey, - ev = EventTemplate( - createdAt = marmotInnerEvent.createdAt.epochSeconds, - content = marmotInnerEvent.content, - tags = marmotInnerEvent.tags, - kind = marmotInnerEvent.kind + + if (isCurrentProfileRoom) { + // Inside an unsigned envelope authored by the sender, which is what + // a conformant peer accepts. The bare wrap is signed and names a + // throwaway key, and fails their author check twice over. + MarmotDirectMessage.envelope( + senderPublicKey = nostrSignerSync.pubKey, + recipientPublicKey = recipient, + wrap = wrap, + createdAt = marmotInnerEvent.createdAt.epochSeconds, + ).encodeToPayload() + } else { + // The bare wrap, signature and all: the one shape every build in a + // legacy room reads, through the carve-out in mip03Rejection. + wrap.toJson().encodeToByteArray() + } + } ?: run { + val rumor = RumorAssembler.assembleRumor( + pubKey = nostrSignerSync.pubKey, + ev = EventTemplate( + createdAt = marmotInnerEvent.createdAt.epochSeconds, + content = marmotInnerEvent.content, + tags = marmotInnerEvent.tags, + kind = marmotInnerEvent.kind + ) ) - ) - logger.d("InnerEvent: ${innerEvent.toJson()}") + // The canonical Marmot app event: six members and no `sig`. The rumor + // serialised as an Event carries `"sig":""`, which a conformant decoder + // refuses outright -- a payload with a signature would be a standalone + // relay event, republishable as a signed statement by its author. The + // id does not move, because NIP-01 never hashed the signature, so every + // row filed under this event's id stays valid, and every build already + // in the field reads the shape: Event.fromJsonOrNull defaults a missing + // signature. This switch alone needs no gating. + MarmotAppEvent.fromEvent(rumor).encodeToPayload() + } - val plainTextBytes = innerEvent.toJson().encodeToByteArray() logger.d("plainTextBytes: ${plainTextBytes.toHex()}") // Step 1: MLS encrypt @@ -787,15 +820,16 @@ abstract class MarmotOutboundDao( val createdAt = TimeUtils.now() val expirationTime = disappearingExpiration(mlsGroup, createdAt) + // No `encoding` tag. `transports/nostr.md`: a sender MUST NOT add one to any + // event shape -- a receiver that switched decoders on it could be steered + // into a different parse of the same bytes -- and the commit template below + // never carried one either. val template = GroupEvent.build( encryptedContentBase64 = encryptedContent, nostrGroupId = localChatRoom.chatRoom.id, createdAt = createdAt, ) { if (expirationTime != null) expiration(expirationTime) - addUnique( - EncodingTag.assemble() - ) } // Step 4: Sign with a fresh ephemeral keypair diff --git a/composeApp/src/commonMain/kotlin/to/curare/compose/database/model/ChatMessage.kt b/composeApp/src/commonMain/kotlin/to/curare/compose/database/model/ChatMessage.kt index 35f0891e..b99449ec 100644 --- a/composeApp/src/commonMain/kotlin/to/curare/compose/database/model/ChatMessage.kt +++ b/composeApp/src/commonMain/kotlin/to/curare/compose/database/model/ChatMessage.kt @@ -661,6 +661,12 @@ data class ChatMessage( * * The decision is [MarmotDirectMessage.classify]'s, which is where it is tested; * this turns it into rows. See docs/marmot-direct-messages.md. + * + * [filedAs] is the id of the `MarmotInnerEvent` row the caller wrote for the + * payload as it arrived -- the wrap's own id in a legacy room, the envelope's in + * a current-profile one. The line for a message this device cannot read has to + * hang off a row that exists, and `ChatMessage.marmotInnerEventId` is a foreign + * key that says so. */ private suspend fun directMessage( database: CurareDatabase, @@ -668,6 +674,7 @@ data class ChatMessage( groupEvent: GroupEvent, chatRoomId: String, wrap: Event, + filedAs: HexKey, senderIdentity: HexKey? ): ChatMessage? { val recipientPublicKey = wrap.tags.firstTagValue("p") @@ -700,7 +707,7 @@ data class ChatMessage( // id every time, so this is what keeps a second delivery from becoming // a second line. ChatMessage.id is autogenerated and would take a // duplicate. - if (database.chatMessageDao().getChatMessagesByMarmotInnerEventId(wrap.id) != null) { + if (database.chatMessageDao().getChatMessagesByMarmotInnerEventId(filedAs) != null) { return null } @@ -708,7 +715,7 @@ data class ChatMessage( giftWrapPayloadId = null, messageType = TYPE_DIRECT_MESSAGE, marmotGroupEventId = groupEvent.id, - marmotInnerEventId = wrap.id, + marmotInnerEventId = filedAs, senderPublicKey = senderIdentity ?: wrap.pubKey, directMessageRecipientPublicKey = recipientPublicKey, isUserMessage = false, @@ -821,7 +828,11 @@ data class ChatMessage( // A direct message is not a nip30303 payload to apply. It is // gift wrapped for one member, and what this device does with it - // depends only on whether its key opens it. + // depends only on whether its key opens it. A legacy room carries + // the wrap itself; a current-profile room carries it inside an + // envelope authored by the sender -- see MarmotDirectMessage. The + // row above was filed under the payload's own id either way, and + // that is the id the line for a message we cannot read hangs off. if (event.kind == GiftWrapEvent.KIND) { return directMessage( database = database, @@ -829,6 +840,24 @@ data class ChatMessage( groupEvent = groupEvent, chatRoomId = groupEventResult.groupId, wrap = event, + filedAs = event.id, + senderIdentity = senderIdentity + ) + } + + if (event.kind == MarmotDirectMessage.ENVELOPE_KIND) { + val wrap = MarmotDirectMessage.wrapInside(event) + if (wrap == null) { + logger.e("Dropping direct message envelope ${event.id}: it holds no wrap") + return null + } + return directMessage( + database = database, + activeKeyPair = activeKeyPair, + groupEvent = groupEvent, + chatRoomId = groupEventResult.groupId, + wrap = wrap, + filedAs = event.id, senderIdentity = senderIdentity ) } diff --git a/composeApp/src/commonMain/kotlin/to/curare/compose/managers/ChronicleManager.kt b/composeApp/src/commonMain/kotlin/to/curare/compose/managers/ChronicleManager.kt index 6c9d895e..6234631a 100644 --- a/composeApp/src/commonMain/kotlin/to/curare/compose/managers/ChronicleManager.kt +++ b/composeApp/src/commonMain/kotlin/to/curare/compose/managers/ChronicleManager.kt @@ -824,6 +824,12 @@ object ChronicleManager { * checked because they bind different chronicles -- a room of one-line dialects * hits the count first and a room of chapters hits the bytes. * + * The bytes are the page's *escaped* bytes -- what it will occupy as the + * `content` of the Marmot app event that carries it, which is what a + * conformant decoder bounds. Dense JSON payloads grow by half on the way in, + * and a page measured raw was over that bound while under this one. See + * [ChronicleEvent.escapedByteSize]. + * * An event too large to share a page with anything is given one of its own. * One too large for even that is dropped with a log rather than failing the * chronicle: a chapter nobody can chronicle is a hole, and a member who gets @@ -837,7 +843,7 @@ object ChronicleManager { events.forEach { event -> // Plus the comma this event needs if it is not first on its page. - val size = event.toJson().encodeToByteArray().size + 1 + val size = ChronicleEvent.escapedByteSize(event.toJson()) + 1 if (2 + size > ChronicleEvent.MAX_PAGE_BYTES) { logger.w( diff --git a/composeApp/src/commonMain/kotlin/to/curare/compose/managers/MarmotInboundManager.kt b/composeApp/src/commonMain/kotlin/to/curare/compose/managers/MarmotInboundManager.kt index 82f86e26..9faa476b 100644 --- a/composeApp/src/commonMain/kotlin/to/curare/compose/managers/MarmotInboundManager.kt +++ b/composeApp/src/commonMain/kotlin/to/curare/compose/managers/MarmotInboundManager.kt @@ -66,21 +66,25 @@ object MarmotInboundManager { /** * Why an inner application event may not be accepted from this sender, or null. * - * MIP-03: an inner event's `pubkey` MUST equal the MLS sender's credential identity, - * so a member cannot mint events claiming a different author. + * An inner event's `pubkey` MUST equal the MLS sender's credential identity, so a + * member cannot mint events claiming a different author. MIP-03 said so and + * `foundation/application-messages.md` says the same. * - * A gift wrap is exempt. Its pubkey is a throwaway key by construction -- see - * [MarmotDirectMessage] -- so there is no author field to compare and the check - * cannot be applied as written. The authorship claim moves inside, to the signed - * kind:13 seal, which [MarmotDirectMessage.classify] binds to this same identity - * before a word is rendered: a verified signature bound to an MLS leaf, rather than a - * plaintext field compared to one. + * In a legacy room a gift wrap is exempt. Its pubkey is a throwaway key by + * construction -- see [MarmotDirectMessage] -- so there is no author field to + * compare and the check cannot be applied as written. The authorship claim moves + * inside, to the signed kind:13 seal, which [MarmotDirectMessage.classify] binds to + * this same identity before a word is rendered: a verified signature bound to an + * MLS leaf, rather than a plaintext field compared to one. * - * The exemption is on kind:1059 alone. Widening it to any other kind, or dropping the - * kind guard, hands every member the ability to publish events as anybody. - * - * Other Marmot clients do not have this carve-out and drop these messages as - * impersonation. See docs/marmot-direct-messages.md. + * The exemption is on kind:1059 alone, and on legacy rooms alone. Widening it to any + * other kind, or dropping the kind guard, hands every member the ability to publish + * events as anybody. A current-profile room never carries a bare wrap -- a direct + * message there travels inside an envelope authored by the sender, which passes the + * check like any other payload -- so a bare wrap arriving in one is what every other + * conformant client says it is: an author that is not the sender, refused. Other + * Marmot clients have no carve-out at all and drop legacy wraps as impersonation. + * See docs/marmot-direct-messages.md and docs/marmot-v2.md, Phase 2. * * [senderIdentity] is required rather than merely compared: every sender-derived * field downstream reads from it -- who a message is from, and whether it is ours -- @@ -90,12 +94,14 @@ object MarmotInboundManager { innerEventKind: Int, innerEventPubKey: HexKey, senderIdentity: HexKey?, + legacyRoom: Boolean, ): String? { if (senderIdentity == null) { return "No MLS credential identity for the sender leaf" } - if (innerEventKind != GiftWrapEvent.KIND && innerEventPubKey != senderIdentity) { + val bareWrapInLegacyRoom = legacyRoom && innerEventKind == GiftWrapEvent.KIND + if (!bareWrapInLegacyRoom && innerEventPubKey != senderIdentity) { return "MIP-03: inner event pubkey ($innerEventPubKey) does not match MLS sender identity ($senderIdentity)" } @@ -332,7 +338,12 @@ object MarmotInboundManager { if (innerAuthor != null) { val senderIdentity = mlsGroup.memberIdentityHex(decrypted.senderLeafIndex) - mip03Rejection(innerAuthor.first, innerAuthor.second, senderIdentity)?.let { reason -> + mip03Rejection( + innerEventKind = innerAuthor.first, + innerEventPubKey = innerAuthor.second, + senderIdentity = senderIdentity, + legacyRoom = !mlsGroup.currentGroupState().isCurrentProfile, + )?.let { reason -> return GroupEventResult.Error(groupId, reason) } } diff --git a/composeApp/src/commonMain/kotlin/to/curare/compose/nostr/MarmotDirectMessage.kt b/composeApp/src/commonMain/kotlin/to/curare/compose/nostr/MarmotDirectMessage.kt index d87c6a9f..3e8e25bc 100644 --- a/composeApp/src/commonMain/kotlin/to/curare/compose/nostr/MarmotDirectMessage.kt +++ b/composeApp/src/commonMain/kotlin/to/curare/compose/nostr/MarmotDirectMessage.kt @@ -1,5 +1,6 @@ package to.curare.compose.nostr +import com.vitorpamplona.quartz.marmot.foundation.appEvents.MarmotAppEvent import com.vitorpamplona.quartz.nip01Core.core.Event import com.vitorpamplona.quartz.nip01Core.core.HexKey import com.vitorpamplona.quartz.nip01Core.core.Kind @@ -20,10 +21,22 @@ import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent * A one-to-one message carried inside a Marmot group, as the application payload of a * kind:445 group event. See docs/marmot-direct-messages.md. * - * The payload is a stock NIP-59 gift wrap: a throwaway-keyed kind:1059 around a + * The message is a stock NIP-59 gift wrap: a throwaway-keyed kind:1059 around a * sender-signed kind:13 seal around the kind:14 rumor that holds the words. Every member * of the group decrypts the MLS layer and sees the wrap; only the recipient can open it. * + * What carries the wrap depends on the room's profile. A legacy room carries the wrap + * itself as the payload -- signed by its throwaway key, with a `pubkey` that names + * nobody -- which is what every build before docs/marmot-v2.md Phase 2 sent and reads, + * and what the carve-out in `MarmotInboundManager.mip03Rejection` exists to admit. A + * current-profile room carries it inside an [envelope]: an unsigned Marmot app event of + * [ENVELOPE_KIND] whose `pubkey` is the sender's account and whose `content` is the + * wrap's JSON. A conformant decoder refuses a payload with a `sig` member and one whose + * author is not the MLS sender, and the wrap fails both; the envelope passes both, and + * keeps every property the wrap was chosen for -- see docs/marmot-direct-messages.md -- + * because the envelope is unsigned and so a claim rather than a proof, and the wrap + * inside it is still signed by nobody. + * * Everything here is a pure function of its arguments, with no database and no MLS state, * which is what makes it testable — Room-backed code cannot be unit-tested in this project. * @@ -33,6 +46,17 @@ import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent * message lives in `MarmotInnerEvent`, never in `GiftWrapPayload` or `GiftWrapMessage`. */ object MarmotDirectMessage { + /** + * The app kind of the envelope a current-profile room carries a direct message in. + * + * In the app's own range, beside the chronicle (30327), its request (30328) and the + * subgroup certificate (30329): a kind that exists only inside MLS and that no relay + * ever sees. Another client renders it as a kind it does not know, which is the + * "sent a private message" line and nothing more -- the same thing a bystander in + * this app sees. + */ + const val ENVELOPE_KIND: Kind = 30330 + /** * A wrap that opened, both layers of it. * @@ -176,6 +200,46 @@ object MarmotDirectMessage { ) } + /** + * The envelope a current-profile room carries [wrap] in. + * + * Unsigned, by construction -- a `MarmotAppEvent` has no signature to give -- and + * authored by [senderPublicKey], which is the MLS sender's account and therefore + * what the standard author check compares against. The recipient rides in a `p` tag + * so a member can tell whose message it is without opening the content, exactly as + * the wrap's own `p` tag lets them today. [createdAt] is the wrap's, unfuzzed, for + * the reason [wrap] gives. + */ + fun envelope( + senderPublicKey: HexKey, + recipientPublicKey: HexKey, + wrap: Event, + createdAt: Long, + ): MarmotAppEvent = + MarmotAppEvent.build( + pubKey = senderPublicKey, + kind = ENVELOPE_KIND, + content = wrap.toJson(), + createdAt = createdAt, + tags = arrayOf(arrayOf("p", recipientPublicKey)), + ) + + /** + * The wrap inside [envelope], or null when it is not an envelope or holds no wrap. + * + * Null rather than a throw for the same reason [open] returns null: the caller is + * inside inbound processing for a kind:445, and one malformed payload must not cost + * the rest. + */ + fun wrapInside(envelope: Event): Event? { + if (envelope.kind != ENVELOPE_KIND) return null + return try { + Event.fromJsonOrNull(envelope.content)?.takeIf { it.kind == GiftWrapEvent.KIND } + } catch (_: Exception) { + null + } + } + /** * The rumor a direct message carries, unsigned and unencrypted. * diff --git a/composeApp/src/commonMain/kotlin/to/curare/compose/nostr/chronicle/ChronicleEvent.kt b/composeApp/src/commonMain/kotlin/to/curare/compose/nostr/chronicle/ChronicleEvent.kt index eb5c5669..75efd332 100644 --- a/composeApp/src/commonMain/kotlin/to/curare/compose/nostr/chronicle/ChronicleEvent.kt +++ b/composeApp/src/commonMain/kotlin/to/curare/compose/nostr/chronicle/ChronicleEvent.kt @@ -113,8 +113,18 @@ class ChronicleEvent( * Sized so the finished kind:445 clears the tightest relay limits with * room for the MLS framing and the NIP-44 expansion that sit outside it; * measure it against a real one rather than trusting the arithmetic. + * + * Under 64 KiB with room to spare, because that is the bound a conformant + * Marmot decoder puts on the *whole* app payload -- `MarmotJson.MAX_INPUT_BYTES` + * -- and this page rides inside one as its `content`, JSON-escaped and + * wrapped in the six members of the envelope. The packer measures the page + * as [escapedByteSize] -- the bytes it will occupy inside the envelope, + * which for dense JSON content is half again the raw page -- against this + * cap, so both the receiver's raw-bytes check and the decoder's bound hold + * by construction. At 64 KiB the raw page alone was over the bound, and an + * MDK peer dropped it without a word. See docs/marmot-v2.md, Phase 2. */ - const val MAX_PAGE_BYTES = 64 * 1024 + const val MAX_PAGE_BYTES = 48 * 1024 /** * The most payloads one page may carry. @@ -126,10 +136,10 @@ class ChronicleEvent( * **Sized under what the byte cap allows, or it would never fire.** An * event carries a 64-character id, a 64-character pubkey and a * 128-character signature before it says anything at all, so the floor is - * around 370 bytes and [MAX_PAGE_BYTES] cannot hold much past 170 of + * around 370 bytes and [MAX_PAGE_BYTES] cannot hold much past 130 of * them. A count cap above that ceiling is a check that cannot fail -- * which is how this was first written, and a test that asserts a page at - * exactly the cap decodes is what caught it. At 128 both caps bind + * exactly the cap decodes is what caught it. At 96 both caps bind * something: the count stops a page of many small payloads, the bytes * stop a page of few large ones. * @@ -141,7 +151,7 @@ class ChronicleEvent( * item costs nonce generation, a FROST session and a native sign, where * a chronicle payload costs a signature verify and an upsert. */ - const val MAX_PAGE_EVENTS = 128 + const val MAX_PAGE_EVENTS = 96 /** * The kinds a chronicle may carry, in the order they have to be applied. @@ -221,6 +231,34 @@ class ChronicleEvent( fun inApplyOrder(payloads: List): List = payloads.sortedBy { applyRank(it.kind) ?: Int.MAX_VALUE } + /** + * The bytes [json] occupies once it is the `content` of a Marmot app event -- + * that is, once it is inside a JSON string. + * + * A page is JSON that travels as JSON: every quote in it becomes `\\"`, every + * backslash `\\\\`, and the payloads' own content, already escaped once inside + * the page, escapes again. Dense JSON content grows by half; content that is + * nothing but quotes doubles. The decoder's 64 KiB bound is on the *escaped* + * form, so the packer has to measure this and not the raw page -- see + * [MAX_PAGE_BYTES]. The escape set is NIP-01's, which is what + * `MarmotAppEvent.toJson` writes. + */ + fun escapedByteSize(json: String): Int { + var size = 0 + for (ch in json) { + size += when { + ch == '"' || ch == '\\' || ch == '\n' || ch == '\r' || ch == '\t' || + ch == '\u0008' || ch == '\u000C' -> 2 + ch < ' ' -> 6 + ch.code < 0x80 -> 1 + ch.code < 0x800 -> 2 + ch.isSurrogate() -> 2 // each half of a pair is two of the four bytes + else -> 3 + } + } + return size + } + /** * One page's payloads as its content: a JSON array of whole events. * diff --git a/composeApp/src/commonTest/kotlin/to/curare/compose/managers/MarmotMip03CarveOutTest.kt b/composeApp/src/commonTest/kotlin/to/curare/compose/managers/MarmotMip03CarveOutTest.kt index 4d0de745..17bb4109 100644 --- a/composeApp/src/commonTest/kotlin/to/curare/compose/managers/MarmotMip03CarveOutTest.kt +++ b/composeApp/src/commonTest/kotlin/to/curare/compose/managers/MarmotMip03CarveOutTest.kt @@ -4,48 +4,77 @@ import com.vitorpamplona.quartz.nip59Giftwrap.wraps.GiftWrapEvent import com.vitorpamplona.quartz.nip17Dm.messages.ChatMessageEvent import com.vitorpamplona.quartz.nip59Giftwrap.seals.SealedRumorEvent import com.vitorpamplona.quartz.nipC7Chats.ChatEvent +import to.curare.compose.nostr.MarmotDirectMessage import kotlin.test.Test import kotlin.test.assertNotNull import kotlin.test.assertNull import kotlin.test.assertTrue /** - * The one hole deliberately left in MIP-03's author check, and its edges. + * The one hole deliberately left in the author check, and its edges. * - * MIP-03 requires an inner application event's pubkey to equal the MLS sender's credential - * identity, which is what stops a member minting events attributed to somebody else. A - * gift wrap cannot satisfy it -- its pubkey is a throwaway key that names nobody -- so - * kind:1059 is exempt, and the authorship claim moves to the signed seal inside. + * An inner application event's pubkey must equal the MLS sender's credential identity, + * which is what stops a member minting events attributed to somebody else. In a legacy + * room a gift wrap cannot satisfy it -- its pubkey is a throwaway key that names nobody + * -- so kind:1059 is exempt there, and the authorship claim moves to the signed seal + * inside. In a current-profile room there is no exemption: a direct message travels + * inside an envelope the sender authors, and a bare wrap is refused like any other + * payload claiming an author it does not own. * - * The exemption is the most dangerous line in this feature. Widened to another kind, or - * stripped of its kind guard, it hands every member of every group the ability to publish - * events as anybody, and nothing else in the pipeline would notice. These tests exist so - * that widening it fails here rather than in a group. + * The exemption is the most dangerous line in this feature. Widened to another kind, + * stripped of its kind guard, or extended to current-profile rooms, it hands every + * member of every group the ability to publish events as anybody, and nothing else in + * the pipeline would notice. These tests exist so that widening it fails here rather + * than in a group. */ class MarmotMip03CarveOutTest { private val sender = "a".repeat(64) private val somebodyElse = "b".repeat(64) private val throwaway = "c".repeat(64) + private fun rejection(kind: Int, pubKey: String, identity: String?, legacyRoom: Boolean = true) = + MarmotInboundManager.mip03Rejection( + innerEventKind = kind, + innerEventPubKey = pubKey, + senderIdentity = identity, + legacyRoom = legacyRoom, + ) + @Test - fun `an ordinary message from its own author is accepted`() { - assertNull(MarmotInboundManager.mip03Rejection(ChatEvent.KIND, sender, sender)) + fun `an ordinary message from its own author is accepted in either room`() { + assertNull(rejection(ChatEvent.KIND, sender, sender, legacyRoom = true)) + assertNull(rejection(ChatEvent.KIND, sender, sender, legacyRoom = false)) } @Test - fun `an ordinary message claiming somebody else is rejected`() { - val reason = MarmotInboundManager.mip03Rejection(ChatEvent.KIND, somebodyElse, sender) + fun `an ordinary message claiming somebody else is rejected in either room`() { + listOf(true, false).forEach { legacy -> + val reason = rejection(ChatEvent.KIND, somebodyElse, sender, legacyRoom = legacy) - assertNotNull(reason) - assertTrue(reason.startsWith("MIP-03"), "expected the MIP-03 rejection, got: $reason") + assertNotNull(reason) + assertTrue(reason.startsWith("MIP-03"), "expected the MIP-03 rejection, got: $reason") + } } @Test - fun `a gift wrap is accepted despite naming nobody`() { + fun `in a legacy room a gift wrap is accepted despite naming nobody`() { // The carve-out. The wrap's pubkey is a throwaway key and matches no member, which // is the point of it; what authenticates the sender is the MLS frame, and what // authenticates the author is the seal inside. See MarmotDirectMessage.classify. - assertNull(MarmotInboundManager.mip03Rejection(GiftWrapEvent.KIND, throwaway, sender)) + assertNull(rejection(GiftWrapEvent.KIND, throwaway, sender, legacyRoom = true)) + } + + @Test + fun `in a current-profile room a bare gift wrap is an author that is not the sender`() { + // No carve-out. A direct message in a current-profile room is an envelope by the + // sender; a bare wrap there is what every other conformant client refuses it as. + assertNotNull(rejection(GiftWrapEvent.KIND, throwaway, sender, legacyRoom = false)) + } + + @Test + fun `an envelope passes the ordinary check because its author is the sender`() { + assertNull(rejection(MarmotDirectMessage.ENVELOPE_KIND, sender, sender, legacyRoom = false)) + assertNotNull(rejection(MarmotDirectMessage.ENVELOPE_KIND, somebodyElse, sender, legacyRoom = false)) } @Test @@ -57,6 +86,7 @@ class MarmotMip03CarveOutTest { ChatEvent.KIND, ChatMessageEvent.KIND, SealedRumorEvent.KIND, + MarmotDirectMessage.ENVELOPE_KIND, GiftWrapEvent.KIND - 1, GiftWrapEvent.KIND + 1, 0, @@ -65,7 +95,7 @@ class MarmotMip03CarveOutTest { kinds.forEach { kind -> assertNotNull( - MarmotInboundManager.mip03Rejection(kind, somebodyElse, sender), + rejection(kind, somebodyElse, sender, legacyRoom = true), "kind $kind was allowed to claim an author it does not own", ) } @@ -76,7 +106,7 @@ class MarmotMip03CarveOutTest { // Required rather than merely compared: with no identity there is nothing to // attribute a line to, and for a gift wrap there is nothing to check the seal // against either. Both kinds must fail, not just the one that does a comparison. - assertNotNull(MarmotInboundManager.mip03Rejection(ChatEvent.KIND, sender, null)) - assertNotNull(MarmotInboundManager.mip03Rejection(GiftWrapEvent.KIND, throwaway, null)) + assertNotNull(rejection(ChatEvent.KIND, sender, null)) + assertNotNull(rejection(GiftWrapEvent.KIND, throwaway, null)) } } diff --git a/composeApp/src/commonTest/kotlin/to/curare/compose/nostr/MarmotDirectMessageTest.kt b/composeApp/src/commonTest/kotlin/to/curare/compose/nostr/MarmotDirectMessageTest.kt index 01d1fe3b..2f8366e5 100644 --- a/composeApp/src/commonTest/kotlin/to/curare/compose/nostr/MarmotDirectMessageTest.kt +++ b/composeApp/src/commonTest/kotlin/to/curare/compose/nostr/MarmotDirectMessageTest.kt @@ -1,5 +1,6 @@ package to.curare.compose.nostr +import com.vitorpamplona.quartz.marmot.foundation.appEvents.MarmotAppEvent import com.vitorpamplona.quartz.nip01Core.core.Event import com.vitorpamplona.quartz.nip01Core.core.firstTagValue import com.vitorpamplona.quartz.nip01Core.core.toHexKey @@ -182,4 +183,55 @@ class MarmotDirectMessageTest { assertNull(MarmotDirectMessage.open(notAWrap, bob)) } + + // ---- The envelope, for current-profile rooms ----------------------------- + + @Test + fun `the envelope is unsigned, by the sender, and carries the wrap whole`() { + val wrap = aliceWrapsForBob() + val envelope = MarmotDirectMessage.envelope(alice.pubKey, bobPublicKey, wrap, at) + + // Six members and no signature: what a conformant decoder accepts, and what a + // leak cannot republish as a signed statement by anybody. + val json = envelope.toJson() + assertTrue(!json.contains("\"sig\""), "the envelope carries a signature") + assertEquals(alice.pubKey, MarmotAppEvent.decode(json).pubKey) + assertEquals(MarmotDirectMessage.ENVELOPE_KIND, MarmotAppEvent.decode(json).kind) + assertEquals(bobPublicKey, envelope.tags.firstTagValue("p")) + + // The wrap inside is the wrap, untouched, and Bob opens it as before. + val inside = assertNotNull(MarmotDirectMessage.wrapInside(Event.fromJsonOrNull(json)!!)) + assertEquals(wrap.id, inside.id) + assertEquals(text, assertNotNull(MarmotDirectMessage.open(inside, bob)).rumor.content) + } + + @Test + fun `the envelope's createdAt is the wrap's, unfuzzed`() { + val envelope = MarmotDirectMessage.envelope(alice.pubKey, bobPublicKey, aliceWrapsForBob(), at) + + assertEquals(at, envelope.createdAt) + } + + @Test + fun `something that is not an envelope holds no wrap`() { + val wrap = aliceWrapsForBob() + val wrongKind = Event( + id = "1".repeat(64), pubKey = alice.pubKey, createdAt = at, + kind = MarmotDirectMessage.ENVELOPE_KIND + 1, tags = emptyArray(), content = wrap.toJson(), sig = "", + ) + val notAWrapInside = Event( + id = "2".repeat(64), pubKey = alice.pubKey, createdAt = at, + kind = MarmotDirectMessage.ENVELOPE_KIND, tags = emptyArray(), content = "not json", sig = "", + ) + val sealInside = Event( + id = "3".repeat(64), pubKey = alice.pubKey, createdAt = at, + kind = MarmotDirectMessage.ENVELOPE_KIND, tags = emptyArray(), + content = alice.signNormal(createdAt = at, kind = SealedRumorEvent.KIND, tags = emptyArray(), content = "").toJson(), + sig = "", + ) + + assertNull(MarmotDirectMessage.wrapInside(wrongKind)) + assertNull(MarmotDirectMessage.wrapInside(notAWrapInside)) + assertNull(MarmotDirectMessage.wrapInside(sealInside)) + } } diff --git a/composeApp/src/commonTest/kotlin/to/curare/compose/nostr/MarmotPayloadBoundsTest.kt b/composeApp/src/commonTest/kotlin/to/curare/compose/nostr/MarmotPayloadBoundsTest.kt new file mode 100644 index 00000000..d1b35404 --- /dev/null +++ b/composeApp/src/commonTest/kotlin/to/curare/compose/nostr/MarmotPayloadBoundsTest.kt @@ -0,0 +1,131 @@ +package to.curare.compose.nostr + +import com.vitorpamplona.quartz.marmot.foundation.appEvents.MarmotAppEvent +import com.vitorpamplona.quartz.marmot.foundation.appEvents.MarmotJson +import com.vitorpamplona.quartz.nip01Core.core.Event +import com.vitorpamplona.quartz.nip01Core.core.toHexKey +import com.vitorpamplona.quartz.nip01Core.signers.EventTemplate +import com.vitorpamplona.quartz.nip01Core.crypto.KeyPair +import com.vitorpamplona.quartz.nip01Core.signers.NostrSignerSync +import com.vitorpamplona.quartz.nip01Core.tags.people.PTag +import com.vitorpamplona.quartz.nip17Dm.messages.ChatMessageEvent +import com.vitorpamplona.quartz.nip59Giftwrap.rumors.RumorAssembler +import to.curare.compose.nostr.chronicle.ChronicleEvent +import to.curare.compose.nostr.nip30303.ChunkEvent +import to.curare.compose.nostr.nip30303.DialectEvent +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertNotNull +import kotlin.test.assertTrue + +/** + * The app's largest payloads fit inside what a conformant Marmot decoder will read. + * + * `MarmotJson` bounds an app payload at 64 KiB, sixteen levels of nesting and 64 + * elements per collection, and refuses anything over before a parser sees it. The + * app's own rooms would still read an oversized page through the legacy fallback; an + * MDK peer drops it without a word. So the caps that decide the app's biggest + * payloads are asserted here against the decoder itself, at their maximum, so that + * raising one of them past the bound fails in a test rather than in a group. + */ +class MarmotPayloadBoundsTest { + private val author = "a".repeat(64) + private val recipient = "d".repeat(64) + private val at = 1_700_000_000L + + private fun signedPayload(kind: Int, content: String, id: String) = Event( + id = id, + pubKey = "b".repeat(64), + createdAt = at, + kind = kind, + tags = arrayOf(arrayOf("alt", "test"), arrayOf("d", "x".repeat(64))), + content = content, + sig = "c".repeat(128), + ) + + /** The app event the outbound path builds for a chronicle page: the rumor, canonical. */ + private fun chronicleAppEvent(payloads: List): MarmotAppEvent { + val template = ChronicleEvent.build( + payloads = payloads, + chronicleId = "e".repeat(64), + index = 0, + count = 1, + recipient = recipient, + createdAt = at, + ) + // From the template's fields, as the outbound queue assembles every rumor from + // the row's: the typed template is not what goes through RumorAssembler. + val rumor = RumorAssembler.assembleRumor( + pubKey = author, + ev = EventTemplate(createdAt = template.createdAt, kind = template.kind, tags = template.tags, content = template.content), + ) + return MarmotAppEvent.fromEvent(rumor) + } + + private fun assertConformant(appEvent: MarmotAppEvent) { + val json = appEvent.toJson() + assertTrue( + MarmotJson.withinResourceBounds(json), + "${json.encodeToByteArray().size} bytes is outside the decoder's bounds", + ) + assertEquals(appEvent.id, MarmotAppEvent.decode(json).id) + } + + @Test + fun `a chronicle page of the most, smallest payloads is a conformant app event`() { + val payloads = (0 until ChronicleEvent.MAX_PAGE_EVENTS).map { + signedPayload(DialectEvent.KIND, content = "d", id = it.toString().padStart(64, '0')) + } + + assertConformant(chronicleAppEvent(payloads)) + } + + /** + * The worst case for the byte bound: content that is itself dense JSON, so every + * quote in the page doubles when the page becomes the app event's `content`. + */ + @Test + fun `a chronicle page at the byte cap is a conformant app event after escaping`() { + val dense = buildString { + while (length < 4_096) append("{\"k\":\"v\",\"n\":[\"a\",\"b\"]}") + } + val payloads = (0 until ChronicleEvent.MAX_PAGE_EVENTS).map { + signedPayload(ChunkEvent.KIND, content = dense, id = it.toString().padStart(64, '0')) + } + // As many dense payloads as the byte cap admits, measured the way the packer + // measures them: by the bytes the page occupies once escaped into the app + // event's content. Measured raw, this page is under the cap and the app + // event is over the decoder's bound -- which is the bug the measure fixes. + val page = (payloads.size downTo 1).firstNotNullOf { n -> + payloads.take(n).takeIf { + ChronicleEvent.escapedByteSize(ChronicleEvent.encodePage(it)) <= ChronicleEvent.MAX_PAGE_BYTES + } + } + assertTrue(page.size < payloads.size, "the byte cap never bound; the page is not at its maximum") + assertTrue( + ChronicleEvent.encodePage(page + payloads[page.size]).encodeToByteArray().size <= ChronicleEvent.MAX_PAGE_BYTES, + "measured raw, one more payload would still have fit -- the escaped measure is not the binding one here", + ) + + assertConformant(chronicleAppEvent(page)) + } + + @Test + fun `a direct message envelope is a conformant app event`() { + val sender = NostrSignerSync(KeyPair()) + val bob = KeyPair() + val wrap = MarmotDirectMessage.wrap( + signer = sender, + recipientPublicKey = bob.pubKey.toHexKey(), + kind = ChatMessageEvent.KIND, + createdAt = at, + tags = arrayOf(PTag.assemble(bob.pubKey.toHexKey(), null)), + content = "x".repeat(8_192), + ) + val envelope = MarmotDirectMessage.envelope(sender.pubKey, bob.pubKey.toHexKey(), wrap, at) + + assertConformant(envelope) + val inside = assertNotNull(MarmotDirectMessage.wrapInside(Event.fromJsonOrNull(envelope.toJson())!!)) + assertEquals(wrap.id, inside.id) + } +} diff --git a/composeApp/src/jvmTest/kotlin/to/curare/compose/database/dao/MarmotKeyPackageIngestJvmTest.kt b/composeApp/src/jvmTest/kotlin/to/curare/compose/database/dao/MarmotKeyPackageIngestJvmTest.kt index 91e3f8cb..8bf96c78 100644 --- a/composeApp/src/jvmTest/kotlin/to/curare/compose/database/dao/MarmotKeyPackageIngestJvmTest.kt +++ b/composeApp/src/jvmTest/kotlin/to/curare/compose/database/dao/MarmotKeyPackageIngestJvmTest.kt @@ -24,6 +24,7 @@ import kotlin.test.assertEquals import kotlin.test.assertNotNull import kotlin.test.assertNull import kotlin.test.assertTrue +import kotlin.io.encoding.Base64 /** * A peer's kind:30443 through the sync funnel and into an invite, on both profiles. @@ -155,9 +156,12 @@ class MarmotKeyPackageIngestJvmTest { @Test fun `a key package that does not verify is not stored`() = runBlocking { val (event, _) = currentProfileKeyPackageEventFor(peer) - val tampered = event.asStoredNostrEvent().copy( - content = event.content.dropLast(4) + "AAAA", - ) + // A byte of the KeyPackage's own signature -- its last field -- flipped after + // signing. Flipped in the decoded bytes rather than the base64 text, because + // rewriting the text's tail can change nothing but its padding. + val bytes = Base64.decode(event.content) + bytes[bytes.lastIndex] = (bytes[bytes.lastIndex].toInt() xor 0x01).toByte() + val tampered = event.asStoredNostrEvent().copy(content = Base64.encode(bytes)) store(tampered) diff --git a/docs/marmot-direct-messages.md b/docs/marmot-direct-messages.md index 4314ce2d..d94e8aa6 100644 --- a/docs/marmot-direct-messages.md +++ b/docs/marmot-direct-messages.md @@ -102,6 +102,28 @@ the first was asserted the wrong way round until one of them failed. > divergence from the spec, and a reason to raise it upstream rather than carry it > indefinitely. +### The envelope, in current-profile rooms + +The carve-out is now confined to legacy rooms, and the upstream answer turned out +to be a shape rather than a rule change. A current-profile room ([marmot-v2.md](./marmot-v2.md), +Phase 2) carries the wrap *inside* an unsigned Marmot app event of +`MarmotDirectMessage.ENVELOPE_KIND` (30330), whose `pubkey` is the sender's account, +whose one tag is `["p", recipient]` and whose `content` is the wrap's JSON. The +envelope passes the author check like any other payload, so there is nothing to +exempt, and `mip03Rejection` takes a `legacyRoom` flag: in a current-profile room a +bare kind:1059 is refused as what every other conformant client refuses it as. + +The properties above survive intact. The envelope is unsigned — a `MarmotAppEvent` +has no signature to give — so a lifted envelope is a *claim* of authorship and not a +proof; the wrap inside is still signed by nobody; the seal-to-leaf binding in +`classify` still decides who wrote the words. What a bystander sees is unchanged: +that a private message was sent, and to whom. + +An old build renders the envelope as a raw-JSON bubble, which is why a legacy room +keeps sending the bare wrap: a current-profile room admits only leaves carrying an +identity proof, and only a build past Phase 3 can make one, so nobody in one runs +an old build. The room's profile is the flag. + ### The sender cannot read their own messages back The throwaway private key is discarded at send time, so nothing can reopen the wrap diff --git a/docs/member-chronicle.md b/docs/member-chronicle.md index 5ccafe1b..ee6fecdb 100644 --- a/docs/member-chronicle.md +++ b/docs/member-chronicle.md @@ -272,6 +272,13 @@ a chronicle is the second place in this protocol where a remote party decides ho much work everyone else does. Measure the 64 KB against a finished kind:445 rather than trusting it -- MLS framing and NIP-44 expansion both sit outside it. +**Re-sized since: 48 KiB and 96.** A conformant Marmot decoder bounds the whole +app payload at 64 KiB, and a page rides inside one as its JSON-escaped `content` +plus six envelope members, so a 64 KiB page was over that bound by construction +and an MDK peer dropped it without a word. 48 KiB leaves room for the escaping +whatever the mix of payloads, and 96 is the count that still binds under it. +See [marmot-v2.md](./marmot-v2.md), Phase 2. + **Test:** codec round-trip; a page over either cap is refused on receive; an array containing a non-event is refused whole.