714354ae9a47a0d07693ea9aa495c532abcd5f00
63 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kgothatso Ngako
|
5abc37e463 |
docs: scope the jvm target, and separate it from testing the daos
Two questions arrived together -- whether Room's own testing guidance applies to this project, and what desktop support would cost -- and they turned out to have opposite answers. Both are now in docs/jvm-target.md, phased, with the blocking work separated from the mechanical work. **The expensive part is already done.** The four-deep native chain -- secp256k1 -> bitcoin-kmp -> lightning-kmp -> lightning-kmp-app -- already builds for JVM, on every android build we do. The comment at composeApp/build.gradle.kts:50 records the mechanism without drawing the conclusion: lightning-kmp-core publishes no android variant, so our android target resolves it to the *jvm* one, which pulls secp256k1-kmp-jni-jvm desktop natives, which is exactly why the build has to name the android artifact by hand. Read the other way round, every JVM artifact in the chain is already compiled from source by the composite build. A jvm target adds no cinterop, no C compilation and no new native constraints. That was the part worth being afraid of, and it is finished. **The blocker is one level down, and smaller than it looks.** lightning-kmp-app/library declares 25 expects and implements them across 35 androidMain files. Its jvmMain holds exactly one: fibiprops.jvm.kt, the Kotlin multiplatform library template's Fibonacci boilerplate, satisfying two of the 25 -- both of them the template's own. So 23 actuals are missing, which is why jvm() is commented out there (library/build.gradle.kts:18), which is why it is commented out here (composeApp/build.gradle.kts:46). Mantra cannot declare the target until the fork does. Six phases, ordered by that dependency. 0 build config; 1 the fourteen mechanical phoenix actuals; 2 the three SQLDelight JDBC drivers and NetworkMonitor; 3 key storage; 4 mantra's own sixteen expects; 5 the desktop entry point. 1-3 are independent and parallelisable, 4 is where the compiler finally checks the whole thing. Roughly a week to a launchable build. **Phase 3 has no day estimate, deliberately.** keyStoreEncryption / keyStoreDecryption and their two graceful* wrappers delegate on android to KeystoreHelper.kt -- 116 lines against AndroidKeyStore, StrongBox attempted first and fallen back from, key material never leaving hardware. Desktop JVM has no equivalent, so this is a decision rather than a port, and the doc gives the three real options against what each actually protects. A fixed-key JCEKS file is named there as a liability rather than a stopgap: this is wallet seed material, and it lands on top of the plaintext-key finding already open against this codebase. Recommended sequencing is a passphrase-derived KEK with the desktop build marked unsuitable for real funds, so phases 4 and 5 can proceed without the security question being quietly treated as answered. Two inherited mistakes are called out rather than carried forward. The old Aux jvmMain put the database in java.io.tmpdir behind a TODO -- the doc says not to inherit that in either phase that touches it. And schedulePlatformLogic goes through WorkManager on android with no desktop counterpart, so the doc asks for an explicit choice between a no-op and an in-process coroutine, written down. **The DAO answer is an appendix, because it is the opposite answer.** None of the above is needed to test the DAOs, and burying that would have been misleading. room3-runtime-android:3.0.1 already exposes the no-Context inMemoryDatabaseBuilder(Function0<T>) overload, and MantraDatabaseConstructor already supplies what it needs, so Room's recommended host-machine form compiles in commonTest and runs under testDebugUnitTest today. The one trap is native and is the secp256k1 problem mirrored: sqlite-bundled-android ships only android-ABI .so under jni/, so a local unit test's JVM cannot load it and BundledSQLiteDriver fails at construction; sqlite-bundled-jvm on the androidUnitTest classpath is the fix. Robolectric neither helps nor is needed -- it cannot load android .so on the host either. Everything structural here was checked against the artifacts rather than recalled: the Room builder overloads by javap on room3-runtime-android, the two sqlite-bundled native layouts by unzipping both, and the availability of room3-runtime-jvm, room3-testing, quartz-jvm and the two SQLDelight drivers by request against the repositories this build actually resolves from. The absence of android.* and java.* imports in commonMain, and of any NFC reference from it, was likewise grepped rather than assumed. **Not verified: anything that requires compiling.** No jvm target was turned on, nothing was built, and the day estimates are estimates. Phase 4 is where dependency-substitution surprises would surface if there are any, and it is precisely the phase nothing here exercises. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kgothatso Ngako
|
39eac61838 |
Merge branch 'mantra' into claude/long-running-chat-sync-8983dc
mantra had moved on ~30 commits, several of them in exactly this area — and it turns out both branches independently found the same bug and drew the same conclusion about the same filter. **The overlap.** |
||
|
Kgothatso Ngako
|
bcdfd2ec94 |
Merge branch 'mantra' into claude/marmot-direct-message-type-7a0473
Twenty-two commits had landed on mantra since this branch left it, several of them in the same files. Merged this way round so mantra stayed untouched until the result compiled and its tests passed. The migration had to be renumbered, and this is the conflict that mattered. mantra is at database version 7 and already has its own 5.json -- for MarmotInnerEvent.payloadEventId, nothing to do with direct messages. This branch had also written a 5.json, for a different schema. Resolved by restoring mantra's 5.json untouched and moving the direct message columns to an AutoMigration(7, 8) with a regenerated 8.json. Taking either 5.json over the other would have left every device validating a migration chain against a schema it was never built from; keeping version = 5 would have made a v7 install refuse to open at all. The regenerated 8.json is two ADD COLUMNs and nothing else, same as before. fromGroupEventResult was restructured on mantra: the kind switch moved into applyInnerEvent, and a SubmissionEvent envelope now wraps nip30303 payloads. Took that structure and re-applied the direct message branch ahead of it rather than inside it -- a gift wrap is not a nip30303 payload to apply, and what happens to it depends only on whether this device's key opens it, so it does not belong in a function about applying submissions. The isUserMessage fix was re-applied to the eight call sites mantra's version has, up from the six it had here. ChatMessageListViewModel and ChatRoomMessagingScreen took mantra's versions with the composer state, the two renderings and the reply action layered back on. docs/README.md keeps both new rows and mantra's closing note about the skipped-keys document. 108 tests pass, up from 50 here and 83 on mantra. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kgothatso Ngako
|
c8cbd936f1 |
docs: record where the sync's safety net is, and where it is not
Two updates after the test pass. long-running-sync.md gains a section naming what each test file pins and, more usefully, the three things they cannot reach: NostrSocketClientImpl's reconnect loop and ordered inbound (exercised only through their extracted arithmetic — covering them wants a fake WebSocketSession), everything downstream of saveNostrEvent (Room-backed, and there is no sqlite driver on the JVM test classpath), and the app on a device. The manual checks stay the manual checks. It also records that the tests were verified by mutation rather than by passing, so the next person knows the assertions were confirmed to bite. dead-code.md's line references are refreshed — the testability seams shifted most of them — and it now says which commit they were correct at and to confirm with the grep rather than trusting them. One entry added: the DefaultNostrSocketClientFactory overload taking an explicit HttpClient has no caller now that everything goes through the interface method. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kgothatso Ngako
|
c24cbed390 |
docs: record what the coverage work found, and what it left uncovered
Three additions. The decision the inbound path makes now has a name and a home -- MarmotDirectMessage.classify -- and the doc says why it is separate from the filing of it: only the filing needs a database, so splitting them is what lets the check that replaces MIP-03 be tested at all. A security property found while writing those tests, which I had asserted backwards. Relabelling a seal with another member's pubkey does not get as far as the signature check: NIP-44 derives the conversation key from the pubkey being claimed, so a relabelled seal is undecryptable by the person it was encrypted for. The label is bound to the key rather than asserted alongside it, and the outcome is a message the recipient genuinely cannot read. verify() catches the narrower case of a seal altered after signing in a way that survives decryption. An honest list of what has no automated test and why -- the recipient validation and the outbound id lookup (both need a database), the two transcript renderings (no Compose UI test dependency in this project), and anything touching a real MlsGroup. Better written down than rediscovered by someone assuming a green suite means the path is covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kgothatso Ngako
|
d110737f9a |
fix: keep a room's MlsGroup alive so a late message can still be read
Two events published in the same second reliably lose one of them. The receiver stores the kind:445 and produces nothing from it -- no inner event, no chat line, no error anybody sees, because MarmotGroupEvent is written before the message is decrypted and so survives while everything downstream silently does not. Observed as a FROST signing session that never started on the receiver: proposeSigning publishes the proposal and then the proposer's own nonce, the relay handed them back in the other order, and the proposal was dropped. The nonce is still sitting there filed against a session that will never exist. The same bug ate a dialect earlier, which then took out the artifact referencing it via a foreign key. MLS is specified to tolerate this. RFC 9420 says a receiver that gets generation N+1 before N keeps the intermediate keys so the older message can still be read, and quartz's SecretTree does exactly that, in a private skippedKeys map. What it does not do is persist it: exportSenderStates() returns the ratchet positions only, so saveState() drops the cache. NostrDao rebuilt the group from stored state for every inbound event, so the cache was empty every single time, and generation N arriving after N+1 failed `require(generation >= applicationGeneration)` and was swallowed. Terminal -- the key is derived from a ratchet that has moved past it, and nothing asks the sender to resend. This keeps the instance alive instead. MlsGroupCache holds one MlsGroup per room, and the inbound path goes through it, so skippedKeys survives from one message to the next. That covers the case that actually bites -- a burst arriving in one sync, decrypted one after another against the same tree -- which is what every bursty flow needs: proposeRitual sends two, addArtifact sends two, and addChapter sends one per paragraph plus one, of which only the ones arriving in ascending generation order survived. Reuse is conditional on the stored state still being exactly what the cache last wrote. Sending a message advances the sender ratchet and saves; so does adding a member. When that happens the cache rebuilds rather than carrying on from a group that has been overtaken -- which is what keeps this from being worse than no cache at all: the fallback is always the old behaviour, never a diverged ratchet. One lock per room, not one overall, because the group is mutable and decryption advances it: two events for the same room decrypted at once would corrupt the tree, and a busy room should not hold up a quiet one. **This is a mitigation, not the fix.** It does not survive a restart, and it does not survive another writer, so a long enough reorder still loses the message. The fix belongs in quartz -- carry skippedKeys through saveState/restore -- and quartz is a mavenCentral binary, not a fork, so it cannot be made here. docs/mls-skipped-keys.md has the analysis, the patch, the migration constraint on the persisted state format, and the three ways to actually land it. Not verified end to end: the proposal that exposed this cannot be recovered, since its generation is already past, so confirming the fix needs a fresh burst. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kgothatso Ngako
|
635cef9311 |
docs: write down how a direct message travels, and what it costs
The reasoning behind this is not recoverable from the code, which is the bar docs/README.md sets for having a document at all. Three things in particular would otherwise have to be rediscovered by whoever changes this next, and two of them are traps. Why the wrap uses a throwaway key rather than the sender's own -- and what that does not buy. It does not hide the sender from the group: MLS authenticates every application message to a leaf, so the identity is there regardless. What it costs is a carve-out in MIP-03's pubkey check and the sender's ability to ever read their own messages back. Why the check that carve-out removes is not a hole. The authorship claim moves from the wrap's plaintext pubkey to the seal's verified signature, bound to the MLS leaf that sent it -- strictly harder to forge than what it replaced. The one query that would broadcast one of these. What this builds is a genuine, correctly signed NIP-59 gift wrap, indistinguishable from what the NIP-17 path would be right to publish, and the only thing keeping it off a relay is that it never becomes a GiftWrapPayload. Written against what shipped rather than what was planned, so it records two deviations. senderIdentity is resolved in NostrDao rather than added to GroupEventResult.ApplicationMessage, because quartz is a binary dependency here and the local checkout is a reference copy, not a build input. And a failed validation drops the message and logs rather than throwing, because the caller is inside storeNostrEvent's transaction. The unbuilt parts are listed as absences rather than left implied: there is no member picker, so a private message can only be a reply to one somebody already sent, and nothing in the UI yet tells a user in words that the group can see who they messaged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kgothatso Ngako
|
c8c962e4f4 |
docs: inventory the unreferenced code in the sync and relay stack
Found while building the long-running sync. One item was orphaned by that
change; the rest was already dead and only became visible because the subsystem
was being read closely. Written down rather than deleted because several pieces
are one decision away from being wanted, and those decisions are not the sync
change's to make.
Every claim is "this identifier appears exactly once in composeApp/src, at its
own declaration", with the two things that method cannot see called out: Room
DAO methods are reached through generated code, and Compose entry points can be
invoked without a textual reference. The DAO cluster is flagged as the least
certain for exactly that reason.
Three findings are more than leftovers:
- RelaysSocketManager.userRelays is a field nothing ever writes. The
`userRelays` inside observeRelays is a different, shadowing local, so the
single-argument publishEvent always takes its FALLBACK_RELAYS branch and the
user's own relay list is never used for publishing. That is a bug wearing
dead code's clothes, and the fix is to populate the field, not to delete it.
- NostrPublisherRepository is entirely unreferenced, and it is the only
consumer of CachingImportRepository.importEvents. RelayPool and
RelaysSocketManager each take a cachingImportRepository parameter they store
and never dereference, satisfied by NO_OP_CACHING_IMPORT_REPOSITORY — so the
whole seam is a parameter passed from nowhere to nothing. Removing the
publisher lets the interface and both parameters go with it.
- sendAUTH is unused because NIP-42 is unimplemented, not because it is
surplus. AuthMessage is parsed and dropped, so a relay answering CLOSED with
auth-required is retried forever and can never succeed. Deleting sendAUTH
means deciding against authenticated relays; that is worth doing on purpose
or not at all. sendCOUNT and CountMessage are a similar matched pair — both
go or neither, since a CountMessage cannot arrive if nothing sends a COUNT.
isRecommendedRelay on the two request entities is separated out as its own risk
class: never written, never read, but a Room column, so it wants a migration
rather than a delete.
Ends with an order to do it in, cheapest and least risky first.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Kgothatso Ngako
|
385c58ba7e |
docs: rewrite the sync note as what exists rather than what to build
The design landed across the six commits before this one, so the note is now
describing code. Reorganised around that: the reasoning that made it worth
writing is unchanged, but "the shape to build" is now "how it holds together"
and points at the classes, and the numbered traps have become properties of the
thing rather than warnings about a thing that did not exist yet.
Three sections earn their place after the fact:
- the two timestamp decisions, which are the ones most likely to be "cleaned
up" by someone who has not read this: no `since` on kind 1059 because our
own wraps are stamped up to two days in the past, and no watermark on 445
even though it would be safe, because `limit` already bounds the burst.
- the four ways a group id can appear, which is why the group filter is
derived from the room list rather than wired at the join sites.
- "Not done", which was previously implicit in a staging plan: connectivity
changes, NIP-42 AUTH, the collector-per-socket router the design originally
called for, and the fact that the DM relay set is one relay.
The "suggested order" section is gone; git log is a better record of it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Kgothatso Ngako
|
178ddd0181 |
docs: write down how a long-running chat sync would work
Every chat sync today is a pull: a screen queues a request row, a pump drains
it, the relay answers, the subscription is closed. Nothing arrives between
pulls, so a message sent one second after EOSE waits for the next time someone
opens a screen.
This note works out what it takes to hold the two chat subscriptions open for
as long as the app is active — kind 1059 p-tagged to us, and kind 445 h-tagged
with every group we belong to — and, more usefully, what in the current
pipeline quietly assumes a subscription is short:
- completeOnSubscriptionEnd finishes the flow at EOSE, which is what releases
the slot and sends the CLOSE,
- SUBSCRIPTION_TIMEOUT hard-kills anything still open at 120s,
- subscriptionSlots is a Semaphore(4) shared with the backfill queue, so a
permanent subscription is a permanently-held permit,
- both saveNostrEvent overloads need a request row to attach provenance to
and to flip to "processed",
- and nothing in the app reconnects a dropped socket at all. That is
invisible today only because every subscription is short and the next
queued request re-opens the socket on its way out.
The design keeps the queue and its three pumps exactly as they are: live
subscriptions replace polling, not reconciliation. Negentropy stays the tool
for first login, the catch-up after a background gap, and "load older".
The group filter is derived from chatRepository.observeChatRoomListByPublicKey
rather than wired at each join site, because a group id can appear four ways
and only one of them (creating a group) is somewhere anyone would think to call
a subscribe function — being added arrives as a Welcome processed deep inside
NostrDao.storeNostrEvent. Observing the room list also closes the loop: a
Welcome lands on the live gift wrap subscription, a ChatRoom row is written,
the Flow re-emits, and the group filter widens without anyone opening a chat.
Two findings fell out of checking the details against our own code:
- `since = now` on kind 1059 would silently drop messages. Gift wraps are
stamped with TimeUtils.randomWithTwoDays(), so a wrap published now can
carry a created_at two days in the past. Kind 445 uses TimeUtils.now() and
can take a watermark — opposite treatment for the two kinds we care about.
- the "sent-messages" filter (kinds=[1059], authors=[me]) cannot match
anything, because gift wraps are signed with a fresh throwaway KeyPair().
It is also unnecessary: createNip17ChatRoom puts the user in their own
participant list, so we wrap a copy to ourselves and the account-wide
#p=[me] subscription already picks it up.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Kgothatso Ngako
|
3dea07135c |
fix: add a group's whole membership in one commit, closing the epoch race
`MarmotOutboundDao.addMembersToChatRoom` stages every member with `proposeAdd` and issues a single `commit()`. Both callers that know their membership up front now use it: `SelectChatRoomTypeViewModel.inviteMembers` at room creation, and `DkgRitualViewModel.inviteAdmins` for the #admins room. Inviting one at a time created an epoch per member, and each of those commits raced the previous member's welcome. MarmotInboundManager refuses future-epoch messages outright, on both wire formats, with no queue and no replay -- so the member who lost that race was silently stuck an epoch behind while the caller saw a successful invite. Deriving isOneMemberInitialGroupCreation narrowed that window; this removes it. No member ever has to process a commit for an epoch they were not yet in, so there is no longer a race to lose. One commit yields one welcome: `buildWelcome` emits an EncryptedGroupSecrets per added member and each joiner finds its own entry by key package reference. The blob is shared, delivery stays per peer, because each welcome event is tagged with that peer's key package. ## Why this needed no schema change Batching at creation time means the single commit happens while the group is still only its creator, which takes the immediate-welcome branch: nothing is broadcast and MarmotCommitResult is never written. The bookkeeping that assumes one peer per commit is simply not on this path. So the batch is taken only when `members().size == 1`, and anything else falls back to inviting sequentially -- correct, if not ideal. Batching into an established group would take the deferred branch, where `peerKeyPackageEventId` is singular and the ack-triggered delivery in DatabaseNostrRepository expects one welcome; making that work needs a list there and a fan-out on acknowledgement. Nothing currently adds several members to an established group, so that is left outstanding and documented rather than speculatively built. The group state is persisted after `commit()` and before any welcome goes out, so a crash between them leaves the group at the epoch the welcomes describe rather than one behind it. ## Reporting Members with no published key package still cannot be added -- a Marmot invite needs one -- and are now returned alongside any that failed to receive their welcome, rather than the two being conflated. Both still only reach the log; the coordinator is not yet told. docs/marmot-membership.md is updated in the same change: batching moves from outstanding work to described behaviour, with the schema constraint that shapes it and the remaining fan-out work recorded. The note about sequential invites is narrowed to where they still happen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kgothatso Ngako
|
8fc1c9e650 |
fix: stop deferring the first invitee's welcome behind a commit nobody needs
`inviteMember` now works out for itself whether the group it is adding to has
anybody to inform:
val isOneMemberInitialGroupCreation = mlsGroup.members().size == 1
read before `addMember` advances the tree. The parameter is gone from the
signature and no caller passes it any more.
Callers were the wrong place for this decision and both of them got it wrong.
`inviteMemberToChatRoom` hardcoded `false`, and `ChatRepository.inviteMember` did
not expose it at all, so every invite made through a group -- room creation in
SelectChatRoomTypeViewModel, and the #admins room -- took the deferred-welcome
path. That includes the first invite, when the group is still only its creator, at
which point:
- the commit has no audience. No other member exists, and nobody outside the
group can decrypt it, so it is noise on the relay.
- the welcome is then withheld until a relay acknowledges that noise. If the ack
never lands, the first invitee receives nothing at all.
Only createMlsDirectMessageChatRoom passed `true`, and only because a DM has
exactly one invite. A group of n has one such invite too -- the first -- and it was
not getting it.
The condition is right at any size, not just for DMs: "the group has nobody to
inform" is true exactly once. Invite two sees one member who must advance, invite
three sees two, and so on. Their commits are encrypted with
`commitResult.preCommitExporterSecret`, the epoch the earlier invitees received in
their own welcome, so they can decrypt and advance. `members()` skips empty leaves,
so this also stays correct for a group that has had members removed.
## What this does and does not fix
It removes a pointless commit and, with DefaultDMRelayList now a single relay, a
single point of failure sitting in front of every group's first member.
It also narrows a silent race rather than closing it. MarmotInboundManager refuses
future-epoch messages outright on both wire formats -- no queue, no replay -- so a
commit arriving before its recipient's welcome is dropped and that member never
advances, while the coordinator sees a successful invite. Previously both commits
went out before either welcome; now welcome 1 is sent before commit 2 exists, so
the first invitee is already at the right epoch. For n >= 3 the window between
welcome 1 and commit 2 remains.
Closing it needs the adds batched into one commit, which is the outstanding work
described in docs/marmot-membership.md. That doc is updated here to describe the
derived flag as current behaviour rather than a proposal, and to keep batching as
the remaining item.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Kgothatso Ngako
|
b99cb8fcd5 |
docs: write down the shared-key subsystem and how Marmot membership fails
First docs in the repo -- README.md is still the stock KMP template. Three documents plus an index, covering the parts whose behaviour is not recoverable by reading the code: where the reasoning lives in a protocol, where a failure mode is silent, or where a decision looked arbitrary and was not. marmot-membership.md is the one that earns its place. Everything about adding a member compiles, the invite reports success, and a member simply never appears -- and the reason is never in the invite code. It records that inviteMemberToChatRoom hardcodes isOneMemberInitialGroupCreation = false and that ChatRepository does not expose it, so every group invite takes the deferred-welcome path including the first, when the group is still just its creator and the commit has no audience at all. Then why that is silent rather than noisy: MarmotInboundManager refuses future-epoch messages outright, on both wire formats, with no queue and no replay, so a commit arriving before its recipient's welcome is dropped and that member never advances. EPOCH_RETENTION_WINDOW retains past epochs and does nothing for messages from ahead. Three options are set out with the per-invite correctness table, including the honest limit that the recommended one narrows the race without closing it. shared-key-derivation.md argues why the paths are not BIP32 -- no chain code exists, hardened derivation is impossible rather than unimplemented, and a FROST tweak takes the scalar as input so the chain code leaves the problem entirely. It records the x-only serialisation trap avoided by choosing the scalar directly, and states the rule that must not be broken: never reconstruct a derived key in the clear, because k = k' - t hands over the group key rather than one derived key. shared-key-ceremony.md covers the seven kinds, the three approval gates and why the coordinator's aggregations are deliberately not among them, faults as values rather than exceptions, and the transcript's idempotency-by-construction. It also writes down the invariant that produces no error when broken: pendingApproval must mirror the gates in advance, or the screen offers an approval that does nothing -- or none while the ritual sits still. Every factual claim was checked against the source rather than recalled, which turned up one correction worth having: there are two future-epoch refusals, for PrivateMessage and for Commit, so the drop covers both wire formats and not just one. Each document leads with the failure mode rather than the architecture, on the grounds that a failure is what sends somebody to docs in the first place, and each lists its known gaps -- including that none of this has run on a physical device. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |