Note Indexer & Sync Engine

Two integration paths — the high-level RPC indexer with built-in persistence, and the framework-agnostic sync engine that drives the React hooks. Plus the building blocks (decryption, nullifier verification, storage adapters) for custom indexers.

UPP scans EVM logs to discover the encrypted notes you own. There are two supported entry points, both exported from @permissionless-technologies/upp-sdk/indexer:

Entry point	When to use
`createSyncEngine`	New integrations. Framework-agnostic, dual-rail (SNARK + STARK V3 + STARK V4), V4 commit/flush queue handling, optional STARK nullifier verifier. This is what the React hooks compose internally.
`makeRpcIndexer`	Legacy SNARK-only callers. Self-contained `Indexer` interface that owns its own `IndexedNote` storage and exposes a balance API. Still maintained for back-compat but feature frozen — new rails ship through the sync engine.

The lower layers — decryptCandidates, verifyNullifiers, verifyStarkNullifiers, storage adapters — are exported individually so anyone can build a custom indexer (Subsquid, subgraph, etc.) without inheriting either of the above.

Sync engine architecture

The pipeline is single-pass per cycle: scan, decrypt, repair, verify, persist. Live mode wraps it in a setInterval plus watchContractEvent subscriptions on CommitmentInserted and both StarkCommitmentInserted topics so a fresh shield or transfer kicks off the next cycle immediately.

`createSyncEngine`

import { createSyncEngine } from '@permissionless-technologies/upp-sdk/indexer'
import { createIndexedDBAdapter } from '@permissionless-technologies/upp-sdk/indexer'
import { createNoteStore } from '@permissionless-technologies/upp-sdk' // INoteStore

const engine = createSyncEngine({
  client: publicClient,           // viem PublicClient
  contractAddress: poolAddress,
  swapModuleAddress,              // optional — only if you index swap events
  noteStore,                      // INoteStore (in-memory + adapter-backed)
  poseidonFn,                     // BN254 Poseidon (for SNARK nullifier hashing)

  // SNARK config — always required
  ownerHash,                      // Poseidon(spendingSecret)
  spendingSecret,                 // bigint
  encryptionPrivKey,              // X25519 32-byte private key

  // STARK config — pass these to also discover STARK notes
  starkOwnerHash,                 // uint128, 4-limb M31 LE-packed
  starkSecret,                    // M31Secret (8 limbs); enables STARK nullifier reverse-check

  fromBlock: deploymentBlock,     // start scanning here
})

await engine.sync()                                    // one-shot
engine.startLiveSync({ intervalMs: 60_000 })           // background poll + watchers
engine.getStarkStateLeaves()                           // ordered V3 leaf stream
engine.getStarkStateLeavesV4()                         // ordered V4 leaf stream

The dual STARK getters are how transfer/withdraw proofs are built: each returns the global stream of commitment[0] (first M31 limb) values in leafIndex order, which Poseidon31MerkleTree.build(...) consumes to compute Merkle paths and the state root. The V3 stream feeds a binary depth-32 tree; V4 feeds the octary depth-11 tree documented on the V4 Octary State Tree page.

Why starkSecret is separate from ownerHash

A STARK note's on-chain payload is 32 bytes plaintext — it carries the 4-limb ownerHash but never the secret. The decoder fills ShieldedNote.ownerSecret with the ownerHash as a placeholder, so the SNARK-style "derive nullifier from per-note secret" recipe doesn't work. The verifier needs your master starkSecret separately to recompute nullifier = poseidon31(starkSecret, leafIndex, commitment).

This is also the only path that heals STARK spent-state after an IndexedDB wipe: the optimistic UI mark on transfer is local-only, so a cold sync without starkSecret brings input notes back as confirmed.

What the sync engine produces

Each sync() returns a small summary; the meaningful state changes land on the NoteStore you passed in:

const { discovered, repaired, candidatesScanned } = await engine.sync()
// discovered:        newly added notes
// repaired:          leafIndex corrections + nullifier-driven status changes
// candidatesScanned: total event payloads attempted

Per-note fields the engine sets:

Field	Set by	Meaning
`commitment`	decryptor	Always set on discovery
`leafIndex`	repair pass (step 4)	Index in the rail's tree; `-1` means not yet inserted
`proofSystem`	decryptor	`'snark'` or `'stark'`
`starkTreeVersion`	producer-side tag (step 3)	`'v3'` or `'v4'` for STARK notes
`pendingQueueIndex`	V4 commit-path join	FIFO queue index for V4 notes awaiting flush; undefined for atomic-path notes
`status`	nullifier verifiers (step 5)	`'pending'`, `'confirmed'`, or `'spent'`

The leafIndex < 0 && pendingQueueIndex !== undefined combination is how the UI distinguishes "transfer is mined, awaiting flush" from "transfer is in the mempool" — see the Commit/Flush page for the lifecycle.

Event topology

The sync engine reads from these event topics on the pool (and optionally the swap module):

Topic	Source	Purpose
`Shielded(token, commitment, leafIndex, encryptedNote, depositor)`	Pool	New SNARK or STARK note from a deposit
`CommitmentInserted(commitment, leafIndex)`	Pool	SNARK tree insert — feeds the SNARK leafIndex map
`StarkCommitmentInserted(commitment, leafIndex)`	Pool	STARK V3 tree insert — feeds V3 leafIndex map + leaf stream
`StarkCommitmentInsertedV4(commitment, leafIndex)`	Pool	STARK V4 tree insert — feeds V4 leafIndex map + leaf stream
`Transferred(outputCommitment1, outputCommitment2, encryptedNote1, encryptedNote2, nullifier1?, nullifier2?)`	Pool	Shared across SNARK / STARK V3 / STARK V4 transfers
`StarkTransferV4Committed(nullifier, queueIndex1, queueIndex2, …)`	Pool	V4 commit-path only; provides the FIFO queue index for `pendingQueueIndex`
`Merged`, `MergeTransfer`	Pool	Used to populate `m31OriginMap` so STARK decoder can resolve full addresses
`SwapOrderFilled`, `SwapOrderClaimed`, `SwapOrderCancelled`	Swap module (optional)	Swap output notes

V3 and V4 events live on the same emitter but distinct topic hashes; commitments are strictly partitioned between rails, so the lookup tries each map in order and exactly one resolves.

Decryption — the building block

If you're writing a custom indexer (Subsquid handler, subgraph mapping, server-side scanner), import the decryptor directly:

import {
  decryptCandidates,
  tryDecryptHashBased,
  matchesSearchTag,
  computeSearchTag,
  type NoteCandidate,
  type HashDecryptionConfig,
} from '@permissionless-technologies/upp-sdk/indexer'

const candidates: NoteCandidate[] = events.map(e => ({
  commitment: e.commitment,
  leafIndex: e.leafIndex,
  packedData: e.encryptedNote,
  txHash: e.txHash,
  tokenAddr: e.token,    // STARK decoder uses this to resolve truncated tokens
}))

const notes = await decryptCandidates(
  candidates,
  { ownerHash, spendingSecret, encryptionPrivKey, starkOwnerHash },
  poseidonFn,
  m31OriginMap,          // optional — needed for STARK address recovery
  m31TokenMap,           // optional — needed for STARK transfer notes
)

Routing is by payload length: exactly 32 bytes goes to the STARK plaintext decoder; anything longer is treated as a v6 ECIES envelope (ownerHash ‖ ephPub ‖ iv ‖ ciphertext). There is no v4/v5 fallback — the envelope is a hard cut.

STARK address recovery requires the reverse maps

A STARK note's payload only carries 31-bit addressToM31(...) truncations of origin and token. Decoding without m31OriginMap / m31TokenMap yields zero-padded pseudo-addresses (e.g. 0x000…7862cae7) that no wallet UI can resolve. The sync engine builds both maps from Shielded, Merged, and MergeTransfer events — replicate that if you're scanning STARK notes outside the engine.

Nullifier reconciliation

Both verifiers are exported so custom indexers can run the same reverse-check the sync engine does:

import {
  verifyNullifiers,            // SNARK rail — _nullifierUsed
  verifyStarkNullifiers,       // STARK rail — _nullifierUsed{V3,V4}
  packDigestToBytes32,         // STARK packing helper
  parseCommitmentDigest,       // STARK commitment unpacking
} from '@permissionless-technologies/upp-sdk/indexer'

The two rails use disjoint on-chain registries:

Rail	Storage key	Selector	Hash scheme
SNARK	`_nullifierUsed[bytes32]`	`nullifierUsed`	BN254 Poseidon over `(ownerSecret, leafIndex, commitment)`
STARK V3	`_nullifierUsed[bytes32]` (legacy STARK pool adapter)	`nullifierUsed`	Poseidon31 over M31 limbs
STARK V4	`_nullifierUsedV4[bytes32]`	`nullifierUsedV4`	Same Poseidon31 hash; partitioned registry

A nullifier marked on one rail does not block a spend on the other. The sync engine runs both verifiers per cycle when starkSecret is configured; ordering is SNARK first, STARK second, so per-tx logs read top-down through the rails.

`makeRpcIndexer` (legacy)

The original entry point. Self-contained — it owns its own Indexer state, picks a storage adapter automatically, and exposes a balance API. Still maintained for SNARK-only callers that don't want to pull in NoteStore or the sync engine wiring.

import { makeRpcIndexer } from '@permissionless-technologies/upp-sdk/indexer'

const indexer = makeRpcIndexer({
  client: publicClient,
  contractAddress: poolAddress,
  chainId: 11155111,
  spendingSecret,
  spendingPubkey,
  masterViewingSecret,
  masterViewingPubKey,
  fromBlock: deploymentBlock,
  batchSize: 200,
})

await indexer.sync()
const balance = indexer.getBalance(token)
const unspent = indexer.getUnspentNotes()
indexer.startLiveSync({
  intervalMs: 30_000,
  onNewNotes: (notes) => console.log('new:', notes.length),
  onNotesSpent: (commitments) => console.log('spent:', commitments.length),
})

The legacy indexer uses an older IndexedNote shape (oneTimeSecret, oneTimePubkeyX/Y) instead of the current ShieldedNote (ownerSecret, ownerHash) — see the @deprecated comment on IndexedNote in indexer/types.ts. New work should use createSyncEngine with a NoteStore.

Storage adapters

Three adapters ship with the SDK plus an auto-selector. Pass one as the storage option on makeRpcIndexer, or wire one into a NoteStore for the sync engine path.

import {
  createIndexedDBAdapter,    // browser default
  createLocalStorageAdapter, // smaller browser fallback (≤ 5 MB)
  createMemoryAdapter,       // tests / SSR
  createAutoAdapter,         // picks the best available
} from '@permissionless-technologies/upp-sdk/indexer'

const storage = createAutoAdapter('upp-account-0x1234')
// → IndexedDB if available, else localStorage, else in-memory

The StorageAdapter interface is plain get/set/delete/clear plus an isAvailable() probe — implement it against any KV store (Redis, SQLite, etc.) for server-side indexers.

Custom indexers

A complete server-side scanner using just the building blocks:

import {
  decryptCandidates,
  verifyNullifiers,
  verifyStarkNullifiers,
} from '@permissionless-technologies/upp-sdk/indexer'

// 1. Pull events from your data source (Subsquid, subgraph, archival node)
const events = await mySubsquidQuery(/* ... */)

// 2. Build maps + candidates the same way the sync engine does
const candidates = buildCandidates(events)
const { m31OriginMap, m31TokenMap, commitmentLeafMap, ... } = buildReverseMaps(events)

// 3. Decrypt
const notes = await decryptCandidates(
  candidates,
  decryptionConfig,
  poseidonFn,
  m31OriginMap,
  m31TokenMap,
)

// 4. Reconcile nullifiers (forward + reverse check)
await verifyNullifiers(noteStore, commitmentLeafMap, poolAddress, client, poseidonFn)
await verifyStarkNullifiers(noteStore, poolAddress, client, starkSecret)

// 5. Persist via your own storage layer

Your viewing key never leaves the device — every helper takes secrets as parameters and reads only public chain state.