Semantic Invariant Enforcement at the Store-Service Boundary [HIGH]

[HayimShaul]

Description
Before committing any write, the store service should validate that the payload satisfies the structural and relational invariants of the target table: a TokenRequest payload must deserialize without error; a Transaction row must reference an existing Requests.tx_id; a TokenLock may only be created for a (tx_id, idx) that exists in Tokens and is not already held by a different consumer; a Wallet row's identity_hash must resolve to a registered identity. These checks prevent malformed, duplicate, or logically impossible records from corrupting vault state, audit trails, and balance calculations. Architects should be aware that exhaustive re-validation against large datasets (e.g., checking every existing token entry on each write) can impose significant latency penalties at scale. The recommended approach is to enforce cheap structural checks unconditionally, and to scope relational/cross-table checks to the minimum set of invariants whose violation would cause downstream correctness failures -- deferring expensive consistency sweeps to background reconciliation jobs rather than the hot write path.

*Short Summary
Table 1: Where Missing Checks Should Be Implemented

    | Validation Type | DB Schema | Application Layer | Recommendation |
    |----------------|-----------|-------------------|----------------|
    | **Unique IDs** | ✅ Implemented | N/A | **Keep in DB** - Primary keys are perfect for this |
    | **Referential Integrity** | ✅ Partial | N/A | **Add missing FKs in DB** - Cheap structural checks |
    | **Required Fields** | ✅ Implemented | N/A | **Keep in DB** - NOT NULL is efficient |
    | **Data Types** | ✅ Implemented | N/A | **Keep in DB** - Type system handles this |
    | **TokenRequest Deserialization** | ❌ Can't do | ✅ **Add here** | **Application Layer** - Requires protobuf parser |
    | **Signature Verification** | ❌ Can't do | ✅ **Add here** | **Application Layer** - Requires crypto library |
    | **Identity Validation** | ❌ Can't do | ✅ **Add here** | **Application Layer** - Requires X.509/key parsing |
    | **TokenLock Existence Check** | ✅ **Add FK** | ✅ **Add check** | **Both** - FK for structure, app for race conditions |
    | **Cross-Table Consistency** | ❌ Too complex | ✅ **Add here** | **Application Layer** - Requires aggregations |
    | **Business Logic** | ❌ Can't express | ✅ **Add here** | **Application Layer** - Requires custom rules |
    | **PP Hash Validation** | ✅ **Add FK** | ✅ **Add check** | **Both** - FK for reference, app for hash matching |

[SurbhiAgarwal1]

Hi @HayimShaul, I would like to work on this issue.

After reviewing the issue description, my understanding is that the primary goal is to enforce semantic and relational invariants at the store-service boundary while keeping the hot write path efficient and avoiding expensive full-state validation.

Proposed approach:

• Introduce validation hooks at the store-service boundary before persistence operations are committed.

• Separate validations into:

  • cheap structural checks enforced unconditionally
  • scoped relational/business validations enforced selectively
  • expensive consistency checks deferred to reconciliation/background jobs

Planned validation areas:

• Structural Validation

  • TokenRequest deserialization validation
  • required payload/schema validation
  • payload format and type consistency

• Referential Integrity

  • validate Transaction references against existing Requests.tx_id
  • validate Wallet identity_hash against registered identities
  • validate TokenLock references against existing Tokens entries

• TokenLock Semantic Validation

  • ensure (tx_id, idx) exists before lock creation
  • prevent conflicting ownership/consumer locks
  • add transactional/race-aware checks where DB-level constraints alone are insufficient

• DB + Application Layer Coordination

  • keep cheap guarantees (PKs, NOT NULL, FKs) at the database layer
  • implement semantic/business invariants in the application layer
  • avoid duplicating validations already efficiently enforced by the DB schema

• Performance Considerations

  • minimize cross-table scans on the write path
  • scope validation queries to only correctness-critical invariants
  • design the validation layer to remain extensible for future reconciliation/audit jobs

I’ll first review the existing store-service flow, schema constraints, and transaction boundaries to identify the appropriate insertion points for invariant enforcement before proposing implementation changes.

[HayimShaul]

• Regarding checking that TokenRequest is deserializable:

The database is written to in 2 cases: (i) when the transaction is still pending (ii) after validation.

In the latter case, the TX is validated, hence is deserializable and must be written to DB to avoid divergence.

For a pending TX it either originated from the local machine or from read from the network.
A TX originated locally is surely deserializale.

A TX received from the network is unmarshalled when read (receivetx.go)

	tx, err := NewTransactionFromBytes(context, msg)
	if err != nil {
		logger.WarnfContext(context.Context(), "failed creating transaction from bytes: [%v], try to unmarshal as signature request...", err)
		// try to unmarshal as SignatureRequest
		var err2 error
		tx, err2 = f.unmarshalAsSignatureRequest(context, msg)
		if err2 != nil {
			return nil, errors.Wrap(errors.Join(err, err2), "failed to receive transaction")
		}
	}

So we conclude unmarshalling is not needed

[HayimShaul]

• regarding referencing an existing tx_id
  This is enforced by the database schema

in transaction.go:

	-- transactions
		CREATE TABLE IF NOT EXISTS %s (
			id CHAR(36) NOT NULL PRIMARY KEY,
			tx_id TEXT NOT NULL REFERENCES %s,
			action_type INT NOT NULL,
			sender_eid TEXT NOT NULL,
			recipient_eid TEXT NOT NULL,
			token_type TEXT NOT NULL,
			amount NUMERIC(78, 0) NOT NULL,
			stored_at TIMESTAMP NOT NULL
		);

We conclude that tx_id does not need to be checked

[HayimShaul]

regarding checking TokenLock:

1. checking for double locking is done by Lock(..) in tokenLock.go

2. trying to lock a nonexisting token is also not possible since we lock tokens whose ids we got from the database

		for {
			t, err := unspentTokens.Next()
			numNext++

			// lock the token
			if _, lockErr := s.locker.Lock(ctx, &t.Id, s.txID, reclaim); lockErr != nil {
				var addErr error

[HayimShaul]

Regarding checking that a wallet matches an existing identity.

Creating a wallet happens only after creating the identity and
since identity are never deleted, we conclude the there is no need to check wallets match an existing ID.

[HayimShaul]

I suggest to close this issue.

[SurbhiAgarwal1]

Hi @HayimShaul,

Thank you for the detailed breakdown! Your points about avoiding redundant unmarshalling and keeping the hot write path as efficient as possible make complete sense.

Before we decide to close this, I wanted to bring up one architectural consideration regarding the relational constraints (like tx_id REFERENCES and wallet/identity consistency):

1. Driver-Agnostic Consistency (SQL vs. KVS):
   While SQL-based storage drivers (Postgres/SQLite) naturally enforce foreign keys at the database schema level, the Token SDK also supports non-relational Key-Value Store (KVS) storage drivers (such as HashiCorp Vault or in-memory maps). In these KVS backends, relational invariants cannot be guaranteed by the database schema itself.

   Should the store-service boundary enforce these semantic relations in the application layer to ensure identical safety guarantees across all storage drivers, or is it acceptable to rely strictly on the database schema for drivers that support it?

2. Dangling Token Locks:
   Regarding TokenLock, if the locking/unlocking API is ever exposed to external service layers or direct client orchestrations, a sanity check ensuring that the token exists in the system before a lock is registered could prevent orphaned or dangling locks. If it remains entirely internal to the transaction builder flows, then the current assumptions are indeed safe.

If you feel that the current boundaries are sufficient for the scope of the project and that KVS drivers do not need application-level relational checks, I am completely fine with closing this issue. Let me know your thoughts!