fix: resolve lock-wait timeouts and cut repost overhead in loan write off and repayment repost (backport #1315)

[mergify]

Summary

Production hit intermittent Lock wait timeout exceeded (1205), Deadlock found (1213), and TimestampMismatchError during bulk Loan Write Off submission and Loan Repayment Repost, plus reposts that ran for hours.

This PR fixes the root causes:

1. Bulk Loan Write Off held one long transaction across the whole batch.
2. Loan Repayment ran its repost inline inside the repayment job, holding locks for the full repost.
3. The repost did large amounts of per-document overhead (UI realtime events, one DB query per iteration, repeated interest recomputation).

---

Root Cause Analysis

Issue 1 — Bulk Loan Write Off: Lock wait timeout exceeded (1205)

What happened (before)

Frappe's bulk action submitted all documents in one request = one transaction and only committed at the end.

Every Loan Write Off posts a Journal Entry into the same shared suspense/waiver accounts. So the whole batch serialised on those account rows, holding locks for the entire batch.

When a concurrent process needed the same rows for longer than innodb_lock_wait_timeout (120s in client prod), it failed with 1205. In the client log the batch broke at document 13 of 30.

The same contention also surfaced as Deadlock (1213) and TimestampMismatchError, the classic trio of concurrent-write contention on shared rows.

Why we changed it

The documents do not need to share a transaction. Each Loan Write Off should submit and commit independently so its account locks release immediately.

The change

Set queue_in_background: 1 on the Loan Write Off DocType.

Frappe's bulk action then routes each document through queue_submission, giving each document its own background job = its own transaction.

Single (non-bulk) submits are unaffected.

+ "queue_in_background": 1,

Before / After (measured on dev, 50 real loans, rolled back)

innodb_lock_wait_timeout = 50s (dev)

BEFORE (queue_in_background = 0):
  inline doc.submit() calls : 50   <- all in ONE transaction
  queue_submission calls    : 0
  Reproduced: QueryTimeoutError (1205, 'Lock wait timeout exceeded') after 50.4s

AFTER (queue_in_background = 1):
  inline doc.submit() calls : 0
  queue_submission calls    : 50   <- each write off = its OWN job/transaction
  No shared lock hold; contention removed.

Extra benefit

One failing write off no longer aborts the batch. Each is an isolated, independently retryable job.

Failures surface in the Submission Queue instead of instantly in the UI.

---

Issue 2 — Loan Repayment Repost: 1205 on Repost Detail insert

What happened (before)

LoanRepayment.create_repost created a Loan Repayment Repost and called repost.submit() inline, inside the repayment's own background job.

A repost regenerates hundreds of accruals, demands, GL and Journal Entries and takes for_update row locks.

Running it in the repayment's transaction kept those locks held for the full repost, so concurrent repayments on the same loan/applicant collided and timed out on the Loan Repayment Repost Detail insert.

Why we changed it

The repost should not extend the repayment's transaction — it should run as its own background job so the repayment commits and releases its locks immediately.

The change

create_repost now inserts the repost and enqueues it as its own background job (enqueue_after_commit=True) instead of submitting inline. Each repost is enqueued under a unique job_id (loan_repayment_repost::{repost.name}), so every repost is enqueued and processed and none is left orphaned in draft.

Note: this does not serialise concurrent reposts for the same loan; each repost runs as an independent job. In the normal flow, backdated repayments are processed sequentially. If strict per-loan serialisation is required, it can be added in a follow-up (per-loan job_id + a pick-any-pending drain of Queued reposts).

- repost.submit()
+ repost.insert()
+ if frappe.flags.in_test:
+     repost.submit()
+ else:
+     frappe.enqueue(
+         process_loan_repayment_repost,
+         queue="long",
+         timeout=36000,
+         enqueue_after_commit=True,
+         job_id=f"loan_repayment_repost::{self.name}",
+         deduplicate=True,
+         repost=repost.name,
+     )

Before / After

Before

• Repayment job transaction spans the whole repost.
• Locks held for minutes.
• 1205 lock wait timeouts.

After

• Repayment commits and releases locks immediately.
• Repost runs in its own job.
• The per-loan dedup key serialises reposts for the same loan.

---

Issue 3 — Repost duration: per-document overhead

A production profile of one repost the loan showed 12,915 s (~3.6 h) for ~200,000 document operations (If Repost from two years ago).

Three removable overheads dominated.

3a. Realtime UI events (~31% of the profile, 207,684 calls)

Every document save/cancel fired notify_update -> publish_realtime (websocket/redis) to update a UI that nobody watches during a background repost.

Change

Set flags.notify_update = False on the accrual/demand documents created and cancelled under frappe.flags.on_repost.

This is honoured by core:

if self.flags.get("notify_update", True):
    self.notify_update()

No financial data changes.

---

3b. One DB query per iteration

calculate_penal_interest_for_loans, make_credit_note_for_charge_waivers and the charge GL path each ran one get_value per demand/row.

Change

Batch-fetch once with get_all(... "name"/detail in [...]).

The penal-interest principal lookup is filtered by:

• loan
• repayment_schedule_detail
• EMI
• Principal

This combination is not unique. Cancelled and submitted duplicates coexist (common right after a repost).

The original get_value returned the row ordered by:

creation DESC
LIMIT 1

(Loan Demand default sort_order is creation desc.)

The batch version mirrors this exactly:

• order_by="creation desc"
• keep the first row per detail

Verified against get_value across 144 duplicate details on 26 loans (including an 11-way duplicate) with 0 mismatches.

The other two maps key by name (primary key, unique), so no selection ambiguity.

---

3c. Repeated interest recomputation

get_per_day_interest was recomputed for every day in the penal date range, though its only date-dependent input is the year divisor.

Change

Cache get_per_day_interest by calendar year within the loop.

Before / After (measured on dev, real data, rolled back)

BENCHMARK 1 — penal-interest principal lookup
(loan with 23 principal demands, 20 iterations)

BEFORE:
  460 DB calls
  279.9 ms
  (one get_value per demand)

AFTER:
   20 DB calls
   74.7 ms
  (one batched get_all)

-> 96% fewer DB calls
-> ~3.7x faster


BENCHMARK 2 — get_per_day_interest year-cache
(730-day date range)

BEFORE:
  0.6 ms
  (recomputed every day)

AFTER:
  0.1 ms
  (computed once per year)


Realtime events (from the production profile):

~207,684 publish_realtime calls
(~31% of the 3.6h)
removed for repost-created/cancelled docs.

Dev run measured 8,880 removable events on a ~4k-document repost.

---

Also included: master-read caching (Issue 1/2 support)

Repeated frappe.db.get_value("Loan Product"/"Company", ...) for the same master record within a transaction were converted to frappe.get_cached_value in the repayment, write off and loan paths.

Masters are immutable within a transaction, so the row is fetched once and reused.

Mutable reads (e.g. Loan.status, which the code writes mid-transaction) were left as live get_value to avoid stale reads.

---

This is an automatic backport of pull request #1315 done by Mergify.

[mergify]

Cherry-pick of 34912b1 has failed:

On branch mergify/bp/version-2-beta/pr-1315
Your branch is up to date with 'origin/version-2-beta'.

You are currently cherry-picking commit 34912b13.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   lending/loan_management/doctype/loan_repayment/loan_repayment.py
	modified:   lending/loan_management/doctype/loan_write_off/loan_write_off.json
	modified:   lending/loan_management/doctype/loan_write_off/loan_write_off.py

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   lending/loan_management/doctype/loan/loan.py

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request