Fix missing task instance history for non-running retries

[GayathriSrividya]

Cherry-picks the proposed fix from #67371 for priority issue #67238.

This updates failure handling to preserve task instance history for retry paths when the task is not in RUNNING state (except RESTARTING, which is already handled).

Also includes the companion test adjustment to avoid detached-session behavior in test_taskinstance.py.

Related: #67238
Related: #65366

Local validation

• uv run ruff format airflow-core/src/airflow/models/taskinstance.py airflow-core/tests/unit/models/test_taskinstance.py
• uv run ruff check --fix airflow-core/src/airflow/models/taskinstance.py airflow-core/tests/unit/models/test_taskinstance.py
• breeze run pytest airflow-core/tests/unit/models/test_taskinstance.py::TestTaskInstance::test_handle_failure_updates_queued_task_updates_state airflow-core/tests/unit/models/test_taskinstance.py::TestTaskInstance::test_task_instance_history_is_created_when_ti_goes_for_retry -xvs (failed locally: Docker not installed)
• uv run --project airflow-core pytest airflow-core/tests/unit/models/test_taskinstance.py::TestTaskInstance::test_handle_failure_updates_queued_task_updates_state airflow-core/tests/unit/models/test_taskinstance.py::TestTaskInstance::test_task_instance_history_is_created_when_ti_goes_for_retry -xvs (failed locally due missing provider module in environment)

[Vamsi-klu]

The guard flip from ti.state == RUNNING to ti.state != RESTARTING in fetch_handle_failure_context looks right. A retry-eligible failure from QUEUED or DEFERRED never reached prepare_db_for_next_try before, so its history row got silently dropped, and RESTARTING is correctly the one state left out since the clear already recorded it. That lines up with is_eligible_to_retry, where RESTARTING is the always eligible cleared-while-running path. The ti = session.merge(ti) change in test_handle_failure_updates_queued_task_updates_state is right too, since the old line discarded the instance the session actually tracks. I checked that RESTARTING and RUNNING both behave the same under either guard, so this only closes the non-running gap.

[GayathriSrividya]

The guard flip from ti.state == RUNNING to ti.state != RESTARTING in fetch_handle_failure_context looks right. A retry-eligible failure from QUEUED or DEFERRED never reached prepare_db_for_next_try before, so its history row got silently dropped, and RESTARTING is correctly the one state left out since the clear already recorded it. That lines up with is_eligible_to_retry, where RESTARTING is the always eligible cleared-while-running path. The ti = session.merge(ti) change in test_handle_failure_updates_queued_task_updates_state is right too, since the old line discarded the instance the session actually tracks. I checked that RESTARTING and RUNNING both behave the same under either guard, so this only closes the non-running gap.

Thanks for validating the guard change and the test fix. I’ll wait for formal review and follow up if anything else comes up.