F/lshell completion upgrade

[amirimatin]

Summary

This PR adds a safe, lshell-owned set of interactive usability improvements inspired by common zsh workflows, while keeping command completion, path visibility, and history behavior under existing lshell policy enforcement.

The work is focused on three areas:

1. Safer and more useful history navigation
2. Clearer and more predictable completion output
3. Consistent history normalization and persistence

Main Changes

1. Prefix-based history navigation on Up / Down

Interactive history navigation now supports prefix search on the current input line.

Behavior:

• When the user types a prefix such as echo a and presses Up, lshell searches backward through history for entries that start with that exact prefix.
• Repeated Up continues moving through older matching entries.
• Down moves forward through the same filtered match set.
• When the user reaches the newest match and presses Down again, the original typed line is restored.

Implementation details:

• Readline setup was moved into a dedicated initialization path in lshell/shellcmd.py.
• GNU Readline callback bindings are used when available to emulate history-beginning-search behavior directly inside lshell.
• The search is prefix-based up to the current cursor position.
• Duplicate matches are removed during navigation so repeated identical history entries do not create noisy cycling.

Key code references:

• lshell/shellcmd.py:486-576
• test/test_session_interaction_functional.py:162-205
• test/test_history_size_unit.py:192-234

2. Incremental history search with Ctrl+R and Ctrl+S

Interactive history search now exposes standard readline incremental search bindings.

Behavior:

• Ctrl+R starts reverse incremental history search
• Ctrl+S starts forward incremental history search

This is wired through readline bindings only. The search operates on the shell history already managed by lshell; it does not introduce any new execution path or external completion engine.

Key code references:

• lshell/shellcmd.py:29-34
• lshell/shellcmd.py:486-510
• test/test_history_size_unit.py:193-220

3. Context-aware completion display

Completion output is now rendered with an lshell-specific display hook instead of relying on the default raw readline listing.

Behavior:

• Completion results are shown with a contextual header such as:
  • Allowed commands
  • Allowed sudo commands
  • Allowed directories
  • Allowed paths
• Candidate lists are de-duplicated
• Candidate lists are sorted case-insensitively
• Output is columnized based on terminal width for more readable interactive output

This makes Tab / Tab-Tab behavior easier to review during interactive use, especially when the same completion path can return many entries.

Key code references:

• lshell/shellcmd.py:161-166
• lshell/shellcmd.py:578-617
• lshell/shellcmd.py:793-798
• test/test_completion.py:71-145

4. Improved allowed-command completion matching

Allowed-command completion now performs a case-insensitive fallback when a case-sensitive prefix match returns nothing.

Behavior:

• Exact prefix matching still runs first
• If no exact-case matches exist, completion retries using a case-insensitive prefix check
• The same behavior also applies to explicitly allowed relative commands such as ./tool.sh

This improves interactive usability without expanding the completion source beyond the configured allowed list.

Key code references:

• lshell/completion.py:18-48
• test/test_parser_jobs_unit.py:303-313

5. Centralized history normalization and persistence policies

History cleanup logic has been extracted into a dedicated helper module so interactive history state and persisted history follow the same rules.

New behavior:

• Redundant unquoted whitespace is reduced before persistence
• Whitespace inside quoted segments is preserved
• Immediate duplicate entries are removed at interactive-entry time
• Persisted history removes older duplicates and keeps the most recent normalized form
• The built-in history command now writes prepared history before reading it back for display, so on-screen history reflects the same cleanup rules as the saved history file

This makes history behavior consistent across:

• the current readline session
• history output
• shutdown-time history writes

Key code references:

• lshell/history.py:1-105
• lshell/shellcmd.py:618-625
• lshell/shellcmd.py:711-712
• lshell/shellcmd.py:727-739
• lshell/builtincmd.py:107-119

6. History safety fixes in interactive edge cases

While wiring the new history behavior, two important edge cases were fixed so history normalization only touches real user-entered readline input.

Fixes:

• Ctrl+D / EOF exit no longer risks persisting the synthetic line value "EOF" into history
• cmdqueue entries are no longer treated as fresh interactive readline history and therefore are not normalized as if the user had typed them

These fixes keep history cleanup limited to genuine interactive input.

Key code references:

• lshell/shellcmd.py:656-712
• test/test_history_size_unit.py:115-190
• test/test_session_interaction_functional.py:238-269

Review Guide

The fastest review path is:

1. lshell/shellcmd.py

   • readline setup
   • history-search state machine
   • completion display hook
   • guarded history normalization and write path

2. lshell/history.py

   • line normalization
   • latest-entry cleanup
   • persisted-history deduplication

3. lshell/completion.py

   • case-insensitive fallback for allowed command completion

4. lshell/builtincmd.py

   • history command persistence consistency

5. Tests

   • test/test_history_size_unit.py
   • test/test_session_interaction_functional.py
   • test/test_completion.py
   • test/test_command_execution.py
   • test/test_parser_jobs_unit.py

Test Coverage Added and Updated

Unit coverage

• Prefix-history binding installation and fallback behavior
• UTF-8-safe cursor offset mapping for readline byte offsets
• Prefix-history traversal with duplicate suppression
• History line normalization rules
• Interactive latest-entry duplicate removal
• Persisted-history deduplication rules
• Case-insensitive allowed-command completion
• Guard behavior for EOF and queued commands

Functional coverage

• Up / Down prefix-based history recall
• history output after normalization and deduplication
• Ctrl+D history persistence regression coverage
• Completion output for:
  • ~/... directory completion
  • ls ~/... file completion
  • ls -l ~/... argument-aware path completion
  • ./command allowed-command completion

Interactive test harness hardening

Interactive pexpect suites now sanitize inherited LSHELL_ARGS and LPS1 so spawned lshell sessions behave deterministically across environments and review runs.

Scope of Changed Files

• lshell/shellcmd.py
• lshell/history.py
• lshell/completion.py
• lshell/builtincmd.py
• test/test_history_size_unit.py
• test/test_session_interaction_functional.py
• test/test_completion.py
• test/test_command_execution.py
• test/test_parser_jobs_unit.py

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.