Skip to content

Split srch flat collector into interactive builder and reusable mover #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ib-bsb-br wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Split srch flat collector into interactive builder and reusable mover #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 137 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,137 @@
`<!-- placeholder for codex to edit -->`
[[
<purpose>
Provide a formal, comprehensive specification for a two-script refactor that separates an interactive srch command builder from a reusable per-file mover, while preserving flat-destination moves, deterministic collision handling, and auditable logging.
</purpose>

<context>
<environment>
<execution_user>root (invoked via sudo by user linaro)</execution_user>
<script_locations>
<builder>/usr/local/bin/srch_flat_collect</builder>
<mover>/usr/local/libexec/srch_move_one</mover>
</script_locations>
<destination_root>/home/linaro/</destination_root>
<log_location>operation_log.txt inside the destination folder</log_location>
</environment>

<constraints>
<constraint>All matched files are moved into a single flat directory under /home/linaro.</constraint>
<constraint>srch -r must invoke the per-file mover script for each matched path.</constraint>
<constraint>Collision handling must use deterministic numeric suffixes before extensions.</constraint>
<constraint>Moves must use copy+delete semantics to support cross-filesystem operations.</constraint>
<constraint>Ownership must be linaro:linaro and permissions must allow read/write for owner, group, and others (a+rwX).</constraint>
<constraint>No archive or zip operation is permitted.</constraint>
<constraint>Interactive prompts must cover feasible srch flags and filters.</constraint>
</constraints>

<domain_notes>
<note>Three draft approaches were provided and must be merged into a single, reusable two-script design.</note>
<note>Reducing heredocs is required to improve versioning and reuse of the per-file mover.</note>
<note>Logging must capture original paths, normalized paths, renames, and errors.</note>
</domain_notes>
</context>

<variables>
<variable name="[[task_request]]" required="true">

@cubic-dev-ai cubic-dev-ai Bot Mar 22, 2026 •
edited
Loading

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The README contains unresolved template placeholders ([[...]]) and prompt-structure tags, which makes the documentation non-actionable for users.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At README.md, line 35:

<comment>The README contains unresolved template placeholders (`[[...]]`) and prompt-structure tags, which makes the documentation non-actionable for users.</comment>

<file context>
@@ -1 +1,137 @@
+  </context>
+
+  <variables>
+    <variable name="[[task_request]]" required="true">
+      <description>Refactor the srch-based collection workflow into a builder and a reusable mover with consistent logging and collision handling.</description>
+    </variable>
</file context>
Fix with Cubic

<description>Refactor the srch-based collection workflow into a builder and a reusable mover with consistent logging and collision handling.</description>
</variable>
<variable name="[[builder_path]]" required="true">
<description>/usr/local/bin/srch_flat_collect</description>
</variable>
<variable name="[[mover_path]]" required="true">
<description>/usr/local/libexec/srch_move_one</description>
</variable>
<variable name="[[destination_folder]]" required="true">
<description>Single-level folder under /home/linaro used as the collection target.</description>
</variable>
</variables>

<instructions>
<instruction>1. Restate the goal: split the workflow into a builder script and a reusable mover while keeping flat moves, collision-safe naming, and audit logging.</instruction>

<instruction>2. Implement /usr/local/bin/srch_flat_collect as an interactive srch option builder that:
(a) collects srch start points;
(b) prompts for name matching (-N/-n/-i with optional negation);
(c) prompts for path matching (-a), exclusions (-e/-E/-Z), depth (-m), and filesystem boundary (-x);
(d) prompts for size and age filters (-s, -o/-O/-P, -y/-Y/-W);
(e) prompts for ownership filters (-u/-U/-g/-G);
(f) prompts for performance options (-t/-I/-q/-Q/-X) and progress/stats (-v/-S/-T/-C);
(g) always enforces -f for regular files.
</instruction>

<instruction>3. Implement /usr/local/libexec/srch_move_one as a standalone mover that:
(a) normalizes paths to absolute form;
(b) allocates collision-free names using numeric suffixes;
(c) performs copy+delete moves into the destination;
(d) logs every action to a .txt file in the destination.
</instruction>

<instruction>4. Ensure the builder exports DESTDIR, LOGFILE, LOCKFILE/LOCKDIR, and DRYRUN so the mover can operate consistently across runs.</instruction>

<instruction>5. Preserve auditability by rotating existing logs and recording the full srch invocation context.</instruction>
</instructions>

<output_format_specification>
<format>Plain text</format>
<requirements>
<requirement>Provide both scripts as versionable files rather than large heredocs embedded in one script.</requirement>
<requirement>Maintain formal, impersonal language and the structure of this template.</requirement>
<requirement>Include a complete example of each script.</requirement>
</requirements>
</output_format_specification>

<examples>
<example>
<input_data>
<task_request>Provide the builder and mover scripts as two separate files for reuse and testing.</task_request>
</input_data>
<output>
Builder: /usr/local/bin/srch_flat_collect
Mover: /usr/local/libexec/srch_move_one

The builder prompts for srch options, sets DESTDIR/LOGFILE/LOCKFILE/DRYRUN, and invokes:
srch <args> -r /usr/local/libexec/srch_move_one <roots>

The mover validates each match, resolves collisions with suffixes, copies to the destination,
deletes the source, and writes audit lines to operation_log.txt.
</output>
</example>
</examples>

<self_check>
<checklist>
<item>The builder and mover are separated into /usr/local/bin and /usr/local/libexec.</item>
<item>srch -r invokes the mover without embedding a large heredoc in the builder.</item>
<item>Collision handling uses deterministic numeric suffixes before extensions.</item>
<item>Moves use copy+delete semantics and are logged in operation_log.txt.</item>
<item>Destination ownership and permissions are set for linaro with a+rwX.</item>
</checklist>
</self_check>

<evaluation_notes>
<test_cases>
<case>Multiple identical basenames across different source paths.</case>
<case>Parallel srch -r runs with lock-guarded name allocation.</case>
<case>Files disappearing between traversal and move.</case>
<case>Paths containing spaces or special characters.</case>
</test_cases>
<success_definition>All matched files land in the flat destination with correct naming, permissions, and complete audit logs.</success_definition>
</evaluation_notes>

<documentation>
<usage>
<step>Install /usr/local/bin/srch_flat_collect and /usr/local/libexec/srch_move_one, then run the builder with sudo.</step>
<step>Follow prompts to build the srch command, confirm the operation, and review operation_log.txt.</step>
</usage>
<known_limitations>
<limitation>Large traversals can be I/O intensive; filters should be tuned before execution.</limitation>
<limitation>Locking uses flock when available and a mkdir fallback otherwise; behavior may vary on some filesystems.</limitation>
</known_limitations>
</documentation>

<observations>
<item>Drafts differed on whether to allow existing destinations; this version prompts for confirmation if the destination exists.</item>
<item>Drafts used both three- and four-digit suffixes; this version standardizes on three-digit suffixes for collisions.</item>
<item>Drafts varied between embedded mover heredocs and external helpers; this version externalizes the mover for reuse.</item>
</observations>
]]
Comment on lines +1 to +137

@gemini-code-assist gemini-code-assist Bot Mar 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The document uses a custom format with [[...]] and XML-like tags. This format is not standard Markdown and will not render correctly in most Markdown viewers (like on GitHub). Consider using standard Markdown syntax (e.g., headers, lists, bold text) to improve readability and maintainability.

253 changes: 253 additions & 0 deletions usr/local/bin/srch_flat_collect
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,253 @@
#!/usr/bin/env bash
# srch flat-file collector: interactive builder + srch invocation.

set -u
set -o pipefail

if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then
echo "ERROR: must run as root (sudo)." >&2
exit 1
fi

if ! command -v srch >/dev/null 2>&1; then
echo "ERROR: srch not found in PATH." >&2
exit 1
fi

if ! command -v fold >/dev/null 2>&1; then
echo "ERROR: fold not found in PATH." >&2
exit 1
fi

umask 000

ts() { date -Is; }
fold_line() { fold -w 1200 -s; }

prompt() {
local label="$1" def="${2-}" ans
if [[ -n "$def" ]]; then
read -r -p "$label [$def]: " ans
echo "${ans:-$def}"
else
read -r -p "$label: " ans
echo "$ans"
fi
}

prompt_yesno() {
local q="$1" def="${2:-n}" ans
while :; do
read -r -p "$q (y/n) [$def]: " ans
ans="${ans:-$def}"
case "${ans,,}" in
y|yes) echo "y"; return 0 ;;
n|no) echo "n"; return 0 ;;
*) echo "Please answer y or n." >&2 ;;
esac
done
}

sanitize_folder_name() {
local s="$1"
s="${s//\//_}"
s="${s//\\/_}"
s="${s//\0/}"
printf '%s' "$s"
}

DEST_BASE="/home/linaro"
DEFAULT_DEST_NAME="srch_flat_$(date +%Y%m%d_%H%M%S)"
DEST_NAME_RAW="$(prompt "Destination folder name under ${DEST_BASE} (single level; no slashes)" "$DEFAULT_DEST_NAME")"
DEST_NAME="$(sanitize_folder_name "$DEST_NAME_RAW")"

if [[ -z "$DEST_NAME" ]]; then
echo "ERROR: destination folder name must not be empty." >&2
exit 1
fi

DESTDIR="${DEST_BASE}/${DEST_NAME}"
LOGFILE="${DESTDIR}/operation_log.txt"
LOCKFILE="${DESTDIR}/.rename.lock"
LOCKDIR="${DESTDIR}/.rename.lockdir"
HELPER="/usr/local/libexec/srch_move_one"

if [[ ! -x "$HELPER" ]]; then
echo "ERROR: helper not found or not executable: $HELPER" >&2
exit 1
fi

if [[ -e "$DESTDIR" ]]; then
use_existing="$(prompt_yesno "Destination exists. Use it anyway?" "n")"
if [[ "$use_existing" != "y" ]]; then
echo "ERROR: destination already exists: $DESTDIR" >&2
exit 1
fi
fi

mkdir -p -- "$DESTDIR" || { echo "ERROR: cannot create destination: $DESTDIR" >&2; exit 1; }
chmod 0777 -- "$DESTDIR" 2>/dev/null || true

if [[ -e "$LOGFILE" ]]; then
mv -f -- "$LOGFILE" "${LOGFILE}.bak.$(date +%Y%m%d%H%M%S)" 2>/dev/null || true
fi
: >"$LOGFILE" || { echo "ERROR: cannot write log: $LOGFILE" >&2; exit 1; }
chmod 0666 -- "$LOGFILE" 2>/dev/null || true

log() {
printf '%s %s\n' "$(ts)" "$*" | fold_line | tee -a "$LOGFILE" >&2
}

log "BEGIN collection"
log "effective_uid=$(id -u) effective_user=$(id -un 2>/dev/null || true)"
log "destination=$DESTDIR"
log "logfile=$LOGFILE"

SRCH_ARGS=()
SRCH_ROOTS=()

roots_raw="$(prompt "srch start directories (space-separated)" "/home/linaro /mnt /userdata /usr/share/doc")"
# shellcheck disable=SC2206
SRCH_ROOTS=( $roots_raw )
Comment on lines +110 to +111

@gemini-code-assist gemini-code-assist Bot Mar 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Unquoted expansion of $roots_raw will cause issues with paths that contain spaces, even if they are quoted in the input. The shellcheck disable=SC2206 indicates awareness, but this is a bug that should be fixed. Use read -r -a with a here-string to correctly parse space-separated arguments into an array, which respects quotes.

read -r -a SRCH_ROOTS <<< "$roots_raw"


threads="$(prompt "Threads (-t): number, or '*' for all cores, or blank for default" "")"
[[ -n "$threads" ]] && SRCH_ARGS+=("-t" "$threads")

name_mode="$(prompt "Name matching mode: none | -N (substring) | -n (regex) | -i (regex, case-insensitive)" "-n")"
name_mode="${name_mode// /}"
if [[ "$name_mode" != "none" && -n "$name_mode" ]]; then
name_neg="$(prompt_yesno "Negate the name match? (prefix '!' to the pattern)" "n")"
name_pat="$(prompt "Pattern for ${name_mode} (empty disables name filter)" "")"
if [[ -n "$name_pat" ]]; then
[[ "$name_neg" == "y" ]] && name_pat="!$name_pat"
case "$name_mode" in
-N|-n|-i) SRCH_ARGS+=("$name_mode" "$name_pat") ;;
*) log "WARN: Unrecognized name mode '$name_mode'; skipping name filter." ;;
esac
fi
fi

use_full="$(prompt_yesno "Apply pattern to full path (-a)?" "n")"
[[ "$use_full" == "y" ]] && SRCH_ARGS+=("-a")

add_e="$(prompt_yesno "Add directory exclude regex rules (-e)?" "n")"
if [[ "$add_e" == "y" ]]; then
while :; do
ex="$(prompt "-e <dir-regex> (blank to stop)" "")"
[[ -z "$ex" ]] && break
SRCH_ARGS+=("-e" "$ex")
done
fi

add_E="$(prompt_yesno "Add exact directory excludes (-E)?" "n")"
if [[ "$add_E" == "y" ]]; then
while :; do
ex2="$(prompt "-E <dir-name> (blank to stop)" "")"
[[ -z "$ex2" ]] && break
SRCH_ARGS+=("-E" "$ex2")
done
fi

use_Z="$(prompt_yesno "Exclude .snapshot directories (-Z)?" "y")"
[[ "$use_Z" == "y" ]] && SRCH_ARGS+=("-Z")

depth="$(prompt "Depth control (-m): e.g., '4', '2-5', '2-' (blank for unlimited)" "")"
[[ -n "$depth" ]] && SRCH_ARGS+=("-m" "$depth")

xdev="$(prompt_yesno "Stay on same filesystem for each start point (-x)?" "n")"
[[ "$xdev" == "y" ]] && SRCH_ARGS+=("-x")

ftype="$(prompt "Type filter (for completeness): f/d/l/b/c/p/k/none" "f")"
if [[ "${ftype,,}" != "f" ]]; then
log "WARN: Operation moves regular files only; forcing '-f' regardless of selection ('$ftype')."
fi
SRCH_ARGS+=("-f")

size_spec="$(prompt "Size constraint (-s): e.g., +1k, -10m, +1k:-10m (blank for none)" "")"
[[ -n "$size_spec" ]] && SRCH_ARGS+=("-s" "$size_spec")

older_days="$(prompt "Older than days (-o) (blank for none)" "")"
[[ -n "$older_days" ]] && SRCH_ARGS+=("-o" "$older_days")

older_mins="$(prompt "Older than minutes (-O) (blank for none)" "")"
[[ -n "$older_mins" ]] && SRCH_ARGS+=("-O" "$older_mins")

older_tstamp="$(prompt "Older than timestamp file (-P <file>) (blank for none)" "")"
[[ -n "$older_tstamp" ]] && SRCH_ARGS+=("-P" "$older_tstamp")

younger_days="$(prompt "Younger than days (-y) (blank for none)" "")"
[[ -n "$younger_days" ]] && SRCH_ARGS+=("-y" "$younger_days")

younger_mins="$(prompt "Younger than minutes (-Y) (blank for none)" "")"
[[ -n "$younger_mins" ]] && SRCH_ARGS+=("-Y" "$younger_mins")

younger_tstamp="$(prompt "Younger than timestamp file (-W <file>) (blank for none)" "")"
[[ -n "$younger_tstamp" ]] && SRCH_ARGS+=("-W" "$younger_tstamp")

inc_user="$(prompt "Owned by user (-u) (blank for none)" "")"
[[ -n "$inc_user" ]] && SRCH_ARGS+=("-u" "$inc_user")

exc_user="$(prompt "Not owned by user (-U) (blank for none)" "")"
[[ -n "$exc_user" ]] && SRCH_ARGS+=("-U" "$exc_user")

inc_group="$(prompt "Owned by group (-g) (blank for none)" "")"
[[ -n "$inc_group" ]] && SRCH_ARGS+=("-g" "$inc_group")

exc_group="$(prompt "Not owned by group (-G) (blank for none)" "")"
[[ -n "$exc_group" ]] && SRCH_ARGS+=("-G" "$exc_group")

inline_dirs="$(prompt "Inline subdir threshold (-I <count>) (blank for none)" "")"
[[ -n "$inline_dirs" ]] && SRCH_ARGS+=("-I" "$inline_dirs")

use_q="$(prompt_yesno "Use FIFO queue (-q)?" "n")"; [[ "$use_q" == "y" ]] && SRCH_ARGS+=("-q")
use_Q="$(prompt_yesno "Queue sorted by inode (-Q)?" "n")"; [[ "$use_Q" == "y" ]] && SRCH_ARGS+=("-Q")
use_X="$(prompt_yesno "Enable extremely-large-dir optimization (-X)?" "n")"; [[ "$use_X" == "y" ]] && SRCH_ARGS+=("-X")

prog_v="$(prompt "Progress interval (-v <N>) (blank for none)" "")"
[[ -n "$prog_v" ]] && SRCH_ARGS+=("-v" "$prog_v")

use_S="$(prompt_yesno "Print summary stats (-S)?" "y")"; [[ "$use_S" == "y" ]] && SRCH_ARGS+=("-S")
use_T="$(prompt_yesno "Print elapsed time (-T)?" "y")"; [[ "$use_T" == "y" ]] && SRCH_ARGS+=("-T")
use_C="$(prompt_yesno "Suppress missing/disappearing entry errors (-C)?" "y")"; [[ "$use_C" == "y" ]] && SRCH_ARGS+=("-C")

dryrun="$(prompt_yesno "Dry-run (log only; do not copy/delete)?" "y")"

extra_raw="$(prompt "Additional srch arguments (space-separated; blank for none)" "")"
if [[ -n "$extra_raw" ]]; then
# shellcheck disable=SC2206
extra_arr=( $extra_raw )
Comment on lines +217 to +218

@gemini-code-assist gemini-code-assist Bot Mar 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Unquoted expansion of $extra_raw will cause issues with arguments that contain spaces, even if they are quoted in the input. The shellcheck disable=SC2206 indicates awareness, but this is a bug that should be fixed. Use read -r -a with a here-string to correctly parse space-separated arguments into an array.

  read -r -a extra_arr <<< "$extra_raw"

for a in "${extra_arr[@]}"; do SRCH_ARGS+=("$a"); done
fi

log "srch_roots=${SRCH_ROOTS[*]}"
log "srch_args=${SRCH_ARGS[*]}"
log "dryrun=$dryrun"

confirm="$(prompt_yesno "Proceed now? (Matches will be processed by srch -r)" "n")"
if [[ "$confirm" != "y" ]]; then
log "ABORTED by user"
exit 0
fi

export DESTDIR LOGFILE LOCKFILE LOCKDIR
export DRYRUN="$dryrun"

set +e
srch "${SRCH_ARGS[@]}" -r "$HELPER" "${SRCH_ROOTS[@]}" 2>&1 | fold -w 1500 -s | tee -a "$LOGFILE"
rc=${PIPESTATUS[0]}
set -e 2>/dev/null || true

log "srch_exit_code=$rc"

chown -R linaro:linaro -- "$DESTDIR" 2>/dev/null || true
chmod -R a+rwX -- "$DESTDIR" 2>/dev/null || true

moved_count="$(grep -c " MOVED " "$LOGFILE" 2>/dev/null || true)"
skip_count="$(grep -c " SKIP " "$LOGFILE" 2>/dev/null || true)"
dry_count="$(grep -c " DRYRUN " "$LOGFILE" 2>/dev/null || true)"
err_count="$(grep -c " ERROR " "$LOGFILE" 2>/dev/null || true)"

log "SUMMARY moved=$moved_count skipped=$skip_count dryrun=$dry_count errors=$err_count"
log "DONE destination=$DESTDIR"

exit 0
Loading