The off-by-one transfer: when your batch manifest says one thing and your files say another

Date-driven file selection is the quiet workhorse of operational data movement. It is also where pipelines fail silently: no crash, no alert, just a batch manifest that contains the wrong files. The session below is an illustrative operator exchange, adapted from the source material; it is not a Validus client engagement. It shows a Bash loop that was supposed to collect files "created from Feb 20 to Mar 5" for an SFTP push — and the two separate defects hiding in twelve lines of script.

The failure was not only in the syntax

The loop used while [[ "$current" <= "$end" ]]. In Bash, the [[ ]] compound command supports string comparisons <, >, ==, != — but not <= or >=. The script would error or misbehave, and because the failure happens before any transfer, the operator may not notice until the batch count is off. Two fixes are standard:

• Loop to an exclusive upper bound: compute end_exclusive="$(date -d "$end + 1 day" +%Y-%m-%d)" and test while [[ "$current" < "$end_exclusive" ]]. Zero-padded ISO-8601 dates sort lexicographically in chronological order, so < on YYYY-MM-DD strings is correct.
• Or compare epoch seconds (date -d "$d" +%s) with -le.

The second defect is more interesting and more common. The files created on Feb 20 embed 2026-02-19 in their names; the file created on Mar 5 embeds 2026-03-04. The naming convention carries the business date of the data, which trails the creation timestamp by one day — the generation timestamps in the names (gen20260220070534) suggest a producer that builds each day's report early the next morning. The operator said "capture files created from Feb 20 to Mar 5" but the loop iterated embedded dates from Feb 21 to Mar 4 — a predicate mismatch, not a typo. The script did not fail loudly; it produced a batch. That silent divergence is the resilience problem worth taking seriously: the set you selected and the set you intended diverged, and only a count check (should be 36) stood between you and an incomplete transfer.

Decision path: match the filename, or trust the filesystem?

There are two legitimate predicates, and they answer different questions.

Option A — match the embedded date in the filename. Loop over the embedded date range (2026-02-19 to 2026-03-04, for files created Feb 20 to Mar 5) and glob Scb_*_"$current"_*.xlsx. This selects on the business date of the data, which is usually what the downstream consumer cares about. Trade-offs: it is coupled to the naming convention (a rename or a format change silently breaks the match), the loop must itself be correct (the original <= bug lived here), and a missing file produces no match rather than an error unless you guard with [[ -f "$f" ]] or nullglob.

Option B — select by filesystem metadata. Use find "$DIR" -maxdepth 1 -type f -name 'Scb_*.xlsx' -newermt "$start" ! -newermt "$end_exclusive". This literally implements "files created between Feb 20 and Mar 5", independent of naming conventions. Trade-offs: -newermt is a GNU find extension (date -d is similarly GNU-only), so the script is pinned to a GNU userland — on macOS or BSD you need date -j -f and a reference-file approach; mtime reflects last modification, and copies, restores, or archive extraction can shift it; and it answers "when was it written", not "what business date does it hold".

The decision rule: pick the predicate that matches the consumer's contract. If downstream systems expect reports for business date D, select on the embedded date. If you are answering an audit question about when files were produced, select on metadata. Write the choice into the runbook, because the two will drift apart on every boundary — month rollover, leap days, DST, and timezone.

The manifest is the audit artifact

The batch file is not a scratch list. It is the exact enumeration of what was transferred — which makes it an audit record. Treat it as one:

• Build it with mktemp (0600 permissions by default) and truncate it deliberately with : > "$BATCH" so a stale manifest can never be reused by accident.
• Write entries with printf 'put %q\n' "$f" so quoting survives edge cases.
• Count with wc -l < "$BATCH" — but remember that a line count validates volume, not content. A stronger check is to diff the manifest against an independently generated expected list before the transfer runs.
• Name and retain the manifest with the job identifier and date, and hash it. When the "what did we actually send on Mar 5" question arrives, the answer is one file, not a grep across shell history.

Two identity and security points belong here. First, an sftp -b batch file runs non-interactively, so anything that needs a prompt — a password, a host-key confirmation — stalls or fails the run. That is why credentials must never live in the manifest or on the command line; non-interactive authentication (SSH keys, an agent, a dedicated transfer identity) is the prerequisite. Second, mktemp's 0600 mode is a default, not a guarantee: do not append secrets to the manifest, and give the transfer identity exactly the permissions the job needs and nothing more.

Hardening for operational resilience

• Pin the timezone. date -d arithmetic uses the local timezone. If the host's TZ or DST rules change, "the same" commands can select a different set. Set TZ=UTC (or a documented fixed zone) inside the script so date math is reproducible.
• Choose the toolchain deliberately. GNU date -d and find -newermt are not portable to BSD/macOS. Either standardize on a GNU userland (and say so in the runbook) or use portable equivalents. The useful side of this for open-source independence: the entire stack here — Bash, GNU coreutils, find, OpenSSH sftp — is standard open-source tooling with no proprietary dependency, so the choice is yours to make and audit, not a licensing decision.
• Fail loudly on the boundaries. set -euo pipefail turns unset variables and failed pipes into errors. Assert the expected set, not just the count, and alert on mismatch.
• Diagnose read-only first. Before anything writes: ls -la to see what actually exists, stat a sample file, run the manifest build as a dry run, and verify the target host and transfer identity are reachable. The transfer itself is the write action — treat it as such. Prerequisites: verified manifest, reachable target, key-based auth confirmed, source files retained. Rollback: the script never deletes sources, the manifest is retained so the transferred set is known exactly, and reconciliation happens on the target against the manifest hash.

Operator checklist

1. State the predicate in one sentence — "created between" (metadata) or "business date of the data" (filename) — and write it in the runbook.
2. Loop to an exclusive upper bound (end + 1 day); never use <= inside [[ ]].
3. Verify date arithmetic across the actual boundaries (month rollover, leap day, DST) before relying on it.
4. Build the manifest read-only; count it, then diff it against an expected list. Do not run the transfer from the same command.
5. Keep credentials out of scripts and manifests; confirm key-based, non-interactive auth works before the transfer.
6. Pin TZ, retain the manifest with job id and date, hash it, and reconcile on the target after the run.