The Script That Dies Quietly: `set -e`, Shell Arithmetic, and the Contracts Your Automation Owes You

Infrastructure estates are full of small shell scripts: glue between systems, aggregation jobs, smoke checks. They are usually the least-reviewed code in the estate, and their failures are easy to miss because a partial run can look exactly like a full one. This article walks one specific failure mode from an internal engineering review — a fifteen-line counting script that aborts mid-loop under set -e with no error message — and draws out the decision path for a CTO, head of infrastructure, or security lead. The one-line fix matters less than the class of bug it represents and the contracts the script was violating.

A small script, a silent failure

The script under review counts occurrences of the four nucleotides (A, C, G, T) in a string, rejects anything else, and prints a tally. It starts with set -euo pipefail, keeps counts in an associative array, and loops character by character:

for (( i=0; i<${#input}; i++ )); do
                char="${input:$i:1}"
                (( counts["$char"]++ ))
            done
            

Run against real input, it dies after its first increment. No message, no traceback — the bash -x trace shows the loop reaching the increment and stopping. The mechanism is a classic shell footgun:

• (( expression )) is a command, and its exit status is 1 when the expression evaluates to zero.
• Post-increment (x++) evaluates to the old value of x.
• The first time a character is seen, its count is 0. The expression evaluates to 0, so the command exits 1.
• With set -e (subject to its usual exemptions for conditionals and || lists), the shell exits.

The script's first piece of real work kills it. That is the "dies quietly" class of bug: not a crash with a traceback, not a loud validation error — the script simply stops, and downstream stages cannot tell a partial run from a complete one.

Illustrative scenario (fictional — not a Validus/Veltiosi client engagement). A platform team notices a small aggregation job in the nightly pipeline failing intermittently. Run by hand, it prints a few lines and exits; in CI it stops after the first line with no message. bash -x shows the loop incrementing a counter and the script terminating on the first increment. The root cause is the arithmetic command above — not the data, the network, or the scheduler.

Diagnose before you touch anything

Establish what the script actually does before changing a line. All of this is read-only:

1. Trace it. bash -x script.sh "GATTACA" prints every command before it runs. The last command in the trace is where the script stopped; the arithmetic command's exit status is the reason.
2. Isolate the suspect line. Run the arithmetic in a throwaway subshell: bash -c 'set -e; x=0; (( x++ )); echo survived' prints nothing — the script died exactly there. Run bash -c 'set -e; x=0; (( x += 1 )); echo survived' and it prints survived. The difference between those two lines is the whole bug.
3. Confirm the environment, not just the laptop. Bash behavior varies by version: ${var^^} requires Bash 4+, and set -u interacts with unset associative-array keys differently across versions. Run bash --version on the hosts that actually execute the script.

The diagnostic principle: separate "the script failed", "the script produced wrong output", and "the script stopped without saying why". An exit code answers the first; bash -x answers the third. Never theorize about a shell failure you haven't traced.

The decision path: trade-offs, not magic

Four defensible fixes, with different trade-offs:

1. (( counts["$char"] += 1 )) — the expression now evaluates to the new count, which cannot be 0 here, so the command exits 0. Minimal diff, but it fixes this line, not the class: any arithmetic expression that can evaluate to zero will still kill the script.
2. Assignment form — counts["$char"]=$(( counts["$char"] + 1 )) — the assignment succeeds regardless of the computed value, so a value of 0 can never abort the script. This decouples arithmetic from exit-status checking, which is the contract you actually want. Slightly more verbose, slightly more correct.
3. (( ... )) || true — explicitly neutralizes the exit status where you genuinely don't care about it. Self-documenting, but easy to overuse into masking real failures; use it sparingly and comment why.
4. Restructure. For a fixed key domain, plain counters or a case statement avoid associative arrays entirely: no iteration-order questions, fewer Bash-version surprises. Associative arrays earn their keep only when keys are dynamic.

Decision rule: in critical paths, choose the fix that removes the failure class (option 2 or 4), not the one that silences the symptom (option 1, sometimes 3). And validate input once, up front — uppercase the whole string with a single expansion and reject early — rather than re-validating every character mid-loop.

Changing a script that runs in production is a deployment, not an edit. Branch it, run the existing test suite, diff before/after output on representative inputs, then roll out. Rollback is a revert — which only exists if the script is in version control and the change is small.

Output and exit codes are contracts

The review surfaced three contract violations that matter more than the arithmetic bug:

• Output shape. The tests expect all four keys in fixed order — A, C, G, T — including zeros, even for empty input. Associative-array iteration order is unspecified, so "${!counts[@]}" is the wrong tool for output. Iterate a fixed key list: for key in A C G T; do printf '%s: %s\n' "$key" "${counts[$key]:-0}"; done. The :-0 default also stops set -u from dying on an absent key.
• Initialization. Zero-initialize counts before scanning so empty input yields the full contract instead of an empty line. Empty input is a case, not an exception.
• Error contract. The tests expect the exact message "Invalid nucleotide in strand" and a non-zero exit. The original printed error and returned 0 — telling the orchestrator the run succeeded. In automation, the exit status is the machine-readable contract and the message is for humans; both are part of the interface and should change with the same ceremony as an API. Inside a function, return sets that function's status, so make sure main ends by calling the function and the script's overall status propagates. And ${1-} — with the dash — keeps set -u from aborting before the script does anything when the argument is missing.

For a security lead: if a script handles credentials, source them from the environment or a secret manager — never embed them in the script or on a command line. A script that fails loudly is easier to audit than one that fails silently: the exit code and the trace are the audit trail.

Operator checklist: hardening shell automation

• [ ] Reproduce with bash -x on the exact failing input and keep the trace.
• [ ] Audit every (( ... )) for expressions that can evaluate to 0 under set -e.
• [ ] Decide explicitly whether set -e is the behavior you want; it guarantees some exit, not a correct one. Pair it with an ERR/EXIT trap that logs the failing line so failures leave a record.
• [ ] Treat exit codes and exact output as a versioned contract; tests assert success/failure and the exact message.
• [ ] Never rely on associative-array iteration order for output; iterate a fixed key list.
• [ ] Initialize all state before processing; test the empty-input case explicitly.
• [ ] Check bash --version on every host class that runs the script, not just your laptop; document the minimum supported version.
• [ ] Validate input once, up front; reject early with a distinct exit code and a message that names the failure.
• [ ] Keep scripts in version control; ship changes to production pipelines as branches with test runs and output diffs; keep rollback to a revert.
• [ ] Make silence impossible: every error path returns a distinct non-zero status and says why.