Selection and markers

selection

Turn model spans into a keep/drop plan over lines, then render it.

The model scores characters; agents read lines. This module is the bridge, and it is deliberately biased toward keeping: dropping a line the agent needed forces it to re-run a command (or fetch the marker), which costs far more than the few tokens a spare line costs. So, whatever the model says:

• lines that carry failure signal (errors, tracebacks, exit codes, pytest E assertion lines) are always kept, plus the body of every Python traceback;
• every kept line pulls in context neighbours on each side;
• the first and last edge lines are kept (command banner, summary line).

Rendering preserves the original bytes of every kept line — the split is on "\n" only, line endings included — so kept content is byte-identical to the input. Each maximal dropped run becomes one marker line; a run whose text is not longer than its marker is kept instead, since replacing it would grow the output.

PrunePlan(content, recoverable=dict(), dropped_lines=0) dataclass

Result of :func:render.

Attributes:

Name	Type	Description
content	str	The pruned text (kept lines verbatim, markers for dropped runs).
recoverable	dict[str, str]	hash -> dropped text for every marker in content.
dropped_lines	int	Number of input lines replaced by markers.

split_lines(content)

Split content on "\n" keeping line endings.

"".join(split_lines(content)) == content always holds. Unlike str.splitlines this never splits on \r, form feeds or other Unicode separators, so character offsets from the model map onto lines exactly.

Parameters:

Name	Type	Description	Default
content	str	Raw tool output.	required

Returns:

Type	Description
list[str]	The lines, each ending in "\n" except possibly the last.

lines_touched(spans, lines)

Return indices of lines that overlap any [start, end) span.

Parameters:

Name	Type	Description	Default
spans	Iterable[tuple[int, int]]	Character spans over "".join(lines).	required
lines	list[str]	Output of :func:split_lines.	required

Returns:

Type	Description
set[int]	The set of touched line indices.

mandatory_lines(lines)

Return indices of lines that must survive regardless of the model.

A line is mandatory if it matches :data:MANDATORY_RE or looks like a pytest assertion detail. A Python traceback is kept whole: the header, its indented frame/source lines, and the first unindented line after them (the exception message).

Parameters:

Name	Type	Description	Default
lines	list[str]	Output of :func:split_lines.	required

Returns:

Type	Description
set[int]	The set of mandatory line indices.

expand_keep(keep, n_lines, *, context, edge)

Grow keep by context neighbours and the first/last edge lines.

Parameters:

Name	Type	Description	Default
keep	set[int]	Line indices already kept.	required
n_lines	int	Total number of lines.	required
context	int	Neighbours kept on each side of every kept line.	required
edge	int	Lines always kept at the start and at the end.	required

Returns:

Type	Description
set[int]	A new set; keep is not modified.

render(lines, keep)

Render kept lines verbatim and replace each dropped run with a marker.

Parameters:

Name	Type	Description	Default
lines	list[str]	Output of :func:split_lines.	required
keep	set[int]	Line indices to keep.	required

Returns:

Name	Type	Description
The	PrunePlan	class:PrunePlan. Output is a pure function of the inputs.

markers

CCR retrieval markers and content hashes for dropped line runs.

Every run of lines the pruner drops is replaced by exactly one marker line and its full text goes into CompressOutput.recoverable. The router persists that map into the CCR store with explicit_hash and the agent's retrieval tool resolves the marker back to the original bytes, so a wrong drop costs one retrieval round trip instead of lost information.

Two constraints come from Headroom, not from us:

• The store only accepts lowercase hex hashes, and the agent-side marker regex (headroom/ccr/marker_resolution.py) is <<ccr:([a-f0-9]{12,24})[^>]*>>. A hash outside that shape is silently unretrievable.
• SmartCrusher's row-offload marker is <<ccr:HASH N_rows_offloaded>>; we mirror it with N_lines_offloaded so agents see one familiar format.

The hash is sha256(text)[:24] — the same rule Headroom uses — so identical input always yields byte-identical output, which keeps the provider's prompt cache warm across turns.

content_hash(text)

Return the CCR hash for text.

Parameters:

Name	Type	Description	Default
text	str	The exact dropped text (line endings included).	required

Returns:

Type	Description
str	The first :data:HASH_LENGTH lowercase hex characters of its sha256.

format_marker(ccr_hash, n_lines)

Return the single-line marker that stands in for a dropped run.

Parameters:

Name	Type	Description	Default
ccr_hash	str	Hash from :func:content_hash for the dropped text.	required
n_lines	int	Number of lines the marker replaces.	required

Returns:

Type	Description
str	<<ccr:HASH N_lines_offloaded>> without a trailing newline.