The DSL filter engine¶

How it works¶

components/dsl is a declarative, user-extensible text-filter engine (adapted from rtk — see THIRD-PARTY-NOTICES), wrapped by cmdfilter. Filters are authored in YAML (no recompile), matched by descending priority then by name, and each runs a fixed 8-stage pipeline. Because filters drop lines they are lossy, which is why the wrapping cmdfilter component is an Offload (it stashes the original first).

flowchart LR
  I[input] --> S1[1 strip_ansi] --> S2["2 replace[]"] --> S3["3 match_output[] + unless"]
  S3 --> S4[4 strip / keep lines] --> S5[5 truncate_lines_at] --> S6[6 head / tail]
  S6 --> S7[7 max_lines] --> S8[8 on_empty] --> O[output + Lossiness]

Filter fields¶

All optional except match:

• match — regex vs the selector (= the first few non-empty lines, (?m) applied so ^/$ anchor per line). When the request pairs the output with the call that produced it, the selector is PREFIXED with a $ <command> line, so a filter can be keyed on the command instead — match: '^\$ (rg|grep)\s' — the way rtk's are. Shape signatures remain the default because they fire on unpaired traffic too.
• family — command family for per-family metrics (builds / tests / iac / pkg / net / …)
• priority — match order; higher first, then by name. Absent (0) is today's name ordering.
• strip_ansi
• replace — chained pattern→replacement, $1 backrefs
• match_output — whole-blob short-circuit: pattern/message/unless
• strip_lines_matching xor keep_lines_matching
• truncate_lines_at — per-line char cap
• head_lines / tail_lines
• cap / cap_reduce — a shared line budget (see below); an explicit max_lines wins
• max_lines — absolute cap with omission marker
• on_empty — replacement when output is blank

priority, and why order matters more here¶

cmdfilter matches on the shape of the output (a command line is available too, but every shipped filter is shape-keyed) and against several leading lines rather than one — so a generic pattern can shadow a specific one in a way rtk's command matching never had to deal with. This is not hypothetical: widening the selector made gcc start claiming make, swift-build and dotnet-build output, because a bare file:line: error: line appears inside all of them. priority makes specific-before-generic explicit instead of dependent on alphabetical luck. Absent, ordering is by name — exactly the previous behavior.

Rule of thumb: a filter whose selector is a tool banner can be high priority; one whose selector is a generic diagnostic shape must be low, so it only catches what nothing else claimed.

Line budgets: cap classes¶

Instead of every filter hand-picking a max_lines, a filter selects a budget by signal density. The class names and the first four values are rtk's (src/core/truncate.rs); buildlog is ours.

cap_reduce: N lowers the chosen cap for an extra-verbose variant, underflow-safe (a deviation can never empty the budget — rtk's reduced helper). Unknown cap names, and a cap_reduce without a cap, are rejected at load. rtk's own TOML filters don't use its cap classes; applying them to the definitions makes the whole set tunable from one map.

Lossiness¶

Lossiness is reported back to cmdfilter, which uses it to pick the recovery hint:

• None — nothing dropped / reversible reformat → no hint
• Tail — a clean contiguous tail dropped → the hint names the cut point, because re-reading from there is cheaper than pulling the whole blob back
• Whole — non-contiguous or whole-blob loss → the hint points at the expand tool

An intra-line cut from truncate_lines_at types as Whole, since every long line loses its own tail and the loss is non-contiguous by nature. It also appends ..., sized to fit inside the cap so the line never grows — a mid-line cut with no marker reads as corrupted output to a model.

Load-time guardrails¶

Mirroring rtk's build.rs, a filter document fails at load, not at first use, when:

• two filters share a name (a silently shadowed filter is a debugging trap);
• a match / replace / match_output / strip_lines_matching regex doesn't compile;
• strip_lines_matching and keep_lines_matching are both set;
• cap names an unknown class, or cap_reduce is set without cap;
• any inline test in the document fails — so a broken filter fails loudly instead of quietly mangling output.

The shipped filter set must additionally have ≥1 test per filter, and each test's input must route to its own filter (TestEveryBuiltinFilterHasTestsAndRoutes). User-supplied documents aren't required to ship tests, but any they do ship must pass.

Example¶

schema_version: 1
filters:
  pytest:
    description: keep failures + summary, drop passing noise
    family: tests
    priority: 10
    match: "(pytest|=+ test session starts)"
    strip_lines_matching: ["^\\s*$", " PASSED", "^\\.+$"]
    cap: buildlog
    on_empty: "pytest: all passed"
tests:                       # inline; run at load, and via dsl.RunTests
  pytest:
    - name: all-green
      input: "pytest\n....\n"
      expected: "pytest: all passed"

Documents load with schema_version: 1 and strict unknown-field rejection.

Write a custom filter¶

Teach cmdfilter to shrink a command output it does not recognise yet. Filters are YAML — no recompile.

Steps¶

1. Capture a real sample of the output you want to filter, exactly as the agent sees it.

2. Write the filter. match is a regex tested against the output's first six non-empty lines, so anchor it on something that actually appears near the top:

   schema_version: 1
   filters:
     pytest:
       description: keep failures + summary, drop passing noise
       match: "(pytest|=+ test session starts)"
       strip_lines_matching: ["^\\s*$", " PASSED", "^\\.+$"]
       max_lines: 80
       on_empty: "pytest: all passed"
   tests:
     pytest:
       - name: all-green
         input: "pytest\n....\n"
         expected: "pytest: all passed"
   

3. Ship the tests block with it. Inline tests run at load time, so a filter that stops doing what its tests say fails to load instead of quietly mangling output.

4. Load it into cmdfilter:

   components:
     cmdfilter:
       filters:
         - |
           schema_version: 1
           filters:
             pytest:
               match: "(pytest|=+ test session starts)"
               strip_lines_matching: ["^\\s*$", " PASSED", "^\\.+$"]
               max_lines: 80
               on_empty: "pytest: all passed"
       disable_builtins: false   # keep the 26 shipped filters too
       min_size: 400             # byte floor; below it the marker costs more than the saving
   

5. Confirm it fires. /stats reports per-family counts and cmdfilter_selector_misses — the frequency-ranked list of output shapes no filter claimed, which doubles as your backlog of filters worth writing.

Filter fields (reference)¶

All optional except match.

Stages run in this order:

flowchart LR
  I[input] --> S1[1 strip_ansi] --> S2["2 replace[]"] --> S3["3 match_output[] + unless"]
  S3 --> S4[4 strip / keep lines] --> S5[5 truncate_lines_at] --> S6[6 head / tail]
  S6 --> S7[7 max_lines] --> S8[8 on_empty] --> O[output]

Filters are lossy, which is why cmdfilter is an Offload: it stashes the original first and appends a <<cg:HASH>> recovery hint only when the filter actually dropped something. A clean contiguous tail cut names its cut point instead, since re-reading from there is cheaper than a full expand.

See also: Components overview · cmdfilter · Choose a preset