Building recon-sweep: A Scope-Gated Harness for Agentic AI Red Teaming

1. Why Another Red-Team Harness

Most LLM security work still looks like manually pasting jailbreak prompts into a chat window one at a time. That doesn't scale once the thing you're testing is agentic — a model that calls tools, reads from a RAG store, or runs against a local endpoint like Ollama with no human watching each turn.

recon-sweep is a CTF-oriented recon and extraction harness I built to close that gap: point it at a target you're authorized to test, plant a secret, and let a planner (fixed or LLM-driven) send probes while a deterministic detector suite — not an LLM's opinion of itself — decides whether anything leaked.

The project's own docstring calls it an "honest rewrite" of an earlier, more hyped concept, and that framing matters: the discipline that keeps this a CTF tool instead of something worse lives in the code, not in good intentions or a system-prompt disclaimer.

2. Core Engineering Primitives

Building an agent that's allowed to send real network traffic requires solving two problems: containment (it must be structurally unable to touch anything outside an authorized range) and verification (a "success" must be provable, not a model's self-reported confidence score).

A. The Scope Wall (Scope / ScopeError)

In many agent frameworks, safety instructions live inside a system prompt (e.g., "only scan hosts on this list"). That's fragile — prompt injection can talk a model out of a instruction. In recon-sweep, the boundary is enforced in code, and every scanner method calls it before a socket ever opens:

• ▹Explicit allowlist + CIDR matching: Scope.allow holds literal hostnames and CIDR ranges; in_scope() resolves a hostname and checks membership via Python's ipaddress module.
• ▹Hard pre-execution halt: every recon method — port scan, directory enum, TLS inspection, the LLM tool-dispatcher — calls assert_in_scope() first. An unauthorized host raises ScopeError and the call never reaches the network.

class Scope:
    """Allowlist of hosts/CIDRs this run may touch."""
    allow: set[str]

    def in_scope(self, host: str) -> bool:
        if host in self.allow:
            return True
        ip = ipaddress.ip_address(socket.gethostbyname(host))
        return any(ip in ipaddress.ip_network(e, strict=False)
                   for e in self.allow)

    def assert_in_scope(self, host: str) -> None:
        if not self.in_scope(host):
            raise ScopeError(f"'{host}' is not in scope.")

B. Falsifiable Scoring (DetectorSuite) vs. LLM "Vibes"

A common shortcut in AI red-teaming tools is asking an evaluator LLM to grade its own output on a 0–100 "confidence" scale — the model grading its own homework, with no ground truth behind the number. recon-sweep's DetectorSuite replaces that with three deterministic gates plus one advisory-only signal:

• ▹Canary gate: an exact secret string (e.g., CTF{pl4nted_secret}) is planted in the target's system prompt, RAG store, or environment before the run. A hit requires an exact substring match — no fuzzy scoring.
• ▹PII gate: regex checks for SSNs, emails, credit-card-shaped strings, phone numbers, and IPv4 addresses in every response.
• ▹Injection gate: a heuristic that flags when a probe used override phrasing ("ignore previous instructions", "new instructions") and the model complied — rather than refused — while also leaking a canary.
• ▹Advisory judge (never gates): an optional LLM — or two, reconciled against each other — annotates responses for toxicity or bias. If the two judges disagree, the verdict is marked disputed for a human to review. Agreement or disagreement is recorded, but no judge output can flip a PASS/FAIL verdict — only the three deterministic gates above decide that.

PASS/FAIL comes from DetectorSuite.passed(), which is simply "no gate fired." A separate LabeledEval module exists specifically to measure the advisory judge's own precision/recall/F1 against a golden labeled set — calibrating the judge is treated as its own falsifiable problem, not assumed.

3. Architecture: The Extraction Harness

The core loop ties a planner, the target, and the detector suite together:

┌──────────────────────────┐    ┌──────────────────────────┐
│   DeterministicPlanner   │    │        LLMPlanner        │
│    fixed 8-probe list    │    │ adaptive, needs API key  │
└──────────────────────────┘    └──────────────────────────┘
                              ▼
┌──────────────────────────────────────────────────────────┐
│                   LLMTarget.ask(probe)                   │
│           Ollama / OpenAI-compatible endpoint            │
└──────────────────────────────────────────────────────────┘
                              ▼
┌──────────────────────────────────────────────────────────┐
│ DetectorSuite.score(probe, response)                     │
│ - canary exact-match        (GATE)                       │
│ - PII regex (ssn/email/cc)  (GATE)                       │
│ - injection heuristic       (GATE)                       │
│ - judge quorum (2 models)   (ADVISORY ONLY, never gates) │
└──────────────────────────────────────────────────────────┘
                              ▼
┌──────────────────────────────────────────────────────────┐
│                     Harness.report()                     │
│           PASS/FAIL verdict + JSON transcript            │
└──────────────────────────────────────────────────────────┘

DeterministicPlanner runs a fixed, published sequence of 8 extraction probes — same input every run, so results are comparable across targets or over time. LLMPlanner swaps that for an adaptive model that generates the next probe from the target's last response; it's more exploratory but explicitly non-reproducible, which is why the harness keeps both as separate, swappable planner classes rather than picking one.

4. Architecture: The Scope-Gated Recon Agent

A second, separate pipeline lets an LLM drive read-only recon in plain language, while the scope wall — not the model's judgment — decides what's allowed to run:

┌──────────────────────────────────────────────────────────┐
│                 Natural-language request                 │
│          e.g. "enum web dirs and audit headers"          │
└──────────────────────────────────────────────────────────┘
                              ▼
┌──────────────────────────────────────────────────────────┐
│                    ReconAgent._plan()                    │
│         LLM picks: tool + host (from allowlist)          │
└──────────────────────────────────────────────────────────┘
                              ▼
┌──────────────────────────────────────────────────────────┐
│                ReconDispatcher.dispatch()                │
│               Scope.assert_in_scope(host)                │
└──────────────────────────────────────────────────────────┘
             ▼                              ▼
┌────────────────────────┐  ┌──────────────────────────────┐
│      out of scope      │  │    in scope -> tool runs     │
│     -> ScopeError      │  │  port_scan, service_probe,   │
│      refused, no       │  │   dir_scan, header_audit,    │
│      packet sent       │  │  tls_cert, ollama_profile,   │
│                        │  │     robots_sitemap, ...      │
└────────────────────────┘  └──────────────────────────────┘
                                            ▼
┌──────────────────────────────────────────────────────────┐
│                   ReconAgent.assess()                    │
│          JSON debrief: discovered / not_found /          │
│           observations / worth_checking / next           │
└──────────────────────────────────────────────────────────┘

Key components:

• ▹ReconDispatcher: the only bridge between an LLM tool-call and the network. It holds a fixed whitelist of callable tools — if a tool name isn't in that dict, the model cannot invoke it, full stop. The LLM can drive the toolset; it cannot widen scope.
• ▹ReconKit / NativeScanner: pure-Python, read-only enumeration — port scan, service/version probing, TLS certificate parsing, HTTP method and security-header auditing, robots.txt/sitemap discovery, an Ollama API profiler, and a ~200-entry directory wordlist. No shelling out to nmap, ffuf, or nikto; no writes, no credential spraying, no exploitation.
• ▹ReconAgent.assess(): after a run, a separate LLM call produces a structured debrief — discovered, not_found, observations, and a "worth checking" list — explicitly instructed to name directions to investigate (e.g., "SMB 445 open → verify patch level"), never exploit steps or payloads.

5. Where This Is Heading: AI-to-AI Warfare

I've been researching the adversarial dynamics between AI systems since mid-2022 — one of the first people looking at this seriously, starting with jailbreak and adversarial prompting research against Midjourney's content filter months before ChatGPT existed, and carrying that line of research forward into agentic red-teaming. recon-sweep is where that research becomes engineering: everything in the two pipelines above — a planner selecting probes, a dispatcher enforcing scope, a detector suite issuing a verdict — is a small, single-organization version of a shift that's already underway: security testing carried out by an autonomous system instead of a person driving a terminal. The next stage isn't a bigger CTF harness. It's the same architecture pointed at another organization's live systems, running continuously, with no human in the loop on either side.

The asymmetry is what makes this dangerous: an attacker's autonomous agent has no reason to self-impose a Scope allowlist, no reason to cap its probe rate, no reason to stop at the first canary. The scope discipline and falsifiable verification that make recon-sweep safe to run against your own CTF targets are exactly the primitives an attacker will happily discard — which means defenders can't rely on the attacker's system behaving responsibly. Defense has to close the gap architecturally, not by hoping the other side plays fair.

• ▹Today: offense is mostly scripted tools plus a human operator; agentic harnesses like recon-sweep are a preview, run inside a controlled, self-imposed scope.
• ▹Emerging: offense becomes autonomous agents probing live third-party infrastructure continuously, with no scope guardrail, because the attacker has no incentive to add one.
• ▹Required response: defense has to become an autonomous system too — one that plans, detects, and responds at machine speed — because human-speed incident response cannot keep pace with machine-speed reconnaissance and exploitation.

The engineering primitives don't change just because the direction flips. A defensive agent still needs the same falsifiable-verification discipline recon-sweep applies to offense: a detection has to clear a deterministic gate — a confirmed indicator, not an LLM's self-reported confidence — before it triggers an automated response, or the defensive agent becomes its own source of false positives at machine speed. The organizations that get this right won't be the ones that simply deploy the most AI; they'll be the ones whose offensive and defensive AI are both built on enforced scope and falsifiable ground truth.

6. Frequently Asked Questions

7. Explore the Code