PromptSlayer
Try Demo

System design

Architecture

PromptSlayer is a streaming evaluation pipeline. The interesting engineering is in keeping thousands of concurrent adversarial probes rate-limit safe, resumable, and scored consistently.

Evaluation pipeline

  1. 1

    Target Model

    Any OpenAI-compatible endpoint, local model, or agent under authorized test.

  2. 2

    Adversarial Test Engine

    Selects, mutates and schedules payloads from the 2,238-entry corpus across 14 categories.

  3. 3

    Target Response

    Streamed back over WebSocket with per-test latency, tokens and rate-limit accounting.

  4. 4

    Semantic Evaluator

    LLM-as-a-Judge scores compliance vs refusal by meaning, not keyword matching.

  5. 5

    Security Classification

    Verdicts: BLOCKED · PARTIAL · BYPASSED · UNCERTAIN · ERROR, with confidence.

  6. 6

    Vulnerability Report

    Category footprint, severity ranking, reproduction steps and remediation guidance.

Service layers

Security console (React)

Next.js · TypeScript · Tailwind · shadcn/ui

Operator UI for run configuration, live monitoring and report review. Subscribes to a single WebSocket channel and reduces streamed events into audit state.

Streaming gateway (WebSockets)

FastAPI WebSocket · JSON event protocol · backpressure-aware

Publishes typed events (phase, test_result, verdict, error, complete) with sequence numbers so a reconnecting client can resume without gaps.

Orchestration API (FastAPI)

FastAPI · Pydantic · asyncio · asyncio background tasks

Validates the target declaration and authorization, plans the run, enforces per-provider rate limits and supervises worker concurrency.

Attack workers

async httpx · retry/backoff · provider adapters

Execute payloads against the target endpoint with jitter and rate-limit accounting. Multi-turn attacks hold conversational state across rounds.

Semantic evaluator

LLM-as-a-Judge · rubric prompts · confidence calibration

Scores each response for compliance by meaning, emitting a verdict plus confidence. Low-confidence cases are routed to UNCERTAIN for human triage instead of being guessed.

Corpus & results store

PostgreSQL · versioned payload corpus · run snapshots

Every run is immutable and comparable: same corpus version, same rubric, so resilience deltas across model releases are meaningful.

WebSocket event protocol

{ "seq": 142, "type": "test_result",
  "run_id": "PS-RUN-0061",
  "test_id": "0142",
  "category": "persona_injection",
  "latency_ms": 612,
  "verdict": "BLOCKED",
  "confidence": 0.96 }

{ "seq": 143, "type": "phase", "phase": "evaluating_responses" }
{ "seq": 144, "type": "complete", "tests": 2238, "resilience": 82 }