PromptSlayer
Try Demo

System design

Architecture

PromptSlayer runs asynchronous audits with bounded concurrency and sends live state snapshots over token-protected WebSockets.

Evaluation pipeline

  1. 1

    Target setup

    The operator selects OpenAI, Anthropic, Google, or a custom endpoint or webhook for an authorized audit.

  2. 2

    Audit scheduling

    The FastAPI backend selects prompts from the corpus and runs tests asynchronously with a semaphore that bounds concurrent requests.

  3. 3

    Provider requests

    A configurable delay spaces requests. HTTP 429 responses are retried using Retry-After when available or exponential backoff.

  4. 4

    Response evaluation

    The backend applies local response checks and can use a configured model-based evaluator. If that evaluation fails, local checks run.

  5. 5

    Live progress

    A token-protected WebSocket sends current audit state snapshots to a connected browser as the run progresses.

  6. 6

    Stored report

    Results are stored in SQLite. Startup cleanup deletes audits older than 30 days; ephemeral storage may remove them earlier.

Service layers

Security console (React)

React · TypeScript · Vite · TanStack Start

Operator UI for audit setup, live progress, and report review. It reads state snapshots from an audit WebSocket.

Live progress (WebSockets)

FastAPI WebSocket · audit token · state snapshots

The backend checks an audit-specific token before accepting a connection and sends current audit state to the connected client.

Audit API (FastAPI)

Python · FastAPI · Pydantic · asyncio

Starts audits as asynchronous tasks, bounds concurrency with an asyncio semaphore, and applies configurable delays between requests.

Provider adapters

OpenAI · Anthropic · Google · custom endpoints/webhooks

Sends prompts to the configured service. HTTP 429 rate-limit responses are retried using Retry-After when available or exponential backoff.

Response evaluation

Local checks · optional model-based evaluator

Classifies target responses with local checks or a configured evaluator. Local checks are used if model-based evaluation fails.

Corpus & results store

JSON prompt corpus · SQLite audit reports

Stores audit reports and results in SQLite. Sensitive credential fields are removed before configuration is saved, and report access requires a view token.

WebSocket state snapshot

{ "type": "state",
  "audit_id": "audit_example",
  "status": "running",
  "progress": {
    "total": 100, "tested": 0, "passed": 0,
    "neutral": 0, "failed": 0, "bypassed": 0,
    "errored": 0, "success_rate": 0,
    "bypass_rate": 0, "average_severity": 0
  },
  "results": [],
  "current_technique": null,
  "in_progress_technique_uids": [] }