AgentMark

The open-source platform to develop, test, and observe your AI agents.

Homepage | Docs

AgentMark is a complete platform for building reliable AI agents. Define prompts in Markdown, run them with any SDK, evaluate quality with datasets, and trace every call in production.

Prompt management — Write prompts as .prompt.mdx files with type-safe inputs, tool definitions, structured outputs, conditionals, loops, and reusable components.

Observability — Trace every LLM call with OpenTelemetry. View traces locally or forward them to AgentMark Cloud for dashboards, alerts, and collaboration.

Evaluations — Test prompts against datasets with built-in evals. Run experiments from the CLI and gate deployments on quality thresholds.

---
name: customer-support-agent
text_config:
  model_name: anthropic/claude-sonnet-4-20250514
  max_calls: 2
  tools:
    search_knowledgebase:
      description: Search the knowledge base for shipping, warranty, and returns info.
      parameters:
        type: object
        properties:
          query:
            type: string
        required: [query]
test_settings:
  props:
    customer_question: "How long does shipping take?"
input_schema:
  type: object
  properties:
    customer_question:
      type: string
  required: [customer_question]
---

<System>
You are a helpful customer service agent. Use the search_knowledgebase tool
when customers ask about shipping, warranty, or returns.
</System>

<User>{props.customer_question}</User>

Run it:

agentmark run-prompt customer-support.prompt.mdx

That's it. The prompt is version-controlled, type-checked, and traceable.

# Scaffold a new project (interactive — picks your language and adapter)
npm create agentmark@latest

# Start the dev server (API + trace UI + hot reload)
agentmark dev

# Run a single prompt
agentmark run-prompt my-prompt.prompt.mdx

# Run an experiment against a dataset
agentmark run-experiment my-prompt.prompt.mdx

AgentMark doesn't call LLM APIs directly. Instead, adapters format your prompt for the SDK you already use.

Want another adapter? Open an issue.

See the examples/ directory for complete, runnable examples:

• Hello World — Simplest possible prompt
• Structured Output — Extract typed JSON with a schema
• Tool Use — Agent with tool calling
• Reusable Components — Import and compose prompts
• Evaluations — Test prompts against datasets
• Production Tracing — Trace LLM calls with the SDK

AgentMark Cloud extends the open-source project with:

• Collaborative prompt editing and version history
• Persistent trace storage with search and filtering
• Dashboards for cost, latency, and quality metrics
• Annotations and human evaluation workflows
• Alerts for quality regressions, cost spikes, and latency
• Two-way Git sync

We welcome contributions! See our contribution guidelines.

• GitHub Issues

MIT License