HITL Setup — Human-in-the-Loop Approval Workflow

Mergen's HITL (Human-in-the-Loop) system suspends sensitive AI agent tool calls and routes them to a Slack thread for human approval before execution resumes. This is the feature that Humanloop was building before being acquired by Anthropic — Mergen ships it today.

How it works

AI agent calls a sensitive tool
    │
    ▼
Mergen gate evaluates → verdict: HOLD
    │
    ├── Promise suspended (handler NOT called yet)
    ├── Slack message fired to your #incidents channel
    │   "Agent wants to: kubectl delete namespace production
    │    Why: Targeted data mutation by AI agent
    │    [✅ Approve] [❌ Deny] [🔒 Approve with Constraints]"
    │
    ▼
Human clicks Approve or Deny in Slack
    │
    ├── Approve → handler executes, result returned to agent
    └── Deny    → agent receives cancellation error, reformulates

If no human responds within the timeout (default: 10 minutes), the hold is automatically denied and logged as a timeout in the Agent Blunder Log.

Step 1: Create a Slack bot

1. Go to api.slack.com/apps → Create New App → From Scratch
2. App name: Mergen | Workspace: your workspace
3. OAuth & Permissions → Bot Token Scopes:
   • chat:write
   • channels:read
   • channels:history (for MERGEN_SLACK_OVERRIDE_LOOP)
4. Install to Workspace → copy the Bot User OAuth Token (xoxb-...)
5. Basic Information → Signing Secret → copy it
6. Invite the bot to your incidents channel: /invite @Mergen

Step 2: Configure Mergen

Add to your environment (or server/.env):

MERGEN_SLACK_BOT_TOKEN=xoxb-your-token-here
MERGEN_SLACK_CHANNEL=#incidents
MERGEN_SLACK_SIGNING_SECRET=your-signing-secret-here
MERGEN_PUBLIC_URL=https://mergen.your-domain.com  # required for approve/deny links

MERGEN_PUBLIC_URL is required. Slack url: buttons open $MERGEN_PUBLIC_URL/hitl/approve?token=<uuid> in the user's browser. Without it, buttons won't work.

Restart Mergen: mergen-server start

Step 3: Verify the HITL webhook

Test by triggering a HOLD manually:

# Simulate a sensitive tool call via the gate
curl -X POST http://127.0.0.1:3000/gate/evaluate \
  -H 'Content-Type: application/json' \
  -H 'x-mergen-secret: your-secret' \
  -d '{"command": "kubectl delete namespace staging", "toolName": "bash"}'

You should see a Slack message appear within 2 seconds. Click Approve or Deny in Slack.

Step 4: Check the pending queue

# List all currently held tool calls
curl http://127.0.0.1:3000/hitl/pending \
  -H 'x-mergen-secret: your-secret'

Response:

{
  "ok": true,
  "count": 1,
  "pending": [{
    "token": "abc123",
    "toolName": "bash",
    "commandArg": "kubectl delete namespace staging",
    "policyReason": "Targeted data mutation or resource removal by AI agent.",
    "suggestedAlt": "Run kubectl get namespace staging first to confirm state, then use kubectl delete with --dry-run=client to preview.",
    "triggeredRules": ["hold_agent_data_mutations"],
    "heldAt": 1720269600000,
    "waitingSec": 45,
    "quorumRequired": 1,
    "approvedCount": 0,
    "approveUrl": "POST /hitl/approve?token=abc123",
    "denyUrl": "POST /hitl/deny?token=abc123"
  }]
}

Step 5: Programmatic approve/deny (CI / scripts)

# Approve
curl -X POST "http://127.0.0.1:3000/hitl/approve?token=TOKEN" \
  -H 'x-mergen-secret: your-secret'

# Deny
curl -X POST "http://127.0.0.1:3000/hitl/deny?token=TOKEN" \
  -H 'x-mergen-secret: your-secret'

# Approve with constraints (agent must honour these params)
curl -X POST "http://127.0.0.1:3000/hitl/approve-constrained?token=TOKEN" \
  -H 'Content-Type: application/json' \
  -H 'x-mergen-secret: your-secret' \
  -d '{"constraints": {"targetNamespace": "staging", "dryRun": true}}'

What triggers a HOLD

By default, the following trigger a HOLD (not a BLOCK) for AI actor tool calls:

To convert a HOLD to a BLOCK (no human approval needed, just deny): add the pattern to a block_* rule in your policy. See mergen-server policy-push.

HITL approval fatigue

If your team is receiving too many HITL notifications, check:

curl http://127.0.0.1:3000/hitl/fatigue -H 'x-mergen-secret: your-secret'

The fatigue endpoint reports whether the hold rate has exceeded the threshold in the last hour and suggests converting noisy HOLD rules to BLOCK rules so humans only see the genuinely ambiguous cases.

RBAC: Who can approve

If your team has multiple on-call engineers, configure RBAC so only people with the responder role can approve HITL holds:

# Add a member with responder role
curl -X POST http://127.0.0.1:3000/rbac/members \
  -H 'Content-Type: application/json' \
  -H 'x-mergen-secret: your-secret' \
  -d '{"memberId": "alice@company.com", "role": "responder"}'

Once any member is configured, approvals require the x-mergen-member header to identify the approver. Anonymous approvals are rejected.

Quorum approvals

For high-risk operations, require two approvers:

{
  "name": "hold_prod_delete_quorum",
  "type": "warn",
  "patterns": ["kubectl delete namespace production"],
  "quorumRequired": 2,
  "reason": "Deleting production namespace requires two approvals."
}

Add via POST /policies and mergen-server policy-pull.

Timeout configuration

Default HITL timeout is 10 minutes. Set MERGEN_HITL_TIMEOUT_SECONDS to change:

MERGEN_HITL_TIMEOUT_SECONDS=300   # 5 minutes
MERGEN_HITL_TIMEOUT_SECONDS=3600  # 1 hour

Timed-out holds are automatically denied and logged. The agent receives a HITL timeout error and reformulates.

A mergen-server restart also denies any pending holds — this is by design, not a bug. A hold's resolver is an in-memory Promise tied to the MCP client's live connection; it can't survive the process it belongs to. Restarting the server (crash, redeploy, mergen-server stop/start, doctor --fix) denies every hold that was still pending at shutdown, logs the tokens, and requires the agent to reformulate/retry the original tool call — the same recovery model already used after a BLOCK. Holds are never silently resumed and never silently dropped; if you see denied holds after a restart, that's the expected dead-letter behavior, not data loss to chase down.

The override corpus loop

When you approve a HITL hold with "Remember for this service", Mergen encodes the approval as an override corpus entry. The next time the same agent action is attempted in the same context, Mergen checks the corpus and can escalate autonomously without paging the on-call — or hold again if the context has changed.

See docs/OVERRIDE_CORPUS.md for the full override corpus documentation.

Troubleshooting

# Confirm HITL is wired correctly
mergen-server doctor