Feedback Observing

Read agent interaction logs across all packs, detect operational patterns (error rates, duration outliers, zero-invocation skills), and emit state-transition FeedbackEvents when patterns change.

This skill closes the agent feedback loop — agents report back what worked and what failed.

Core Principle

State-transition only emission. Only emit FeedbackEvents when a skill's operational state changes (healthy → concerning, concerning → healthy). This prevents recursive signal inflation from repeated analysis of the same stable state.

Triggers

/feedback-observe                    # Analyze all packs' agent logs
/feedback-observe --pack observer    # Specific pack only
/feedback-observe --since 7d         # Time range (default: 30d)

When to Use

• Periodically to check agent health across packs
• After deploying new skills or modifying existing ones
• When suspecting a skill is silently failing
• As part of a daily/weekly maintenance routine

Workflow

Phase 1: Collect Logs

Glob: grimoires/*/agent-logs/*.jsonl

For each JSONL file:
  Parse each line (skip malformed lines, log warning)
  Filter by --pack if specified
  Filter by --since if specified (compare ts field)

Collect all log entries into a unified dataset.

If no log files found: Report "No agent logs found. Skills need to emit logs — see agent-log-format.md" and exit.

Phase 2: Detect Patterns

Group entries by pack and skill, then compute:

For each (pack, skill) pair:

  1. ERROR RATE
     error_count = entries where status="error"
     total_count = all entries
     error_rate = error_count / total_count

     IF error_rate > 0.20 → state = "concerning"
     IF error_rate <= 0.10 → state = "healthy"
     IF 0.10 < error_rate <= 0.20 → state = "warning"

  2. DURATION OUTLIERS
     Compute median duration_ms across all entries for this skill
     IF any entry has duration_ms > 2x median → flag as outlier

     IF outlier_count / total_count > 0.20 → state = "slow"

  3. ZERO-INVOCATION SKILLS
     Compare skills found in logs against manifest.json skills arrays
     Skills in manifest but NOT in logs → "dormant" (potential discoverability problem)

  4. RETRY PATTERNS
     Detect same skill invoked 3+ times within 5 minutes → "friction"

  5. MISSING ARTIFACT PATTERNS
     Entries where error contains "not found" or "missing" → "missing dependency"

Phase 3: Compare with Previous Report

Read previous report: grimoires/observer/agent-feedback/report-latest.md

IF previous report exists:
  Parse previous state per skill
  Compare current state with previous state

  ONLY emit FeedbackEvents for STATE TRANSITIONS:
    - healthy → concerning (degradation)
    - concerning → healthy (recovery)
    - new skill appeared as "dormant"
    - skill crossed duration outlier threshold for first time

  DO NOT re-emit for unchanged states.

IF no previous report:
  Treat all current states as new (emit for any non-healthy state)

Phase 4: Emit State-Transition FeedbackEvents

For each state transition detected, emit via the Loa event bus:

bash
1source .claude/scripts/lib/event-bus.sh
2
3emit_event "observer.state_transition" \
4  '{
5    "domain": "code",
6    "target": {
7      "type": "artifact",
8      "selector": "{pack}/{skill}"
9    },
10    "signal": {
11      "direction": "{negative for degradation, positive for recovery}",
12      "weight": 0.5,
13      "specificity": 0.7,
14      "content": "{state_transition}: {description}",
15      "kind": "behavioral"
16    },
17    "fingerprint": "{sha256 of '{pattern_type}:{pack}:{skill}:{window_start}:{window_end}'}"
18  }' \
19  "observer/feedback-observing" \
20  "" "" \
21  "{pack}/{skill}"

The bus auto-generates id, time, specversion in the CloudEvents envelope.

Fingerprint deduplication: The fingerprint field maps to the bus idempotency key. Same fingerprint = same finding for the same time window. The bus .idempotency/ store prevents duplicate processing.

Phase 5: Generate Report

Write to grimoires/observer/agent-feedback/report-{YYYY-MM-DD}.md:

markdown
1---
2type: agent-feedback-report
3generated: "{RFC 3339 UTC}"
4period: "{start} to {end}"
5packs_analyzed: {N}
6skills_analyzed: {N}
7transitions_detected: {N}
8---
9
10# Agent Feedback Report: {date}
11
12## Skill Health Matrix
13
14| Pack | Skill | Invocations | Error Rate | Median Duration | State |
15|------|-------|-------------|-----------|-----------------|-------|
16| observer | observing-users | 12 | 8% | 340ms | healthy |
17| artisan | inscribing-taste | 5 | 40% | 120ms | concerning |
18| ... | ... | ... | ... | ... | ... |
19
20## State Transitions (since last report)
21
22| Skill | Previous | Current | Evidence |
23|-------|----------|---------|----------|
24| artisan/inscribing-taste | healthy | concerning | Error rate 8% → 40% |
25
26## Duration Analysis
27
28| Skill | Median | P95 | Outliers |
29|-------|--------|-----|----------|
30| ... | ... | ... | ... |
31
32## Dormant Skills
33
34| Pack | Skill | In Manifest | Invocations |
35|------|-------|-------------|-------------|
36| ... | ... | yes | 0 |
37
38## Suggested Actions
39
40- {actionable recommendation per concerning skill}

Also copy to grimoires/observer/agent-feedback/report-latest.md (overwrite).

Create grimoires/observer/agent-feedback/ directory if it doesn't exist.

Error Handling

Integration Points

• Reads: grimoires/*/agent-logs/*.jsonl, grimoires/observer/agent-feedback/report-latest.md
• Writes: grimoires/observer/agent-feedback/report-{date}.md, grimoires/shared/feedback/events/{date}.jsonl
• Depends on: Skills emitting agent logs (Task 1.7)
• Consumed by: /artisan-patterns cross-domain analysis, /stale confidence system

Validation

• Reads all packs' agent logs via glob
• Correctly computes error rate, duration outliers
• Only emits FeedbackEvents on state transitions
• Fingerprint prevents duplicate emissions for same window
• Report includes all analyzed skills
• report-latest.md is updated
• --pack filter works correctly
• --since filter works correctly
• Handles empty/missing log directories gracefully