From d25f5339128aaec657be54b23e68f279120547c4 Mon Sep 17 00:00:00 2001 From: Simon Strandgaard Date: Fri, 17 Apr 2026 15:03:37 +0200 Subject: [PATCH] docs: add self_audit direction notes to module docstring Captures thoughts on future improvements to the self_audit checklist: - A dedicated system prompt for VIOLATES_KNOWN_PHYSICS to reduce false positives (elephant-alpha model especially prone to flagging non-physics plans as physics violations). - Proposed new checklist items: * Find fabricated evidence (hallucinated laws, articles, citations). * Call out fake confidence where evidence is insufficient. * Attack on vague filler language ("develop a robust strategy", "implement a communication strategy") that doesn't specify the actual strategy. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../self_audit/self_audit.py | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/worker_plan/worker_plan_internal/self_audit/self_audit.py b/worker_plan/worker_plan_internal/self_audit/self_audit.py index 9b50e2e45..129c60253 100644 --- a/worker_plan/worker_plan_internal/self_audit/self_audit.py +++ b/worker_plan/worker_plan_internal/self_audit/self_audit.py @@ -9,6 +9,26 @@ Value: It serves as a prioritized "fix-it" list. It tells the project manager which fires are the biggest. They don't need to worry about the team size (a ⚠️ Medium risk) if the entire project is a 🛑 High "Legal Minefield." It focuses their attention on the foundational, existential threats to the project's success. + +The "violates known physics" detection is often gets triggered, freaking out about "faster than light travel" on documents that have nothing to do with FTL. +The "elephant-alpha" model especially struggle with detecting that, when it otherwise does an ok job at everything else. +I'm considering making a dedicated system prompt only for VIOLATES_KNOWN_PHYSICS, that is have lower rate of false positives. + +Adding a new checklist item: +Find fabricated evidence. Where things aren't true. +When it hallucinates laws or articles that are non-existing. + +Adding a new checklist item: +Calling out fake confidence in the plan, where there isn’t sufficient evidence. + +Adding a new checklist item: +attack on vague filler language. +It's filler language when the document repeatedly says +- “develop a robust strategy” +- “implement a communication strategy,” +Without specify the actual strategy. +that is placeholder language. + Calibation. Skewed distribution: 16 “High” flags out of 20 reads like alarm fatigue. Yes, the majority may be red. Holding the items up against each other is not the job for this checklist. It's further downstream. I'm not going to reserve High for the most significant issues.