DocumentGenerator_Architecture

MILTON DocumentGenerator Architecture

Overview

The MILTON.DocumentGenerator is a worker service responsible for generating content (e.g. Requirements, Scanners, Test Cases) via LLMs. It operates exclusively through messages, receiving generation requests from the API via RabbitMQ (using Wolverine), reading payloads from an S3-backed claim-check store, processing them without direct database access (except for Wolverine’s envelope and saga storage), and replying with the results.

The architecture follows a Clean Architecture structure:

• Domain: Contains JSON models and configuration shapes.
• Contracts: Contains transport models and Wolverine messages.
• Features: Grouped by slice (Requirements, Scanners, TestCases, Common). Each slice contains a Handler (acting as the message consumer) and a Core (pure, decoupled business logic invoking the LLM).
• Sagas: Orchestrates multi-step processes via Wolverine Saga.
• Infrastructure: E.g., Notifications.
• Persistence: Contains the Wolverine backing database setup (though no application state is stored here).

Sagas

DocumentProcessingSaga

Orchestrates document block processing. The saga is DB-free—it receives the processable block list (with types) in DocumentCreatedEvent and carries it as its own state, dispatching each block to the API via Prepare*GenerationCommand and advancing on BlockProcessedEvent.

• Triggered By: DocumentCreatedEvent (from API)
• State: Pending blocks, currently processed block ID, processed count, failed block IDs.
• Workflow:
  1. Starts and sends a progress notification.
  2. Processes the next block by publishing a Prepare{Type}GenerationCommand (e.g. PrepareRequirementGenerationCommand) to the API.
  3. Pauses until the API responds with a BlockProcessedEvent.
  4. If BlockProcessedEvent spawns new blocks, they are added to the processing queue.
  5. Repeats until all blocks are processed.
  6. On completion, notifies the user and sends GeneratePdfCommand.

Domain Models

Domain models reflect JSON properties stored within block shapes in the API database, or configuration definitions.

• RequirementModel: Defines properties for a Requirement block (status, safetyLevel, sourceFile, lineNumber, requirementId, rationale).
• ClusterModel: Defines a cluster of source files for Scanners (sourceFiles, outputMode, contextDescription, moduleName).
• LlmPresetConfig: Represents an overridden configuration for an LLM (apiKey, providerUrl, model, temperature, systemPrompt, maxContextTokens, maxOutputTokens).
• BlockTypes: Constants for block types (Requirement, TestCase, TestResult, Text, Mermaid, Scanner, TraceMatrix).

Message Contracts

Orchestration Contracts

• ProcessableBlockRef: (Guid Id, string Type)
• DocumentCreatedEvent: Starts the saga (Guid DocumentId, int ProjectId, List<ProcessableBlockRef> ProcessableBlocks)
• BlockProcessedEvent: Reports a block applied (Guid DocumentProcessingSagaId, Guid BlockId, bool Success, string Message, List<ProcessableBlockRef> SpawnedBlocks)
• ProcessNextBlockCommand: Internal saga self-command to process the next block.
• DocumentGenerationCompletedEvent: Saga completion event.
• GeneratePdfCommand: Command sent to the API to generate a PDF (Guid DocumentId).

Feature-Specific Contracts

Requirements

• RequirementGenerationPayload: S3 Claim-check payload (RequirementGenerationInput Input, List<LlmPresetConfig>? PresetConfigs).
• PrepareRequirementGenerationCommand: Saga → API (Guid SagaId, Guid BlockId).
• RequirementGenerationRequested: API → Worker (Guid SagaId, Guid BlockId, string InputClaimCheckKey).
• RequirementGenerationCompleted: Worker → API (Guid SagaId, Guid BlockId, bool Success, string Message, string ResultClaimCheckKey).

Scanners

• ScannerGenerationPayload: S3 Claim-check payload (ScannerGenerationInput Input, List<LlmPresetConfig>? PresetConfigs).
• ScannerGenerationInput: Inputs for clustering (SourceFiles, RepoUrl, Branch, Granularity, etc.).
• ScannerGenerationResult: Output of clustering (Clusters, BlockTypes).
• PrepareScannerGenerationCommand: Saga → API (Guid SagaId, Guid BlockId).
• ScannerGenerationRequested: API → Worker (Guid SagaId, Guid BlockId, string InputClaimCheckKey).
• ScannerGenerationCompleted: Worker → API (Guid SagaId, Guid BlockId, bool Success, string Message, string ResultClaimCheckKey).

TestCases (Similar structure to Requirements/Scanners, managing PrepareTestCaseGenerationCommand, TestCaseGenerationRequested, TestCaseGenerationCompleted, and an S3 claim check payload).