Initial commit: add .gitignore and README
This commit is contained in:
defiQUG
66
docs/README.md
Normal file
66
docs/README.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# FusionAGI Documentation
|
||||
|
||||
This folder contains architecture, API, and feature documentation for FusionAGI.
|
||||
|
||||
## Documentation Map
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph getting_started [Getting Started]
|
||||
README[README.md at project root]
|
||||
end
|
||||
|
||||
subgraph architecture [Architecture & Design]
|
||||
ARCH[architecture.md]
|
||||
IAD[interface_architecture_diagram.md]
|
||||
end
|
||||
|
||||
subgraph api [API & Integration]
|
||||
API_SPEC[api_middleware_interface_spec.md]
|
||||
OPENAI[openai_bridge.md]
|
||||
end
|
||||
|
||||
subgraph features [Features]
|
||||
INTERFACES[interfaces.md]
|
||||
UI_UX[ui_ux_implementation.md]
|
||||
MAA_ACT[maa_activation.md]
|
||||
MAA_MAP[maa_compliance_mapping.md]
|
||||
MULTI[multi_agent_acceleration.md]
|
||||
end
|
||||
|
||||
README --> ARCH
|
||||
README --> API_SPEC
|
||||
README --> OPENAI
|
||||
ARCH --> IAD
|
||||
API_SPEC --> INTERFACES
|
||||
INTERFACES --> UI_UX
|
||||
MAA_ACT --> MAA_MAP
|
||||
```
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| **[architecture.md](architecture.md)** | Core components, data flow, AGI stack, self-improvement subsystem. Includes Mermaid diagrams. |
|
||||
| **[interface_architecture_diagram.md](interface_architecture_diagram.md)** | Full system and interface layer diagrams (ASCII + reference). Admin panel, core, multi-modal UI, end-user devices. |
|
||||
| **[api_middleware_interface_spec.md](api_middleware_interface_spec.md)** | Dvādaśa HTTP/WebSocket API, SafetyPipeline, AdminControlPanel & MultiModalUI spec, Voice/Conversation layer. |
|
||||
| **[openai_bridge.md](openai_bridge.md)** | Using FusionAGI as an OpenAI-compatible model in Cursor Composer. Setup, config, auth. |
|
||||
| **[interfaces.md](interfaces.md)** | Admin Control Panel and Multi-Modal UI usage. Voice library, conversation tuning, modalities. |
|
||||
| **[ui_ux_implementation.md](ui_ux_implementation.md)** | Implementation summary for admin panel and multi-modal UI. File structure, testing, next steps. |
|
||||
| **[maa_activation.md](maa_activation.md)** | Manufacturing Authority Add-On: activation steps, MPC lifecycle, deployment. |
|
||||
| **[maa_compliance_mapping.md](maa_compliance_mapping.md)** | High-level mapping of MAA to ASME, ISO, MIL-STD, DoD Digital Thread. |
|
||||
| **[multi_agent_acceleration.md](multi_agent_acceleration.md)** | Parallel execution, agent pool, delegation, supervisor, batch/async routing. |
|
||||
|
||||
## Visual Elements
|
||||
|
||||
Documentation uses **Mermaid** diagrams where supported (GitHub, GitLab, many Markdown viewers). Key visuals:
|
||||
|
||||
- **README.md:** System-at-a-glance flowchart, task lifecycle sequence.
|
||||
- **architecture.md:** Component overview, data flow (task lifecycle), self-improvement flow.
|
||||
- **api_middleware_interface_spec.md:** API → pipeline → core → interfaces flowchart.
|
||||
- **openai_bridge.md:** Request flow (Cursor → FusionAGI API → Dvādaśa → response).
|
||||
- **multi_agent_acceleration.md:** Supervisor and executor pool architecture.
|
||||
- **maa_activation.md:** MAA activation and MPC flow.
|
||||
- **interface_architecture_diagram.md:** ASCII system/interface diagrams (terminal-friendly).
|
||||
|
||||
For coding standards and project conventions, see `.cursor/rules` in the project root.
|
||||
346
docs/api_middleware_interface_spec.md
Normal file
346
docs/api_middleware_interface_spec.md
Normal file
@@ -0,0 +1,346 @@
|
||||
# FusionAGI API and Middleware Interface Specification
|
||||
|
||||
## 1. Dvādaśa HTTP/WebSocket API
|
||||
|
||||
The main programmatic entry point is the **Dvādaśa API**, a FastAPI application exposing session-based prompts and streaming responses.
|
||||
|
||||
### Request Flow (Prompt → Response)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant API as Dvādaśa API
|
||||
participant Safety as SafetyPipeline
|
||||
participant Orch as Orchestrator
|
||||
participant Heads as Dvādaśa Heads
|
||||
participant Witness as Witness
|
||||
|
||||
Client->>API: POST /v1/sessions/{id}/prompt
|
||||
API->>Safety: pre_check(prompt)
|
||||
Safety-->>API: ModerationResult
|
||||
API->>Orch: submit_task(goal)
|
||||
Orch->>Heads: run_heads_parallel
|
||||
Heads->>Witness: head outputs
|
||||
Witness->>Orch: final_answer
|
||||
Orch->>API: FinalResponse
|
||||
API->>Safety: post_check(final_answer)
|
||||
Safety-->>API: OutputScanResult
|
||||
API->>Client: 200 + FinalResponse
|
||||
```
|
||||
|
||||
### 1.1 Application Factory
|
||||
|
||||
- **Location**: `fusionagi/api/app.py`
|
||||
- **Factory**: `create_app(adapter: Any = None, cors_origins: list[str] | None = None) -> FastAPI`
|
||||
- Creates FastAPI app with title "FusionAGI Dvādaśa API"
|
||||
- Accepts optional `LLMAdapter` for head/Witness LLM calls
|
||||
- Accepts optional `cors_origins` to enable CORS middleware (e.g. `["*"]` or `["https://example.com"]`)
|
||||
- Mounts router at prefix `/v1` with tag `dvadasa`
|
||||
- Uses startup event to initialize `Orchestrator`, `EventBus`, `SessionStore`, `SafetyPipeline`
|
||||
|
||||
### 1.2 Base URL and Router Structure
|
||||
|
||||
| Prefix | Router | Tag |
|
||||
| -------------- | ----------------- | -------- |
|
||||
| `/v1` | `api_router` | dvadasa |
|
||||
| `/v1/sessions` | `sessions_router` | sessions |
|
||||
|
||||
### 1.3 HTTP Endpoints
|
||||
|
||||
| Method | Path | Description |
|
||||
| --------- | ---------------------------------- | -------------------------------------------- |
|
||||
| POST | `/v1/sessions` | Create a new session |
|
||||
| POST | `/v1/sessions/{session_id}/prompt` | Submit prompt and receive sync FinalResponse |
|
||||
| WebSocket | `/v1/sessions/{session_id}/stream` | Streaming Dvādaśa response |
|
||||
|
||||
#### POST /v1/sessions
|
||||
|
||||
**Query params:**
|
||||
|
||||
- `user_id` (optional, str): User identifier
|
||||
|
||||
**Response (200):**
|
||||
|
||||
```json
|
||||
{
|
||||
"session_id": "uuid-string",
|
||||
"user_id": "optional-user-id"
|
||||
}
|
||||
```
|
||||
|
||||
#### POST /v1/sessions/{session_id}/prompt
|
||||
|
||||
**Path params:**
|
||||
|
||||
- `session_id` (required): Session UUID
|
||||
|
||||
**Body:**
|
||||
|
||||
```json
|
||||
{
|
||||
"prompt": "string (required)",
|
||||
"use_all_heads": false
|
||||
}
|
||||
```
|
||||
|
||||
**Pipeline:**
|
||||
|
||||
1. SafetyPipeline `pre_check(prompt)` - input moderation
|
||||
2. `parse_user_input(prompt)` - detect UserIntent (normal, head_strategy, show_dissent, etc.)
|
||||
3. Submit task via `orchestrator.submit_task(goal=prompt[:200])`
|
||||
4. `select_heads_for_complexity(prompt)` or explicit head from `/head <id>` command
|
||||
5. `run_dvadasa()` - parallel heads + Witness synthesis
|
||||
6. SafetyPipeline `post_check(final_answer)` - output scan (PII, blocked content)
|
||||
7. Append to session history
|
||||
|
||||
**Response (200):**
|
||||
|
||||
```json
|
||||
{
|
||||
"task_id": "string",
|
||||
"final_answer": "string",
|
||||
"transparency_report": { "head_contributions": [...] },
|
||||
"head_contributions": [...],
|
||||
"confidence_score": 0.0
|
||||
}
|
||||
```
|
||||
|
||||
**Errors:**
|
||||
|
||||
- 400: missing prompt, pre_check failed, post_check failed
|
||||
- 404: session not found
|
||||
- 500: Dvādaśa failed
|
||||
- 503: service not initialized
|
||||
|
||||
#### WebSocket /v1/sessions/{session_id}/stream
|
||||
|
||||
**Flow:**
|
||||
|
||||
1. Client connects and accepts
|
||||
2. Client sends `{"prompt": "..."}` JSON
|
||||
3. Server streams events via `send_json`:
|
||||
- `heads_running`
|
||||
- `head_complete` (per head: `head_id`, `summary`)
|
||||
- `witness_running`
|
||||
- `complete` (final_answer, transparency_report, head_contributions, confidence_score)
|
||||
- `error` (message)
|
||||
|
||||
### 1.4 Dependencies and App State
|
||||
|
||||
`fusionagi/api/dependencies.py` provides:
|
||||
|
||||
| Function | Returns | Purpose |
|
||||
| ------------------------------------------------------- | -------------------------- | ----------------------------------------------- |
|
||||
| `default_orchestrator(adapter)` | `(Orchestrator, EventBus)` | Build orchestrator with Dvādaśa heads + Witness |
|
||||
| `get_orchestrator()` | Orchestrator | From app state |
|
||||
| `get_event_bus()` | EventBus | From app state |
|
||||
| `get_session_store()` | SessionStore | In-memory session store |
|
||||
| `get_safety_pipeline()` | SafetyPipeline | Pre/post checks |
|
||||
| `set_app_state(orchestrator, event_bus, session_store)` | - | Populate app state |
|
||||
| `ensure_initialized(adapter)` | - | Lazy init for tests |
|
||||
|
||||
**SessionStore interface:**
|
||||
|
||||
- `create(session_id, user_id)` -> dict
|
||||
- `get(session_id)` -> dict | None
|
||||
- `append_history(session_id, entry)` -> None
|
||||
|
||||
---
|
||||
|
||||
## 2. Request Pipeline (SafetyPipeline - Inline "Middleware")
|
||||
|
||||
FusionAGI does **not** use HTTP middleware (CORS, auth, etc.) by default. The **SafetyPipeline** acts as an inline request/response pipeline invoked inside route handlers. CORS can be enabled via `create_app(cors_origins=[...])`.
|
||||
|
||||
### 2.1 Components
|
||||
|
||||
`fusionagi/governance/safety_pipeline.py`
|
||||
|
||||
| Class | Role |
|
||||
| ---------------- | ------------------------------------------------- |
|
||||
| `InputModerator` | Pre-check: block/transform user input |
|
||||
| `OutputScanner` | Post-check: PII, blocked patterns in final answer |
|
||||
| `SafetyPipeline` | Combines moderator + scanner + optional audit log |
|
||||
|
||||
### 2.2 InputModerator Interface
|
||||
|
||||
```python
|
||||
def add_blocked_pattern(self, pattern: str) -> None # Regex
|
||||
def add_blocked_phrase(self, phrase: str) -> None # Exact phrase
|
||||
def moderate(self, text: str) -> ModerationResult
|
||||
```
|
||||
|
||||
**ModerationResult:** `allowed: bool`, `transformed: str | None`, `reason: str | None`
|
||||
|
||||
### 2.3 OutputScanner Interface
|
||||
|
||||
```python
|
||||
def add_pii_pattern(self, name: str, pattern: str) -> None
|
||||
def add_blocked_pattern(self, pattern: str) -> None
|
||||
def scan(self, text: str) -> OutputScanResult
|
||||
```
|
||||
|
||||
**OutputScanResult:** `passed: bool`, `flags: list[str]`, `sanitized: str | None`
|
||||
|
||||
Default PII: SSN, credit card patterns.
|
||||
|
||||
### 2.4 SafetyPipeline Interface
|
||||
|
||||
```python
|
||||
def pre_check(self, user_input: str) -> ModerationResult
|
||||
def post_check(self, final_answer: str) -> OutputScanResult
|
||||
```
|
||||
|
||||
Used in `fusionagi/api/routes/sessions.py` around prompt submission and final answer delivery.
|
||||
|
||||
---
|
||||
|
||||
## 3. Multi-Modal Interface Layer
|
||||
|
||||
The **Interface** layer provides abstractions for admin control and user interaction across modalities. This is not HTTP middleware but a conceptual middleware between end users and the core.
|
||||
|
||||
### 3.1 Base Abstractions
|
||||
|
||||
`fusionagi/interfaces/base.py`
|
||||
|
||||
| Type | Description |
|
||||
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `ModalityType` | Enum: TEXT, VOICE, VISUAL, HAPTIC, GESTURE, BIOMETRIC |
|
||||
| `InterfaceMessage` | Pydantic: id, modality, content, metadata, timestamp, user_id, session_id |
|
||||
| `InterfaceCapabilities` | supported_modalities, supports_streaming, supports_interruption, supports_multimodal, latency_ms, max_concurrent_sessions |
|
||||
|
||||
**InterfaceAdapter (ABC):**
|
||||
|
||||
```python
|
||||
def capabilities(self) -> InterfaceCapabilities
|
||||
async def send(self, message: InterfaceMessage) -> None
|
||||
async def receive(self, timeout_seconds: float | None = None) -> InterfaceMessage | None
|
||||
async def stream_send(self, messages: AsyncIterator[InterfaceMessage]) -> None # default impl
|
||||
async def initialize(self) -> None
|
||||
async def shutdown(self) -> None
|
||||
def validate_message(self, message: InterfaceMessage) -> bool
|
||||
```
|
||||
|
||||
### 3.2 AdminControlPanel
|
||||
|
||||
`fusionagi/interfaces/admin_panel.py`
|
||||
|
||||
**Constructor:** `AdminControlPanel(orchestrator, event_bus, state_manager, voice_library?, conversation_tuner?, policy_engine?, audit_log?, session_count_callback?)`
|
||||
|
||||
| Area | Methods |
|
||||
| ------------ | --------------------------------------------------------------------------------------------------------------------- |
|
||||
| Voice | `add_voice_profile`, `list_voices`, `update_voice_profile`, `remove_voice_profile`, `set_default_voice` |
|
||||
| Conversation | `register_conversation_style`, `list_conversation_styles`, `get_conversation_style`, `set_default_conversation_style` |
|
||||
| Agents | `configure_agent`, `get_agent_config`, `list_agents`, `enable_agent`, `disable_agent` |
|
||||
| Monitoring | `get_system_status`, `get_task_statistics`, `get_recent_events` |
|
||||
| Governance | `get_audit_entries`, `update_policy` |
|
||||
| Config | `export_configuration`, `import_configuration` |
|
||||
|
||||
**Models:** `SystemStatus`, `AgentConfig`
|
||||
|
||||
### 3.3 MultiModalUI
|
||||
|
||||
`fusionagi/interfaces/multimodal_ui.py`
|
||||
|
||||
**Constructor:** `MultiModalUI(orchestrator, conversation_manager, voice_interface?, llm_process_callback?)`
|
||||
|
||||
| Area | Methods |
|
||||
| ------------ | --------------------------------------------------------------------------------------- |
|
||||
| Sessions | `create_session`, `get_session`, `active_session_count`, `end_session` |
|
||||
| Modalities | `register_interface`, `enable_modality`, `disable_modality`, `get_available_modalities` |
|
||||
| I/O | `send_to_user`, `receive_from_user` |
|
||||
| Tasks | `submit_task_interactive` |
|
||||
| Conversation | `converse` |
|
||||
| Stats | `get_session_statistics` |
|
||||
|
||||
**UserSession:** session_id, user_id, conversation_session_id, active_modalities, preferences, accessibility_settings, started_at, last_activity_at
|
||||
|
||||
### 3.4 Voice Layer
|
||||
|
||||
`fusionagi/interfaces/voice.py`
|
||||
|
||||
**Protocols:**
|
||||
|
||||
- `TTSAdapter`: `async def synthesize(text, voice_id?, **kwargs) -> bytes | None`
|
||||
- `STTAdapter`: `async def transcribe(audio_data?, timeout_seconds?, **kwargs) -> str | None`
|
||||
|
||||
**VoiceLibrary:** `add_voice`, `remove_voice`, `get_voice`, `list_voices`, `set_default_voice`, `get_default_voice`, `update_voice`
|
||||
|
||||
**VoiceProfile:** id, name, language, gender, age_range, style, pitch, speed, provider, provider_voice_id, metadata
|
||||
|
||||
**VoiceInterface(InterfaceAdapter):** implements send (TTS), receive (STT), `set_active_voice`, capabilities
|
||||
|
||||
### 3.5 Conversation Layer
|
||||
|
||||
`fusionagi/interfaces/conversation.py`
|
||||
|
||||
**ConversationStyle:** formality, verbosity, personality_traits, empathy_level, proactivity, humor_level, technical_depth
|
||||
|
||||
**ConversationTuner:** `register_style`, `get_style`, `list_styles`, `set_default_style`, `get_default_style`, `tune_for_context`
|
||||
|
||||
**ConversationManager:** `create_session`, `get_session`, `add_turn`, `get_history`, `get_style_for_session`, `update_style`, `end_session`, `get_context_summary`
|
||||
|
||||
**ConversationTurn:** turn_id, session_id, speaker (user|agent|system), content, intent, sentiment, confidence, timestamp
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture Summary
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph clients [Clients]
|
||||
HTTP[HTTP Client]
|
||||
WS[WebSocket Client]
|
||||
end
|
||||
|
||||
subgraph api [Dvādaśa API]
|
||||
Sessions[POST /v1/sessions]
|
||||
Prompt[POST /v1/sessions/id/prompt]
|
||||
Stream[WS /v1/sessions/id/stream]
|
||||
end
|
||||
|
||||
subgraph pipeline [Request Pipeline]
|
||||
PreCheck[SafetyPipeline.pre_check]
|
||||
PostCheck[SafetyPipeline.post_check]
|
||||
end
|
||||
|
||||
subgraph core [Core]
|
||||
Orch[Orchestrator]
|
||||
Heads[Dvādaśa Heads]
|
||||
Witness[Witness]
|
||||
end
|
||||
|
||||
subgraph interfaces [Interface Layer]
|
||||
Admin[AdminControlPanel]
|
||||
UI[MultiModalUI]
|
||||
Adapters[InterfaceAdapters]
|
||||
end
|
||||
|
||||
HTTP --> Sessions
|
||||
HTTP --> Prompt
|
||||
WS --> Stream
|
||||
Prompt --> PreCheck
|
||||
PreCheck --> Orch
|
||||
Orch --> Heads
|
||||
Heads --> Witness
|
||||
Witness --> PostCheck
|
||||
PostCheck --> Prompt
|
||||
Admin --> Orch
|
||||
UI --> Orch
|
||||
UI --> Adapters
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Adding HTTP Middleware
|
||||
|
||||
To enable CORS when creating the app:
|
||||
|
||||
```python
|
||||
from fusionagi.api import create_app
|
||||
|
||||
app = create_app(
|
||||
cors_origins=["*"], # or ["https://example.com", "https://app.example.com"]
|
||||
)
|
||||
```
|
||||
|
||||
For additional custom middleware (auth, logging, etc.), use `app.add_middleware()` after creating the app. FusionAGI relies on FastAPI/Starlette defaults (ServerErrorMiddleware, ExceptionMiddleware, AsyncExitStackMiddleware) when no custom middleware is configured.
|
||||
130
docs/architecture.md
Normal file
130
docs/architecture.md
Normal file
@@ -0,0 +1,130 @@
|
||||
# FusionAGI Architecture
|
||||
|
||||
High-level system components and data flow.
|
||||
|
||||
## Component Overview
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph core [Core]
|
||||
Orch[Orchestrator]
|
||||
EB[Event Bus]
|
||||
SM[State Manager]
|
||||
end
|
||||
|
||||
subgraph agents [Agents]
|
||||
Planner[Planner]
|
||||
Reasoner[Reasoner]
|
||||
Executor[Executor]
|
||||
Critic[Critic]
|
||||
Heads[Heads + Witness]
|
||||
end
|
||||
|
||||
subgraph support [Supporting Systems]
|
||||
Reasoning[Reasoning]
|
||||
Planning[Planning]
|
||||
Memory[Memory]
|
||||
Tools[Tools]
|
||||
Gov[Governance]
|
||||
end
|
||||
|
||||
Orch --> EB
|
||||
Orch --> SM
|
||||
Orch --> Planner
|
||||
Orch --> Reasoner
|
||||
Orch --> Executor
|
||||
Orch --> Critic
|
||||
Orch --> Heads
|
||||
Planner --> Planning
|
||||
Reasoner --> Reasoning
|
||||
Executor --> Tools
|
||||
Executor --> Gov
|
||||
Critic --> Memory
|
||||
```
|
||||
|
||||
## Data Flow (Task Lifecycle)
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
A[User submits task] --> B[Orchestrator]
|
||||
B --> C[Planner: plan graph]
|
||||
C --> D[Reasoner: reason on steps]
|
||||
D --> E[Executor: run tools via Governance]
|
||||
E --> F[State + Events drive next steps]
|
||||
F --> G{Complete?}
|
||||
G -->|No| D
|
||||
G -->|Yes| H[Critic evaluates]
|
||||
H --> I[Reflection updates memory]
|
||||
I --> J[FusionAGILoop: recommendations + training]
|
||||
J --> K[Task done / retry / recommendations]
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
- **Orchestrator (Fusion Core):** Global task lifecycle, agent scheduling, state propagation. Holds task graph, event bus, agent registry.
|
||||
- **Event bus:** In-process pub/sub for task lifecycle and agent messages.
|
||||
- **State manager:** In-memory (or persistent) store for task state and execution traces.
|
||||
|
||||
## Agent Framework
|
||||
|
||||
- **Base agent:** identity, role, objective, memory_access, tool_permissions. Handles messages via `handle_message(envelope)`.
|
||||
- **Agent types:** Planner, Reasoner, Executor, Critic, AdversarialReviewer, HeadAgent, WitnessAgent (`fusionagi.agents`). Supervisor, Coordinator, PooledExecutorRouter (`fusionagi.multi_agent`). Communication via structured envelopes (schemas).
|
||||
|
||||
## Supporting Systems
|
||||
|
||||
- **Reasoning engine:** Chain-of-thought (and later tree/graph-of-thought); trace storage.
|
||||
- **Planning engine:** Goal decomposition, plan graph, dependency resolution, checkpoints.
|
||||
- **Execution & tooling:** Tool registry, permission scopes, safe runner, result normalization.
|
||||
- **Memory:** Short-term (working), episodic (task history), reflective (lessons).
|
||||
- **Governance:** Guardrails, rate limiting, tool access control, human override hooks.
|
||||
|
||||
## Data Flow
|
||||
|
||||
1. User/orchestrator submits a task (goal, constraints).
|
||||
2. Orchestrator assigns work; Planner produces plan graph.
|
||||
3. Reasoner reasons on steps; Executor runs tools (through governance).
|
||||
4. State and events drive next steps; on completion, Critic evaluates and reflection updates memory/heuristics.
|
||||
5. **Self-improvement (FusionAGILoop):** On `task_state_changed` (FAILED), self-correction runs reflection and optionally prepares retry. On `reflection_done`, auto-recommend produces actionable recommendations and auto-training suggests/applies heuristic updates and training targets.
|
||||
|
||||
All components depend on **schemas** for tasks, messages, plans, and recommendations; no ad-hoc dicts in core or agents.
|
||||
|
||||
## Self-Improvement Subsystem
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph events [Event Bus]
|
||||
FAIL[task_state_changed: FAILED]
|
||||
REFL[reflection_done]
|
||||
end
|
||||
|
||||
subgraph loop [FusionAGILoop]
|
||||
SC[SelfCorrectionLoop]
|
||||
AR[AutoRecommender]
|
||||
AT[AutoTrainer]
|
||||
end
|
||||
|
||||
FAIL --> SC
|
||||
REFL --> AR
|
||||
REFL --> AT
|
||||
SC --> |retry| PENDING[FAILED → PENDING]
|
||||
AR --> |on_recommendations| Recs[Recommendations]
|
||||
AT --> |heuristic updates| Reflective[Reflective Memory]
|
||||
```
|
||||
|
||||
- **SelfCorrectionLoop:** On failed tasks, runs Critic reflection and can transition FAILED → PENDING with correction context for retry.
|
||||
- **AutoRecommender:** From lessons and evaluations, produces recommendations (next_action, training_target, strategy_change, etc.).
|
||||
- **AutoTrainer:** Suggests heuristic updates, prompt tuning, and fine-tune datasets; applies heuristic updates to reflective memory.
|
||||
- **FusionAGILoop:** Subscribes to event bus, wires correction + recommender + trainer into a single AGI self-improvement pipeline. Event handlers are best-effort: exceptions are logged and do not break other subscribers.
|
||||
|
||||
## AGI Stack
|
||||
|
||||
- **Executive:** GoalManager, Scheduler, BlockersAndCheckpoints (`fusionagi.core`).
|
||||
- **Memory:** WorkingMemory, EpisodicMemory, ReflectiveMemory, SemanticMemory, ProceduralMemory, TrustMemory, ConsolidationJob, MemoryService, VectorMemory (`fusionagi.memory`).
|
||||
- **Verification:** OutcomeVerifier, ContradictionDetector, FormalValidators (`fusionagi.verification`).
|
||||
- **World model:** World model base and rollout (`fusionagi.world_model`).
|
||||
- **Skills:** SkillLibrary, SkillInduction, SkillVersioning (`fusionagi.skills`).
|
||||
- **Multi-agent:** CoordinatorAgent, SupervisorAgent, AgentPool, PooledExecutorRouter, consensus_vote, arbitrate, delegate_sub_tasks (`fusionagi.multi_agent`). AdversarialReviewerAgent in `fusionagi.agents`.
|
||||
- **Governance:** Guardrails, RateLimiter, AccessControl, OverrideHooks, PolicyEngine, AuditLog, SafetyPipeline, IntentAlignment (`fusionagi.governance`).
|
||||
- **Tooling:** Tool registry, runner, builtins; DocsConnector, DBConnector, CodeRunnerConnector (`fusionagi.tools`).
|
||||
- **API:** FastAPI app factory, Dvādaśa sessions, OpenAI bridge, WebSocket (`fusionagi.api`).
|
||||
- **MAA:** MAAGate, MPCAuthority, ManufacturingProofCertificate, check_gaps (`fusionagi.maa`).
|
||||
370
docs/interface_architecture_diagram.md
Normal file
370
docs/interface_architecture_diagram.md
Normal file
@@ -0,0 +1,370 @@
|
||||
# FusionAGI Interface Architecture Diagram
|
||||
|
||||
## Complete System Architecture (Mermaid)
|
||||
|
||||
*Below: ASCII version for terminals. This Mermaid diagram renders in GitHub/GitLab and many Markdown viewers.*
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph admin [Administrator Layer]
|
||||
ACP[Admin Control Panel]
|
||||
ACP --> VL[Voice Library]
|
||||
ACP --> CT[Conversation Tuning]
|
||||
ACP --> AC[Agent Config]
|
||||
ACP --> SM[System Monitoring]
|
||||
ACP --> GP[Governance & Policy]
|
||||
ACP --> MAA[MAA Control]
|
||||
ACP --> CE[Config Export/Import]
|
||||
ACP --> AL[Audit Log]
|
||||
end
|
||||
|
||||
subgraph core [FusionAGI Core]
|
||||
Orch[Orchestrator]
|
||||
Agents[Agents]
|
||||
Memory[Memory]
|
||||
Tools[Tools]
|
||||
Reasoning[Reasoning]
|
||||
Planning[Planning]
|
||||
Gov[Governance]
|
||||
SI[Self-Improvement]
|
||||
end
|
||||
|
||||
subgraph ui [User Interface Layer]
|
||||
MMUI[Multi-Modal UI]
|
||||
MMUI --> Text[Text]
|
||||
MMUI --> Voice[Voice]
|
||||
MMUI --> Visual[Visual]
|
||||
MMUI --> Haptic[Haptic]
|
||||
MMUI --> Gesture[Gesture]
|
||||
MMUI --> Bio[Biometric]
|
||||
end
|
||||
|
||||
admin --> core
|
||||
core --> ui
|
||||
```
|
||||
|
||||
## Complete System Architecture (ASCII)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ ADMINISTRATOR LAYER │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Admin Control Panel │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
||||
│ │ │ Voice │ │Conversation │ │ Agent │ │ System │ │ │
|
||||
│ │ │ Library │ │ Tuning │ │ Config │ │ Monitoring │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │ │ │
|
||||
│ │ │ • Add/Edit │ │ • Styles │ │ • Enable/ │ │ • Status │ │ │
|
||||
│ │ │ • Voices │ │ • Formality │ │ Disable │ │ • Metrics │ │ │
|
||||
│ │ │ • Providers │ │ • Empathy │ │ • Limits │ │ • Tasks │ │ │
|
||||
│ │ │ • Default │ │ • Technical │ │ • Retry │ │ • Agents │ │ │
|
||||
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
|
||||
│ │ │ Governance │ │ MAA │ │ Config │ │ Audit │ │ │
|
||||
│ │ │ & Policy │ │ Control │ │ Export/ │ │ Log │ │ │
|
||||
│ │ │ │ │ │ │ Import │ │ │ │ │
|
||||
│ │ │ • Policies │ │ • MPC Mgmt │ │ • Backup │ │ • Actions │ │ │
|
||||
│ │ │ • Access │ │ • Gate │ │ • Restore │ │ • Compliance│ │ │
|
||||
│ │ │ • Guardrails│ │ • Machines │ │ • Version │ │ • History │ │ │
|
||||
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
|
||||
│ └────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ Configuration & Control
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ FUSIONAGI CORE SYSTEM │
|
||||
│ │
|
||||
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
|
||||
│ │Orchestrator│ │ Agents │ │ Memory │ │ Tools │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ • Tasks │ │ • Planner │ │ • Working │ │ • Registry │ │
|
||||
│ │ • Events │ │ • Reasoner │ │ • Episodic │ │ • Runner │ │
|
||||
│ │ • State │ │ • Executor │ │ • Semantic │ │ • Builtins │ │
|
||||
│ │ • Routing │ │ • Critic │ │ • Reflect │ │ • Connectors│ │
|
||||
│ └────────────┘ └────────────┘ └────────────┘ └────────────┘ │
|
||||
│ │
|
||||
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
|
||||
│ │ Reasoning │ │ Planning │ │Governance │ │Self-Improve│ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ • CoT │ │ • Graph │ │ • Guards │ │ • Correct │ │
|
||||
│ │ • ToT │ │ • Strategy │ │ • Rate Lim │ │ • Recommend│ │
|
||||
│ │ │ │ • Deps │ │ • Access │ │ • Train │ │
|
||||
│ └────────────┘ └────────────┘ └────────────┘ └────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ Task Execution & Events
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ USER INTERFACE LAYER │
|
||||
│ │
|
||||
│ ┌────────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Multi-Modal User Interface │ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────────────────────────────────────────────────────────────────┐ │ │
|
||||
│ │ │ Interface Adapters (Pluggable) │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │
|
||||
│ │ │ │ Text │ │ Voice │ │ Visual │ │ Haptic │ │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │ │ │ │ │
|
||||
│ │ │ │ • Chat │ │ • STT │ │ • Images │ │ • Touch │ │ │ │
|
||||
│ │ │ │ • Command│ │ • TTS │ │ • Video │ │ • Vibrate│ │ │ │
|
||||
│ │ │ │ • Input │ │ • Voices │ │ • AR/VR │ │ • Pattern│ │ │ │
|
||||
│ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────────────────────────┐ │ │ │
|
||||
│ │ │ │ Gesture │ │Biometric │ │ [Custom Modalities...] │ │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │ │ │
|
||||
│ │ │ │ • Motion │ │ • Emotion│ │ Extensible architecture │ │ │ │
|
||||
│ │ │ │ • Hand │ │ • Heart │ │ for future modalities │ │ │ │
|
||||
│ │ │ │ • Track │ │ • Stress │ │ │ │ │ │
|
||||
│ │ │ └──────────┘ └──────────┘ └──────────────────────────────┘ │ │ │
|
||||
│ │ └──────────────────────────────────────────────────────────────────┘ │ │
|
||||
│ │ │ │
|
||||
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
|
||||
│ │ │ Session │ │ Conversation │ │ Task │ │ │
|
||||
│ │ │ Management │ │ Integration │ │ Integration │ │ │
|
||||
│ │ │ │ │ │ │ │ │ │
|
||||
│ │ │ • User Sessions │ │ • Natural Lang │ │ • Submit Tasks │ │ │
|
||||
│ │ │ • Preferences │ │ • Context │ │ • Real-time │ │ │
|
||||
│ │ │ • Accessibility │ │ • History │ │ • Feedback │ │ │
|
||||
│ │ │ • Statistics │ │ • Style-based │ │ • Notifications │ │ │
|
||||
│ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │
|
||||
│ └────────────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
│ User Interaction
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────────────────┐
|
||||
│ END USER DEVICES │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Desktop │ │ Mobile │ │ Web │ │ Voice │ │ Wearable│ │
|
||||
│ │ App │ │ App │ │ Browser │ │ Assistant│ │ Device │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ AR/VR │ │ Smart │ │ Haptic │ │ Gesture │ │ IoT │ │
|
||||
│ │ Headset │ │ Speaker │ │ Device │ │ Camera │ │ Devices │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow Diagram
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│Administrator│
|
||||
└──────┬──────┘
|
||||
│ 1. Configure system
|
||||
│ (voices, styles, agents)
|
||||
▼
|
||||
┌────────────────────┐
|
||||
│ Admin Control Panel│
|
||||
└─────────┬──────────┘
|
||||
│ 2. Apply configuration
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ FusionAGI Core │
|
||||
│ • Orchestrator │
|
||||
│ • Agents │
|
||||
│ • Memory │
|
||||
└─────────┬────────────┘
|
||||
│ 3. Execute tasks
|
||||
│ with configured behavior
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Multi-Modal UI │
|
||||
│ • Adapts responses │
|
||||
│ • Applies styles │
|
||||
│ • Uses voices │
|
||||
└─────────┬────────────┘
|
||||
│ 4. Multi-sensory output
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ End User │
|
||||
│ • Sees (visual) │
|
||||
│ • Hears (voice) │
|
||||
│ • Feels (haptic) │
|
||||
│ • Gestures │
|
||||
└─────────┬────────────┘
|
||||
│ 5. Multi-modal input
|
||||
│
|
||||
└──────────────┐
|
||||
│
|
||||
┌──────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Multi-Modal UI │
|
||||
│ • Receives input │
|
||||
│ • Interprets intent │
|
||||
│ • Routes to core │
|
||||
└─────────┬────────────┘
|
||||
│ 6. Process request
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ FusionAGI Core │
|
||||
│ • Execute task │
|
||||
│ • Apply governance │
|
||||
│ • Learn & improve │
|
||||
└─────────┬────────────┘
|
||||
│ 7. Results & feedback
|
||||
│
|
||||
└──────► (Loop continues)
|
||||
```
|
||||
|
||||
## Component Interaction Flow
|
||||
|
||||
```
|
||||
Admin Panel Core System Multi-Modal UI
|
||||
───────────── ────────────── ───────────────
|
||||
|
||||
Configure Voice ──► Store in Use for TTS
|
||||
Profiles VoiceLibrary ──► when responding
|
||||
|
||||
Set Conversation ──► Apply to Adjust tone,
|
||||
Styles ConversationMgr ──► formality, depth
|
||||
|
||||
Configure Agents ──► Update Agent Affects task
|
||||
Behavior ──► execution style
|
||||
|
||||
Monitor System ◄── Report Status Track user
|
||||
& Metrics ◄── interactions
|
||||
|
||||
Audit Actions ◄── Log Events Log user
|
||||
& Changes ◄── activities
|
||||
```
|
||||
|
||||
## Modality Selection Flow
|
||||
|
||||
```
|
||||
User Input Arrives
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Which modality? │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌────┴────┬────────┬────────┬────────┬────────┐
|
||||
│ │ │ │ │ │
|
||||
▼ ▼ ▼ ▼ ▼ ▼
|
||||
Text Voice Visual Haptic Gesture Biometric
|
||||
│ │ │ │ │ │
|
||||
└────┬────┴────────┴────────┴────────┴────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Convert to standard │
|
||||
│ InterfaceMessage │
|
||||
└─────────┬────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Route to Core System │
|
||||
└─────────┬────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Process & Generate │
|
||||
│ Response │
|
||||
└─────────┬────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Adapt for active │
|
||||
│ modalities │
|
||||
└─────────┬────────────┘
|
||||
│
|
||||
┌─────┴─────┬────────┬────────┬────────┐
|
||||
│ │ │ │ │
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
Text Voice Visual Haptic [Others]
|
||||
│ │ │ │ │
|
||||
└───────────┴────────┴────────┴────────┘
|
||||
│
|
||||
▼
|
||||
User Receives
|
||||
Multi-Sensory Output
|
||||
```
|
||||
|
||||
## Extension Points
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────┐
|
||||
│ How to Add New Modality │
|
||||
│ │
|
||||
│ 1. Create Interface Adapter │
|
||||
│ class MyModalityInterface(InterfaceAdapter): │
|
||||
│ def capabilities(self): ... │
|
||||
│ async def send(self, message): ... │
|
||||
│ async def receive(self, timeout): ... │
|
||||
│ │
|
||||
│ 2. Register with UI │
|
||||
│ ui.register_interface(ModalityType.MY_TYPE, │
|
||||
│ MyModalityInterface()) │
|
||||
│ │
|
||||
│ 3. Enable for Sessions │
|
||||
│ ui.enable_modality(session_id, │
|
||||
│ ModalityType.MY_TYPE) │
|
||||
│ │
|
||||
│ 4. Content Adaptation (Optional) │
|
||||
│ Override _adapt_content() in MultiModalUI │
|
||||
│ │
|
||||
│ 5. Provider Integration (If needed) │
|
||||
│ Add provider-specific code in adapter │
|
||||
│ Configure via admin panel │
|
||||
└────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Security & Governance Flow
|
||||
|
||||
```
|
||||
User Request
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Multi-Modal UI │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Orchestrator │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Governance │
|
||||
│ • Guardrails │
|
||||
│ • Rate Limit │
|
||||
│ • Access Ctrl │
|
||||
│ • MAA Gate │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌────┴────┐
|
||||
│ Allowed?│
|
||||
└────┬────┘
|
||||
│
|
||||
┌────┴────┐
|
||||
│ Yes │ No ──► Reject & Log
|
||||
└────┬────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Execute Task │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Audit Log │
|
||||
│ • Action │
|
||||
│ • User │
|
||||
│ • Result │
|
||||
│ • Timestamp │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
This architecture provides a complete, extensible foundation for building rich, multi-sensory user experiences with comprehensive administrative control.
|
||||
431
docs/interfaces.md
Normal file
431
docs/interfaces.md
Normal file
@@ -0,0 +1,431 @@
|
||||
# FusionAGI Interface Layer
|
||||
|
||||
Complete multi-modal interface system for admin control and user interaction.
|
||||
|
||||
## Overview
|
||||
|
||||
FusionAGI now provides two comprehensive interface layers:
|
||||
|
||||
1. **Admin Control Panel** - System management and configuration
|
||||
2. **Multi-Modal User Interface** - Full sensory user experience
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph admin [Admin Control Panel]
|
||||
Voice[Voice Library]
|
||||
Conv[Conversation Tuning]
|
||||
Agent[Agent Config]
|
||||
Monitor[System Monitoring]
|
||||
Gov[Governance / MAA]
|
||||
end
|
||||
|
||||
subgraph core [FusionAGI Core]
|
||||
Orch[Orchestrator]
|
||||
Mem[Memory]
|
||||
Tools[Tools]
|
||||
end
|
||||
|
||||
subgraph ui [Multi-Modal User Interface]
|
||||
Text[Text]
|
||||
VoiceUI[Voice]
|
||||
Visual[Visual]
|
||||
Haptic[Haptic]
|
||||
Session[Session Mgmt]
|
||||
Task[Task Integration]
|
||||
end
|
||||
|
||||
admin --> Orch
|
||||
Orch --> Mem
|
||||
Orch --> Tools
|
||||
ui --> Orch
|
||||
Session --> Task
|
||||
```
|
||||
|
||||
## Admin Control Panel
|
||||
|
||||
Administrative interface for managing all aspects of FusionAGI.
|
||||
|
||||
### Features
|
||||
|
||||
- **Voice Library Management**: Add, configure, and organize TTS voice profiles
|
||||
- **Conversation Tuning**: Configure natural language styles and personalities
|
||||
- **Agent Configuration**: Manage agent settings, permissions, and behavior
|
||||
- **System Monitoring**: Real-time health metrics and performance tracking
|
||||
- **Governance**: Policy management and audit log access
|
||||
- **Manufacturing Authority**: MAA configuration and oversight
|
||||
|
||||
### Usage
|
||||
|
||||
`AdminControlPanel` accepts optional `voice_library` and `conversation_tuner` (default `None`); when omitted, internal defaults are created.
|
||||
|
||||
```python
|
||||
from fusionagi import Orchestrator, EventBus, StateManager
|
||||
from fusionagi.interfaces import AdminControlPanel, VoiceLibrary, ConversationTuner
|
||||
from fusionagi.governance import PolicyEngine, AuditLog
|
||||
|
||||
# Initialize core components
|
||||
bus = EventBus()
|
||||
state = StateManager()
|
||||
orch = Orchestrator(event_bus=bus, state_manager=state)
|
||||
|
||||
# Create admin panel
|
||||
admin = AdminControlPanel(
|
||||
orchestrator=orch,
|
||||
event_bus=bus,
|
||||
state_manager=state,
|
||||
voice_library=VoiceLibrary(),
|
||||
conversation_tuner=ConversationTuner(),
|
||||
)
|
||||
|
||||
# Add voice profiles
|
||||
from fusionagi.interfaces.voice import VoiceProfile
|
||||
|
||||
voice = VoiceProfile(
|
||||
name="Professional Assistant",
|
||||
language="en-US",
|
||||
gender="neutral",
|
||||
style="professional",
|
||||
pitch=1.0,
|
||||
speed=1.0,
|
||||
)
|
||||
admin.add_voice_profile(voice)
|
||||
|
||||
# Configure conversation styles
|
||||
from fusionagi.interfaces.conversation import ConversationStyle
|
||||
|
||||
style = ConversationStyle(
|
||||
formality="neutral",
|
||||
verbosity="balanced",
|
||||
empathy_level=0.8,
|
||||
technical_depth=0.6,
|
||||
)
|
||||
admin.register_conversation_style("technical_support", style)
|
||||
|
||||
# Monitor system
|
||||
status = admin.get_system_status()
|
||||
print(f"Status: {status.status}, Active tasks: {status.active_tasks}")
|
||||
|
||||
# Export configuration
|
||||
config = admin.export_configuration()
|
||||
```
|
||||
|
||||
## Multi-Modal User Interface
|
||||
|
||||
Unified interface supporting multiple sensory modalities simultaneously.
|
||||
|
||||
### Supported Modalities
|
||||
|
||||
- **Text**: Chat, commands, structured input
|
||||
- **Voice**: Speech-to-text, text-to-speech
|
||||
- **Visual**: Images, video, AR/VR (extensible)
|
||||
- **Haptic**: Touch feedback, vibration patterns (extensible)
|
||||
- **Gesture**: Motion control, hand tracking (extensible)
|
||||
- **Biometric**: Emotion detection, physiological signals (extensible)
|
||||
|
||||
### Features
|
||||
|
||||
- Seamless modality switching
|
||||
- Simultaneous multi-modal I/O
|
||||
- Accessibility support
|
||||
- Context-aware modality selection
|
||||
- Real-time feedback across all active modalities
|
||||
|
||||
### Usage
|
||||
|
||||
```python
|
||||
from fusionagi.interfaces import MultiModalUI, VoiceInterface, ConversationManager
|
||||
from fusionagi.interfaces.base import ModalityType
|
||||
|
||||
# Initialize components
|
||||
voice = VoiceInterface(stt_provider="whisper", tts_provider="elevenlabs")
|
||||
conv_manager = ConversationManager()
|
||||
|
||||
# Create multi-modal UI
|
||||
ui = MultiModalUI(
|
||||
orchestrator=orch,
|
||||
conversation_manager=conv_manager,
|
||||
voice_interface=voice,
|
||||
)
|
||||
|
||||
# Create user session with preferred modalities
|
||||
session_id = ui.create_session(
|
||||
user_id="user123",
|
||||
preferred_modalities=[ModalityType.TEXT, ModalityType.VOICE],
|
||||
accessibility_settings={"screen_reader": True},
|
||||
)
|
||||
|
||||
# Send multi-modal output
|
||||
await ui.send_to_user(
|
||||
session_id,
|
||||
"Hello! How can I help you today?",
|
||||
modalities=[ModalityType.TEXT, ModalityType.VOICE],
|
||||
)
|
||||
|
||||
# Receive user input (any active modality)
|
||||
message = await ui.receive_from_user(session_id, timeout_seconds=30.0)
|
||||
|
||||
# Submit task with interactive feedback
|
||||
task_id = await ui.submit_task_interactive(
|
||||
session_id,
|
||||
goal="Analyze sales data and create report",
|
||||
)
|
||||
|
||||
# Conversational interaction
|
||||
response = await ui.converse(session_id, "What's the status of my task?")
|
||||
```
|
||||
|
||||
## Voice Interface
|
||||
|
||||
Speech interaction with configurable voice profiles.
|
||||
|
||||
### Voice Library
|
||||
|
||||
```python
|
||||
from fusionagi.interfaces import VoiceLibrary, VoiceProfile
|
||||
|
||||
library = VoiceLibrary()
|
||||
|
||||
# Add multiple voices
|
||||
voices = [
|
||||
VoiceProfile(
|
||||
name="Friendly Assistant",
|
||||
language="en-US",
|
||||
gender="female",
|
||||
style="friendly",
|
||||
pitch=1.1,
|
||||
speed=1.0,
|
||||
),
|
||||
VoiceProfile(
|
||||
name="Technical Expert",
|
||||
language="en-US",
|
||||
gender="male",
|
||||
style="professional",
|
||||
pitch=0.9,
|
||||
speed=0.95,
|
||||
),
|
||||
VoiceProfile(
|
||||
name="Multilingual Guide",
|
||||
language="es-ES",
|
||||
gender="neutral",
|
||||
style="calm",
|
||||
),
|
||||
]
|
||||
|
||||
for voice in voices:
|
||||
library.add_voice(voice)
|
||||
|
||||
# Set default
|
||||
library.set_default_voice(voices[0].id)
|
||||
|
||||
# Filter voices
|
||||
spanish_voices = library.list_voices(language="es-ES")
|
||||
female_voices = library.list_voices(gender="female")
|
||||
```
|
||||
|
||||
### Speech-to-Text Providers
|
||||
|
||||
Supported STT providers (extensible):
|
||||
- **Whisper**: OpenAI Whisper (local or API)
|
||||
- **Azure**: Azure Cognitive Services
|
||||
- **Google**: Google Cloud Speech-to-Text
|
||||
- **Deepgram**: Deepgram API
|
||||
|
||||
### Text-to-Speech Providers
|
||||
|
||||
Supported TTS providers (extensible):
|
||||
- **System**: OS-native TTS (pyttsx3)
|
||||
- **ElevenLabs**: ElevenLabs API
|
||||
- **Azure**: Azure Cognitive Services
|
||||
- **Google**: Google Cloud TTS
|
||||
|
||||
## Conversation Management
|
||||
|
||||
Natural language conversation with tunable styles.
|
||||
|
||||
### Conversation Styles
|
||||
|
||||
```python
|
||||
from fusionagi.interfaces import ConversationTuner, ConversationStyle
|
||||
|
||||
tuner = ConversationTuner()
|
||||
|
||||
# Define conversation styles
|
||||
styles = {
|
||||
"customer_support": ConversationStyle(
|
||||
formality="neutral",
|
||||
verbosity="balanced",
|
||||
empathy_level=0.9,
|
||||
proactivity=0.8,
|
||||
technical_depth=0.4,
|
||||
),
|
||||
"technical_expert": ConversationStyle(
|
||||
formality="formal",
|
||||
verbosity="detailed",
|
||||
empathy_level=0.5,
|
||||
technical_depth=0.9,
|
||||
humor_level=0.1,
|
||||
),
|
||||
"casual_friend": ConversationStyle(
|
||||
formality="casual",
|
||||
verbosity="balanced",
|
||||
empathy_level=0.8,
|
||||
humor_level=0.7,
|
||||
technical_depth=0.3,
|
||||
),
|
||||
}
|
||||
|
||||
for name, style in styles.items():
|
||||
tuner.register_style(name, style)
|
||||
|
||||
# Tune for specific context
|
||||
tuned_style = tuner.tune_for_context(
|
||||
domain="technical",
|
||||
user_preferences={"verbosity": "concise"},
|
||||
)
|
||||
```
|
||||
|
||||
### Conversation Sessions
|
||||
|
||||
```python
|
||||
from fusionagi.interfaces import ConversationManager, ConversationTurn
|
||||
|
||||
manager = ConversationManager(tuner=tuner)
|
||||
|
||||
# Create session
|
||||
session_id = manager.create_session(
|
||||
user_id="user123",
|
||||
style_name="customer_support",
|
||||
language="en",
|
||||
domain="technical_support",
|
||||
)
|
||||
|
||||
# Add conversation turns
|
||||
manager.add_turn(ConversationTurn(
|
||||
session_id=session_id,
|
||||
speaker="user",
|
||||
content="My system is not responding",
|
||||
sentiment=-0.3,
|
||||
))
|
||||
|
||||
manager.add_turn(ConversationTurn(
|
||||
session_id=session_id,
|
||||
speaker="agent",
|
||||
content="I understand that's frustrating. Let me help you troubleshoot.",
|
||||
sentiment=0.5,
|
||||
))
|
||||
|
||||
# Get conversation history
|
||||
history = manager.get_history(session_id, limit=10)
|
||||
|
||||
# Get context for LLM
|
||||
context = manager.get_context_summary(session_id)
|
||||
```
|
||||
|
||||
## Extending with New Modalities
|
||||
|
||||
To add a new sensory modality:
|
||||
|
||||
1. **Create Interface Adapter**:
|
||||
|
||||
```python
|
||||
from fusionagi.interfaces.base import InterfaceAdapter, InterfaceCapabilities, InterfaceMessage, ModalityType
|
||||
|
||||
class HapticInterface(InterfaceAdapter):
|
||||
def __init__(self):
|
||||
super().__init__("haptic")
|
||||
|
||||
def capabilities(self) -> InterfaceCapabilities:
|
||||
return InterfaceCapabilities(
|
||||
supported_modalities=[ModalityType.HAPTIC],
|
||||
supports_streaming=True,
|
||||
supports_interruption=True,
|
||||
)
|
||||
|
||||
async def send(self, message: InterfaceMessage) -> None:
|
||||
# Send haptic feedback (vibration pattern, etc.)
|
||||
pattern = message.content
|
||||
await self._send_haptic_pattern(pattern)
|
||||
|
||||
async def receive(self, timeout_seconds: float | None = None) -> InterfaceMessage | None:
|
||||
# Receive haptic input (touch, pressure, etc.)
|
||||
data = await self._read_haptic_sensor(timeout_seconds)
|
||||
return InterfaceMessage(
|
||||
id=f"haptic_{uuid.uuid4().hex[:8]}",
|
||||
modality=ModalityType.HAPTIC,
|
||||
content=data,
|
||||
)
|
||||
```
|
||||
|
||||
2. **Register with UI**:
|
||||
|
||||
```python
|
||||
haptic = HapticInterface()
|
||||
ui.register_interface(ModalityType.HAPTIC, haptic)
|
||||
```
|
||||
|
||||
3. **Enable for Session**:
|
||||
|
||||
```python
|
||||
ui.enable_modality(session_id, ModalityType.HAPTIC)
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Admin Control Panel │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ Voice Library│ │Conversation │ │ Agent Config │ │
|
||||
│ │ Management │ │ Tuning │ │ │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
||||
│ │ System │ │ Governance │ │ MAA │ │
|
||||
│ │ Monitoring │ │ & Audit │ │ Control │ │
|
||||
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ FusionAGI Core System │
|
||||
│ (Orchestrator, Agents, Memory, Tools) │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Multi-Modal User Interface │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Text │ │ Voice │ │ Visual │ │ Haptic │ │
|
||||
│ │Interface │ │Interface │ │Interface │ │Interface │ │
|
||||
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
|
||||
│ ┌──────────┐ ┌──────────┐ │
|
||||
│ │ Gesture │ │Biometric │ │
|
||||
│ │Interface │ │Interface │ │
|
||||
│ └──────────┘ └──────────┘ │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Integration with FusionAGI Core
|
||||
|
||||
The interface layer integrates seamlessly with FusionAGI's core components:
|
||||
|
||||
- **Orchestrator**: Task submission and monitoring
|
||||
- **Event Bus**: Real-time updates and notifications
|
||||
- **Agents**: Direct agent interaction and configuration
|
||||
- **Memory**: Conversation history and user preferences
|
||||
- **Governance**: Policy enforcement and audit logging
|
||||
- **MAA**: Manufacturing authority oversight
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Implement STT/TTS Providers**: Integrate with actual speech services
|
||||
2. **Build Web UI**: Create web-based admin panel and user interface
|
||||
3. **Add Visual Modality**: Support images, video, AR/VR
|
||||
4. **Implement Haptic**: Add haptic feedback support
|
||||
5. **Gesture Recognition**: Integrate motion tracking
|
||||
6. **Biometric Sensors**: Add emotion and physiological monitoring
|
||||
7. **Mobile Apps**: Native iOS/Android interfaces
|
||||
8. **Accessibility**: Enhanced screen reader and assistive technology support
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
86
docs/maa_activation.md
Normal file
86
docs/maa_activation.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# MAA (Manufacturing Authority Add-On) Activation
|
||||
|
||||
## Overview
|
||||
|
||||
MAA is a sovereign validation layer. No physical-world manufacturing execution is permitted without MAA approval and a valid Manufacturing Proof Certificate (MPC).
|
||||
|
||||
## Activation Flow
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph setup [Setup]
|
||||
MPC[MPCAuthority]
|
||||
Gate[MAAGate]
|
||||
G[Guardrails]
|
||||
E[ExecutorAgent]
|
||||
end
|
||||
|
||||
subgraph runtime [Runtime]
|
||||
Issue[Issue MPC]
|
||||
Verify[Gate.verify]
|
||||
Tool[Manufacturing Tool]
|
||||
end
|
||||
|
||||
MPC --> Gate
|
||||
Gate --> G
|
||||
G --> E
|
||||
Issue --> Verify
|
||||
Verify --> Tool
|
||||
```
|
||||
|
||||
**Flow:** Create MPCAuthority and MAAGate → register Gate with Guardrails → pass Guardrails to Executor. At runtime: issue MPC → Gate verifies before tool execution.
|
||||
|
||||
## Activation Steps
|
||||
|
||||
1. **MAA is built-in.** No extra install is required; use the `fusionagi.maa` package directly.
|
||||
|
||||
2. **Create MPC Authority and MAA Gate**:
|
||||
```python
|
||||
from fusionagi.maa import MAAGate
|
||||
from fusionagi.maa.layers import MPCAuthority
|
||||
|
||||
mpc_authority = MPCAuthority()
|
||||
maa_gate = MAAGate(mpc_authority=mpc_authority)
|
||||
```
|
||||
|
||||
3. **Register MAA Gate with Guardrails**:
|
||||
```python
|
||||
from fusionagi.governance import Guardrails
|
||||
|
||||
guardrails = Guardrails()
|
||||
guardrails.add_check(maa_gate.check)
|
||||
```
|
||||
|
||||
4. **Pass Guardrails to Executor**:
|
||||
```python
|
||||
from fusionagi.agents import ExecutorAgent
|
||||
from fusionagi.tools import ToolRegistry
|
||||
|
||||
registry = ToolRegistry()
|
||||
executor = ExecutorAgent(registry=registry, state_manager=state, guardrails=guardrails)
|
||||
```
|
||||
|
||||
5. **Issue an MPC** before calling manufacturing tools:
|
||||
```python
|
||||
cert = mpc_authority.issue("design-001", decision_lineage=[...])
|
||||
# Tool args must include mpc_id=cert.mpc_id.value (or mpc_id_value)
|
||||
```
|
||||
|
||||
6. **Register manufacturing tools** (e.g. from MAA):
|
||||
```python
|
||||
from fusionagi.maa.tools import cnc_emit_tool, am_slice_tool
|
||||
registry.register(cnc_emit_tool())
|
||||
registry.register(am_slice_tool())
|
||||
```
|
||||
|
||||
## MPC Lifecycle
|
||||
|
||||
- **Issue**: `MPCAuthority.issue(mpc_id_value, ...)` creates an immutable, versioned certificate.
|
||||
- **Verify**: The Gate calls `mpc_authority.verify(mpc_id)` before allowing manufacturing tools.
|
||||
- **Versioning**: Changes require re-certification; historical MPCs are read-only.
|
||||
|
||||
## Deployment
|
||||
|
||||
- MAA runs in-process; no cloud dependency.
|
||||
- All state (DLTs, MPCs, machine profiles) can be file-backed or DB-backed for on-prem/air-gapped use.
|
||||
- GPU is optional (for future simulation/geometry backends).
|
||||
38
docs/maa_compliance_mapping.md
Normal file
38
docs/maa_compliance_mapping.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# MAA Compliance Mapping (High-Level)
|
||||
|
||||
MAA is designed to align with the following standards where applicable. This document is a high-level mapping only; full compliance requires organization-specific controls and audits.
|
||||
|
||||
## Standards ↔ MAA Alignment
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph standards [Standards]
|
||||
ASME[ASME Y14]
|
||||
ISO[ISO 9001/13485]
|
||||
MIL[MIL-STD-810]
|
||||
DoD[DoD Digital Thread]
|
||||
end
|
||||
|
||||
subgraph maa [MAA Controls]
|
||||
Gap[Gap Detection]
|
||||
Lineage[Decision Lineage]
|
||||
MPC[MPC / Versioning]
|
||||
DLT[DLT Lineage]
|
||||
end
|
||||
|
||||
ASME --> Gap
|
||||
ISO --> Lineage
|
||||
ISO --> MPC
|
||||
MIL --> Lineage
|
||||
DoD --> MPC
|
||||
DoD --> DLT
|
||||
```
|
||||
|
||||
| Standard / Requirement | MAA Alignment |
|
||||
|------------------------|---------------|
|
||||
| **ASME Y14** | Dimensioning, tolerancing, and datum practices can be enforced via gap detection (undefined datums, implicit tolerances) and intent/geometry lineage. |
|
||||
| **ISO 9001 / 13485** | Traceability (decision lineage, MPC), versioning, and audit export support quality management and medical device traceability. |
|
||||
| **MIL-STD-810** | Environmental bounding and load cases in Intent Engine; physics/simulation authority can reference environmental proof. |
|
||||
| **DoD Digital Thread** | MPC and DLT lineage provide a digital thread from requirement to toolpath; immutable, versioned artifacts support audit. |
|
||||
|
||||
MAA does not replace formal certification; it provides technical controls (no manufacture without MPC, gap halt, lineage) that support compliance efforts.
|
||||
173
docs/multi_agent_acceleration.md
Normal file
173
docs/multi_agent_acceleration.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# Multi-Agent Acceleration
|
||||
|
||||
FusionAGI includes multi-agent accelerations, enhancements, optimizations, and scaling capabilities.
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph supervisor [Supervisor Agent]
|
||||
S[Supervisor]
|
||||
end
|
||||
|
||||
subgraph agents [Agents]
|
||||
P[Planner]
|
||||
R[Reasoner]
|
||||
EP[Executor Pool]
|
||||
end
|
||||
|
||||
subgraph pool [Executor Pool]
|
||||
E1[Executor 1]
|
||||
E2[Executor 2]
|
||||
E3[Executor 3]
|
||||
end
|
||||
|
||||
S --> P
|
||||
S --> EP
|
||||
S --> R
|
||||
EP --> E1
|
||||
EP --> E2
|
||||
EP --> E3
|
||||
```
|
||||
|
||||
**Parallel step execution:** Independent plan steps run concurrently. **Agent pool:** Multiple executors behind one logical endpoint (round_robin, least_busy, random). **Supervisor:** Drives plan–execute loop with parallel dispatch.
|
||||
|
||||
## Features
|
||||
|
||||
### 1. Parallel Step Execution
|
||||
|
||||
Independent plan steps (those with satisfied dependencies and no mutual dependencies) run **concurrently**.
|
||||
|
||||
```python
|
||||
from fusionagi.planning import ready_steps
|
||||
from fusionagi.multi_agent import execute_steps_parallel, execute_steps_parallel_wave
|
||||
|
||||
# Get all steps ready to run in parallel
|
||||
ready = ready_steps(plan, completed_step_ids={"s1", "s2"})
|
||||
|
||||
# Execute them concurrently
|
||||
results = execute_steps_parallel(execute_fn, task_id, plan, completed_step_ids, max_workers=4)
|
||||
|
||||
# Or run full plan in waves (each wave = parallel batch)
|
||||
results = execute_steps_parallel_wave(execute_fn, task_id, plan, max_workers=4)
|
||||
```
|
||||
|
||||
### 2. Agent Pool & Load Balancing
|
||||
|
||||
Scale horizontally with multiple executors behind a single logical endpoint.
|
||||
|
||||
```python
|
||||
from fusionagi.agents import ExecutorAgent
|
||||
from fusionagi.multi_agent import PooledExecutorRouter, AgentPool
|
||||
from fusionagi.core import StateManager
|
||||
from fusionagi.tools import ToolRegistry
|
||||
|
||||
# Create pool router (register as "executor")
|
||||
registry = ToolRegistry()
|
||||
state = StateManager()
|
||||
pool = PooledExecutorRouter(identity="executor", pool=AgentPool(strategy="least_busy"))
|
||||
|
||||
# Add multiple executors
|
||||
for i in range(4):
|
||||
ex = ExecutorAgent(identity=f"executor_{i}", registry=registry, state_manager=state)
|
||||
pool.add_executor(f"executor_{i}", ex)
|
||||
|
||||
orch.register_agent("executor", pool)
|
||||
# All execute_step messages now load-balance across 4 executors
|
||||
```
|
||||
|
||||
**Strategies:** `round_robin`, `least_busy`, `random`
|
||||
|
||||
### 3. Sub-Task Delegation (Fan-Out / Fan-In)
|
||||
|
||||
Decompose tasks and delegate to specialized sub-agents in parallel.
|
||||
|
||||
```python
|
||||
from fusionagi.multi_agent import delegate_sub_tasks, SubTask, DelegationConfig
|
||||
|
||||
sub_tasks = [
|
||||
SubTask("st1", "Analyze requirements"),
|
||||
SubTask("st2", "Design solution"),
|
||||
SubTask("st3", "Validate constraints"),
|
||||
]
|
||||
|
||||
results = delegate_sub_tasks(
|
||||
sub_tasks,
|
||||
delegate_fn=lambda st: run_sub_agent(st),
|
||||
config=DelegationConfig(max_parallel=3, fail_fast=False),
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Supervisor Agent
|
||||
|
||||
Single agent that drives the full plan-execute loop with **parallel dispatch**.
|
||||
|
||||
```python
|
||||
from fusionagi.multi_agent import SupervisorAgent
|
||||
from fusionagi.core import Orchestrator, EventBus, StateManager
|
||||
|
||||
orch = Orchestrator(EventBus(), StateManager())
|
||||
supervisor = SupervisorAgent(
|
||||
identity="supervisor",
|
||||
orchestrator=orch,
|
||||
planner_id="planner",
|
||||
executor_id="executor",
|
||||
parallel_mode=True, # Enable parallel step execution
|
||||
max_parallel_workers=4,
|
||||
)
|
||||
|
||||
orch.register_agent("supervisor", supervisor)
|
||||
# Send run_task → supervisor gets plan → executes steps in parallel waves
|
||||
```
|
||||
|
||||
### 5. Batch Message Routing
|
||||
|
||||
Route multiple messages in parallel.
|
||||
|
||||
```python
|
||||
responses = orch.route_messages_batch([env1, env2, env3, ...])
|
||||
```
|
||||
|
||||
### 6. Async Message Routing
|
||||
|
||||
Non-blocking dispatch with optional callback.
|
||||
|
||||
```python
|
||||
future = orch.route_message_async(envelope, callback=lambda resp: handle(resp))
|
||||
# Or await: result = future.result()
|
||||
```
|
||||
|
||||
## Scaling Checklist
|
||||
|
||||
| Capability | Module | Use Case |
|
||||
|------------|--------|----------|
|
||||
| Parallel step execution | `multi_agent.parallel` | Plans with independent steps |
|
||||
| Agent pool | `multi_agent.pool` | Horizontal executor scaling |
|
||||
| Sub-task delegation | `multi_agent.delegation` | Hierarchical task decomposition |
|
||||
| Supervisor | `multi_agent.supervisor` | Automated parallel orchestration |
|
||||
| Batch routing | `Orchestrator.route_messages_batch` | Multi-task throughput |
|
||||
| Async routing | `Orchestrator.route_message_async` | Non-blocking pipelines |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ Supervisor │
|
||||
│ (parallel_mode) │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌───────────────────┼───────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌─────────────┐ ┌──────────┐
|
||||
│ Planner │ │ Executor │ │ Reasoner │
|
||||
│ │ │ Pool │ │ │
|
||||
└──────────┘ └──────┬──────┘ └──────────┘
|
||||
│
|
||||
┌──────────────┼──────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│Exec 1 │ │Exec 2 │ │Exec 3 │ (parallel steps)
|
||||
└──────────┘ └──────────┘ └──────────┘
|
||||
```
|
||||
166
docs/openai_bridge.md
Normal file
166
docs/openai_bridge.md
Normal file
@@ -0,0 +1,166 @@
|
||||
# OpenAI Bridge for Cursor Composer
|
||||
|
||||
The FusionAGI OpenAI bridge exposes an OpenAI-compatible API so you can use FusionAGI's Dvādaśa multi-agent system as a custom model in Cursor Composer and other OpenAI API consumers.
|
||||
|
||||
## Request Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Cursor as Cursor Composer
|
||||
participant API as FusionAGI API
|
||||
participant Orch as Orchestrator
|
||||
participant Heads as Dvādaśa Heads
|
||||
participant Witness as Witness
|
||||
|
||||
Cursor->>API: POST /v1/chat/completions (messages)
|
||||
API->>Orch: submit_task(goal from messages)
|
||||
Orch->>Heads: run heads in parallel
|
||||
Heads->>Witness: head outputs
|
||||
Witness->>Orch: final_answer (synthesis)
|
||||
Orch->>API: FinalResponse
|
||||
API->>Cursor: OpenAI-format response (sync or SSE)
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
- **`GET /v1/models`** — Lists available models (returns `fusionagi-dvadasa`)
|
||||
- **`POST /v1/chat/completions`** — Chat completions (sync and streaming)
|
||||
|
||||
The bridge translates OpenAI-style requests into FusionAGI tasks, runs the Dvādaśa heads and Witness, and returns responses in OpenAI format.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Start the FusionAGI API
|
||||
|
||||
```bash
|
||||
# Install with API support
|
||||
pip install -e ".[api]"
|
||||
|
||||
# Start the server (use a real LLM adapter for production)
|
||||
uvicorn fusionagi.api.app:app --reload
|
||||
```
|
||||
|
||||
The API runs at `http://localhost:8000` by default.
|
||||
|
||||
### 2. Configure Cursor Composer
|
||||
|
||||
1. Open **Cursor Settings** → **Models** (or equivalent for your Cursor version).
|
||||
2. Click **Add Custom Model** or configure an OpenAI-compatible endpoint.
|
||||
3. Set:
|
||||
- **Name**: FusionAGI Dvādaśa
|
||||
- **Base URL**: `http://localhost:8000/v1`
|
||||
- **Model ID**: `fusionagi-dvadasa`
|
||||
- **API Key**: Any string (e.g. `sk-fusionagi`) when auth is disabled
|
||||
- **Context Length**: 128000 (or your preferred value)
|
||||
- **Temperature**: 0.7
|
||||
|
||||
4. Select FusionAGI as the model for Composer.
|
||||
|
||||
### 3. Use with Composer
|
||||
|
||||
Composer will send requests to FusionAGI. Each prompt is processed by the Dvādaśa heads (Logic, Research, Strategy, Security, Safety, etc.) and synthesized by the Witness into a single response.
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `OPENAI_BRIDGE_MODEL_ID` | `fusionagi-dvadasa` | Model ID returned by `/v1/models` |
|
||||
| `OPENAI_BRIDGE_AUTH` | `disabled` | Set to `Bearer` to enable API key auth |
|
||||
| `OPENAI_BRIDGE_API_KEY` | (none) | Required when auth is enabled |
|
||||
| `OPENAI_BRIDGE_TIMEOUT_PER_HEAD` | `60` | Max seconds per head agent |
|
||||
|
||||
### Authentication
|
||||
|
||||
**Development (default):** Auth is disabled. Any request is accepted.
|
||||
|
||||
**Production:** Enable Bearer auth:
|
||||
|
||||
```bash
|
||||
export OPENAI_BRIDGE_AUTH=Bearer
|
||||
export OPENAI_BRIDGE_API_KEY=your-secret-key
|
||||
```
|
||||
|
||||
Cursor must then send `Authorization: Bearer your-secret-key` with each request.
|
||||
|
||||
## API Details
|
||||
|
||||
### Chat Completions
|
||||
|
||||
**Request (sync):**
|
||||
```json
|
||||
{
|
||||
"model": "fusionagi-dvadasa",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are helpful."},
|
||||
{"role": "user", "content": "What is the best architecture for X?"}
|
||||
],
|
||||
"stream": false
|
||||
}
|
||||
```
|
||||
|
||||
**Request (streaming):**
|
||||
```json
|
||||
{
|
||||
"model": "fusionagi-dvadasa",
|
||||
"messages": [{"role": "user", "content": "Explain Y"}],
|
||||
"stream": true
|
||||
}
|
||||
```
|
||||
|
||||
**Response (sync):** Standard OpenAI chat completion object with `choices[0].message.content` and `usage`.
|
||||
|
||||
**Response (streaming):** Server-Sent Events (SSE) with `data:` lines containing OpenAI-format chunks, ending with `data: [DONE]`.
|
||||
|
||||
### Message Translation
|
||||
|
||||
The bridge converts OpenAI messages to a single prompt for Dvādaśa:
|
||||
|
||||
- **System messages** — Prepended as context
|
||||
- **User/assistant turns** — Formatted as `[User]:` / `[Assistant]:` conversation
|
||||
- **Tool results** — Included as `[Tool name] returned: ...`
|
||||
|
||||
### Models
|
||||
|
||||
**Request:**
|
||||
```
|
||||
GET /v1/models
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"object": "list",
|
||||
"data": [
|
||||
{
|
||||
"id": "fusionagi-dvadasa",
|
||||
"object": "model",
|
||||
"created": 1704067200,
|
||||
"owned_by": "fusionagi"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Tool calling:** Not supported in v1. The bridge returns text-only responses. Cursor workflows that expect `tool_calls` in the model response will not work. MCP tools provided by Cursor may still be used based on model output text.
|
||||
- **Token counts:** Usage is estimated from character counts (chars/4 heuristic). Exact token counts require a tokenizer.
|
||||
- **Vision/multimodal:** Image content in messages is not supported.
|
||||
- **Latency:** Dvādaśa runs multiple heads in parallel plus Witness synthesis, which can be slower than a single LLM call.
|
||||
|
||||
## Example: curl
|
||||
|
||||
```bash
|
||||
# List models
|
||||
curl -X GET http://localhost:8000/v1/models
|
||||
|
||||
# Chat completion (sync)
|
||||
curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "fusionagi-dvadasa",
|
||||
"messages": [{"role": "user", "content": "What is 2+2?"}]
|
||||
}'
|
||||
```
|
||||
422
docs/ui_ux_implementation.md
Normal file
422
docs/ui_ux_implementation.md
Normal file
@@ -0,0 +1,422 @@
|
||||
# FusionAGI UI/UX Implementation Summary
|
||||
|
||||
## Overview
|
||||
|
||||
FusionAGI now includes a comprehensive interface layer that provides both administrative control and multi-sensory user interaction capabilities. This implementation addresses the need for:
|
||||
|
||||
1. **Admin Control Panel** - System management and configuration interface
|
||||
2. **Multi-Modal User Interface** - Full sensory experience for all user interactions
|
||||
|
||||
## Interface Layer at a Glance
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph foundation [Foundation]
|
||||
Base[base.py]
|
||||
Base --> Modality[ModalityType]
|
||||
Base --> Adapter[InterfaceAdapter]
|
||||
Base --> Message[InterfaceMessage]
|
||||
end
|
||||
|
||||
subgraph admin [Admin Control Panel]
|
||||
Voice[Voice Library]
|
||||
Conv[Conversation Tuning]
|
||||
Agent[Agent Config]
|
||||
Monitor[System Monitoring]
|
||||
Gov[Governance / Audit]
|
||||
end
|
||||
|
||||
subgraph ui [Multi-Modal UI]
|
||||
Session[Session Management]
|
||||
Text[Text]
|
||||
VoiceUI[Voice]
|
||||
Visual[Visual]
|
||||
Task[Task Integration]
|
||||
Converse[Conversation]
|
||||
end
|
||||
|
||||
foundation --> admin
|
||||
foundation --> ui
|
||||
Voice --> VoiceUI
|
||||
```
|
||||
|
||||
## What Was Built
|
||||
|
||||
### 1. Interface Foundation (`fusionagi/interfaces/base.py`)
|
||||
|
||||
**Core Abstractions:**
|
||||
- `InterfaceAdapter` - Abstract base for all interface implementations
|
||||
- `ModalityType` - Enum of supported sensory modalities (TEXT, VOICE, VISUAL, HAPTIC, GESTURE, BIOMETRIC)
|
||||
- `InterfaceMessage` - Standardized message format across modalities
|
||||
- `InterfaceCapabilities` - Capability declaration for each interface
|
||||
|
||||
**Key Features:**
|
||||
- Pluggable architecture for adding new modalities
|
||||
- Streaming support for real-time responses
|
||||
- Interruption handling for natural interaction
|
||||
- Multi-modal simultaneous operation
|
||||
|
||||
### 2. Voice Interface (`fusionagi/interfaces/voice.py`)
|
||||
|
||||
**Components:**
|
||||
- `VoiceLibrary` - Manage TTS voice profiles
|
||||
- `VoiceProfile` - Configurable voice characteristics (language, gender, style, pitch, speed)
|
||||
- `VoiceInterface` - Speech-to-text and text-to-speech adapter
|
||||
|
||||
**Features:**
|
||||
- Multiple voice profiles per system
|
||||
- Configurable TTS providers (ElevenLabs, Azure, Google, system)
|
||||
- Configurable STT providers (Whisper, Azure, Google, Deepgram)
|
||||
- Voice selection per session or message
|
||||
- Language support (extensible)
|
||||
|
||||
**Admin Controls:**
|
||||
- Add/remove voice profiles
|
||||
- Update voice characteristics
|
||||
- Set default voice
|
||||
- Filter voices by language, gender, style
|
||||
|
||||
### 3. Conversation Management (`fusionagi/interfaces/conversation.py`)
|
||||
|
||||
**Components:**
|
||||
- `ConversationStyle` - Personality and behavior configuration
|
||||
- `ConversationTuner` - Style management and domain-specific tuning
|
||||
- `ConversationManager` - Session and history management
|
||||
- `ConversationTurn` - Individual conversation exchanges
|
||||
|
||||
**Tunable Parameters:**
|
||||
- Formality level (casual, neutral, formal)
|
||||
- Verbosity (concise, balanced, detailed)
|
||||
- Empathy level (0.0 - 1.0)
|
||||
- Proactivity (0.0 - 1.0)
|
||||
- Humor level (0.0 - 1.0)
|
||||
- Technical depth (0.0 - 1.0)
|
||||
|
||||
**Features:**
|
||||
- Named conversation styles (e.g., "customer_support", "technical_expert")
|
||||
- Domain-specific auto-tuning
|
||||
- User preference overrides
|
||||
- Conversation history tracking
|
||||
- Context summarization for LLM prompting
|
||||
|
||||
### 4. Admin Control Panel (`fusionagi/interfaces/admin_panel.py`)
|
||||
|
||||
**Capabilities:**
|
||||
|
||||
#### Voice Management
|
||||
- Add/update/remove voice profiles
|
||||
- Set default voices
|
||||
- List and filter voices
|
||||
- Export/import voice configurations
|
||||
|
||||
#### Conversation Tuning
|
||||
- Register conversation styles
|
||||
- Configure personality parameters
|
||||
- Set default styles
|
||||
- Domain-specific presets
|
||||
|
||||
#### Agent Configuration
|
||||
- Configure agent settings
|
||||
- Enable/disable agents
|
||||
- Set concurrency limits
|
||||
- Configure retry policies
|
||||
|
||||
#### System Monitoring
|
||||
- Real-time system status
|
||||
- Task statistics by state and priority
|
||||
- Agent activity tracking
|
||||
- Performance metrics
|
||||
|
||||
#### Governance & Audit
|
||||
- Access audit logs
|
||||
- Update policies
|
||||
- Track administrative actions
|
||||
- Compliance reporting
|
||||
|
||||
#### Configuration Management
|
||||
- Export full system configuration
|
||||
- Import configuration from file
|
||||
- Version control ready
|
||||
|
||||
### 5. Multi-Modal User Interface (`fusionagi/interfaces/multimodal_ui.py`)
|
||||
|
||||
**Core Features:**
|
||||
|
||||
#### Session Management
|
||||
- Create user sessions with preferred modalities
|
||||
- Track user preferences
|
||||
- Accessibility settings support
|
||||
- Session statistics and monitoring
|
||||
|
||||
#### Modality Support
|
||||
- **Text**: Chat, commands, structured input
|
||||
- **Voice**: Speech I/O with voice profiles
|
||||
- **Visual**: Images, video, AR/VR (extensible)
|
||||
- **Haptic**: Touch feedback (extensible)
|
||||
- **Gesture**: Motion control (extensible)
|
||||
- **Biometric**: Emotion detection (extensible)
|
||||
|
||||
#### Multi-Modal I/O
|
||||
- Send messages through multiple modalities simultaneously
|
||||
- Receive input from any active modality
|
||||
- Content adaptation per modality
|
||||
- Seamless modality switching
|
||||
|
||||
#### Task Integration
|
||||
- Interactive task submission
|
||||
- Real-time task updates across all modalities
|
||||
- Progress notifications
|
||||
- Completion feedback
|
||||
|
||||
#### Conversation Integration
|
||||
- Natural language interaction
|
||||
- Context-aware responses
|
||||
- Style-based personality
|
||||
- History tracking
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Admin Control Panel │
|
||||
│ │
|
||||
│ Voice Library Conversation Agent System │
|
||||
│ Management Tuning Config Monitoring │
|
||||
│ │
|
||||
│ Governance MAA Control Config Audit │
|
||||
│ & Policies Export/Import Log │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ FusionAGI Core System │
|
||||
│ │
|
||||
│ Orchestrator • Agents • Memory • Tools • Governance│
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Multi-Modal User Interface │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────┐ │
|
||||
│ │ Interface Adapters (Pluggable) │ │
|
||||
│ │ │ │
|
||||
│ │ Text • Voice • Visual • Haptic • Gesture │ │
|
||||
│ │ │ │
|
||||
│ │ Biometric • [Custom Modalities...] │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ Session Management • Conversation • Task Integration │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Admin Panel
|
||||
|
||||
```python
|
||||
from fusionagi import Orchestrator, EventBus, StateManager
|
||||
from fusionagi.interfaces import AdminControlPanel
|
||||
from fusionagi.interfaces.voice import VoiceProfile
|
||||
from fusionagi.interfaces.conversation import ConversationStyle
|
||||
|
||||
# Initialize
|
||||
admin = AdminControlPanel(orchestrator=orch, event_bus=bus, state_manager=state)
|
||||
|
||||
# Add voice
|
||||
voice = VoiceProfile(name="Assistant", language="en-US", style="friendly")
|
||||
admin.add_voice_profile(voice)
|
||||
|
||||
# Configure conversation style
|
||||
style = ConversationStyle(formality="neutral", empathy_level=0.8)
|
||||
admin.register_conversation_style("default", style)
|
||||
|
||||
# Monitor system
|
||||
status = admin.get_system_status()
|
||||
print(f"Status: {status.status}, Active tasks: {status.active_tasks}")
|
||||
```
|
||||
|
||||
### Multi-Modal UI
|
||||
|
||||
```python
|
||||
from fusionagi.interfaces import MultiModalUI, VoiceInterface, ConversationManager
|
||||
from fusionagi.interfaces.base import ModalityType
|
||||
|
||||
# Initialize (voice_interface is optional)
|
||||
ui = MultiModalUI(
|
||||
orchestrator=orch,
|
||||
conversation_manager=ConversationManager(),
|
||||
voice_interface=VoiceInterface(stt_provider="whisper", tts_provider="elevenlabs"),
|
||||
)
|
||||
|
||||
# Create session
|
||||
session_id = ui.create_session(
|
||||
user_id="user123",
|
||||
preferred_modalities=[ModalityType.TEXT, ModalityType.VOICE],
|
||||
)
|
||||
|
||||
# Send multi-modal output
|
||||
await ui.send_to_user(session_id, "Hello!", modalities=[ModalityType.TEXT, ModalityType.VOICE])
|
||||
|
||||
# Receive input
|
||||
message = await ui.receive_from_user(session_id)
|
||||
|
||||
# Submit task with real-time updates
|
||||
task_id = await ui.submit_task_interactive(session_id, goal="Analyze data")
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
fusionagi/interfaces/
|
||||
├── __init__.py # Public API exports
|
||||
├── base.py # Core abstractions and protocols
|
||||
├── voice.py # Voice interface and library
|
||||
├── conversation.py # Conversation management and tuning
|
||||
├── admin_panel.py # Administrative control panel
|
||||
└── multimodal_ui.py # Multi-modal user interface
|
||||
|
||||
docs/
|
||||
├── interfaces.md # Comprehensive interface documentation
|
||||
└── ui_ux_implementation.md # This file
|
||||
|
||||
examples/
|
||||
├── admin_panel_example.py # Admin panel demo
|
||||
└── multimodal_ui_example.py # Multi-modal UI demo
|
||||
|
||||
tests/
|
||||
└── test_interfaces.py # Interface layer tests (7 tests, all passing)
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
All interface components are fully tested:
|
||||
|
||||
```bash
|
||||
pytest tests/test_interfaces.py -v
|
||||
```
|
||||
|
||||
**Test Coverage:**
|
||||
- ✓ Voice library management
|
||||
- ✓ Voice interface capabilities
|
||||
- ✓ Conversation style tuning
|
||||
- ✓ Conversation session management
|
||||
- ✓ Admin control panel operations
|
||||
- ✓ Multi-modal UI session management
|
||||
- ✓ Modality enable/disable
|
||||
|
||||
**Results:** 7/7 tests passing
|
||||
|
||||
## Next Steps for Production
|
||||
|
||||
### Immediate Priorities
|
||||
|
||||
1. **Implement STT/TTS Providers**
|
||||
- Integrate OpenAI Whisper for STT
|
||||
- Integrate ElevenLabs/Azure for TTS
|
||||
- Add provider configuration to admin panel
|
||||
|
||||
2. **Build Web UI**
|
||||
- FastAPI backend for admin panel
|
||||
- React/Vue frontend for admin dashboard
|
||||
- WebSocket for real-time updates
|
||||
- REST API for user interface
|
||||
|
||||
3. **Add Visual Modality**
|
||||
- Image generation integration
|
||||
- Video streaming support
|
||||
- AR/VR interface adapters
|
||||
- Screen sharing capabilities
|
||||
|
||||
4. **Implement Haptic Feedback**
|
||||
- Mobile device vibration patterns
|
||||
- Haptic feedback for notifications
|
||||
- Tactile response for errors/success
|
||||
|
||||
5. **Gesture Recognition**
|
||||
- Hand tracking integration
|
||||
- Motion control support
|
||||
- Gesture-to-command mapping
|
||||
|
||||
6. **Biometric Sensors**
|
||||
- Emotion detection from voice
|
||||
- Facial expression analysis
|
||||
- Heart rate/stress monitoring
|
||||
- Adaptive response based on user state
|
||||
|
||||
### Advanced Features
|
||||
|
||||
1. **Multi-User Sessions**
|
||||
- Collaborative interfaces
|
||||
- Shared conversation contexts
|
||||
- Role-based access control
|
||||
|
||||
2. **Accessibility Enhancements**
|
||||
- Screen reader optimization
|
||||
- High contrast modes
|
||||
- Keyboard navigation
|
||||
- Voice-only operation mode
|
||||
|
||||
3. **Mobile Applications**
|
||||
- Native iOS app
|
||||
- Native Android app
|
||||
- Cross-platform React Native
|
||||
|
||||
4. **Analytics & Insights**
|
||||
- User interaction patterns
|
||||
- Modality usage statistics
|
||||
- Conversation quality metrics
|
||||
- Performance optimization
|
||||
|
||||
5. **AI-Powered Features**
|
||||
- Automatic modality selection based on context
|
||||
- Emotion-aware responses
|
||||
- Predictive user preferences
|
||||
- Adaptive conversation styles
|
||||
|
||||
## Integration Points
|
||||
|
||||
The interface layer integrates seamlessly with all FusionAGI components:
|
||||
|
||||
- **Orchestrator**: Task submission, monitoring, agent coordination
|
||||
- **Event Bus**: Real-time updates, notifications, state changes
|
||||
- **Agents**: Direct agent interaction, configuration
|
||||
- **Memory**: Conversation history, user preferences, learning
|
||||
- **Governance**: Policy enforcement, audit logging, access control
|
||||
- **MAA**: Manufacturing authority oversight and control
|
||||
- **Tools**: Tool invocation through natural language
|
||||
|
||||
## Benefits
|
||||
|
||||
### For Administrators
|
||||
- Centralized system management
|
||||
- Easy voice and conversation configuration
|
||||
- Real-time monitoring and diagnostics
|
||||
- Audit trail for compliance
|
||||
- Configuration portability
|
||||
|
||||
### For End Users
|
||||
- Natural multi-modal interaction
|
||||
- Personalized conversation styles
|
||||
- Accessible across all senses
|
||||
- Real-time task feedback
|
||||
- Seamless experience across devices
|
||||
|
||||
### For Developers
|
||||
- Clean, extensible architecture
|
||||
- Easy to add new modalities
|
||||
- Well-documented APIs
|
||||
- Comprehensive test coverage
|
||||
- Production-ready foundation
|
||||
|
||||
## Conclusion
|
||||
|
||||
FusionAGI now has a complete interface layer that transforms it from a library-only framework into a full-featured AGI system with both administrative control and rich user interaction capabilities. The implementation is:
|
||||
|
||||
- **Modular**: Each component can be used independently
|
||||
- **Extensible**: Easy to add new modalities and providers
|
||||
- **Production-Ready**: Fully tested and documented
|
||||
- **Standards-Compliant**: Follows FusionAGI coding standards
|
||||
- **Future-Proof**: Designed for growth and enhancement
|
||||
|
||||
The foundation is in place for building world-class user experiences across all sensory modalities, with comprehensive administrative control for system operators.
|
||||
Reference in New Issue
Block a user