541 lines
12 KiB
Markdown
541 lines
12 KiB
Markdown
# NowYouSeeMe API Reference
|
|
|
|
## Overview
|
|
|
|
The NowYouSeeMe holodeck environment provides comprehensive APIs for multi-device connectivity, Azure integration, and real-time 6DOF pose estimation. This document describes the complete API surface for all components.
|
|
|
|
## Table of Contents
|
|
|
|
1. [Device Management API](#device-management-api)
|
|
2. [Azure Integration API](#azure-integration-api)
|
|
3. [SLAM Processing API](#slam-processing-api)
|
|
4. [UI/UX API](#uiux-api)
|
|
5. [Configuration API](#configuration-api)
|
|
6. [Network Protocols](#network-protocols)
|
|
|
|
## Device Management API
|
|
|
|
### DeviceManager Class
|
|
|
|
The `DeviceManager` class handles discovery, connection, and management of multiple device types simultaneously.
|
|
|
|
#### Constructor
|
|
```cpp
|
|
DeviceManager(int discoveryPort = 8080, int maxConnections = 100)
|
|
```
|
|
|
|
#### Device Types
|
|
- `WIFI_CARD` - WiFi network interfaces
|
|
- `ACCESS_POINT` - WiFi access points
|
|
- `CELL_PHONE` - Mobile devices
|
|
- `HOTSPOT` - Mobile hotspots
|
|
- `BLUETOOTH` - Bluetooth devices
|
|
- `ZIGBEE` - ZigBee devices
|
|
- `LORA` - LoRa devices
|
|
- `CUSTOM` - Custom network interfaces
|
|
|
|
#### Device Status
|
|
- `DISCONNECTED` - Device not connected
|
|
- `CONNECTING` - Connection in progress
|
|
- `CONNECTED` - Successfully connected
|
|
- `STREAMING` - Actively streaming data
|
|
- `ERROR` - Connection error
|
|
|
|
#### Methods
|
|
|
|
##### Discovery Management
|
|
```cpp
|
|
bool startDiscovery()
|
|
void stopDiscovery()
|
|
```
|
|
|
|
##### Device Connection
|
|
```cpp
|
|
bool connectToDevice(const std::string& deviceId)
|
|
bool disconnectFromDevice(const std::string& deviceId)
|
|
void disconnectAll()
|
|
```
|
|
|
|
##### Data Transmission
|
|
```cpp
|
|
bool sendData(const std::string& deviceId, const std::vector<uint8_t>& data)
|
|
std::vector<uint8_t> receiveData(const std::string& deviceId, int timeout_ms = 1000)
|
|
```
|
|
|
|
##### Device Information
|
|
```cpp
|
|
std::vector<DeviceInfo> getConnectedDevices()
|
|
std::vector<DeviceInfo> getAllDevices()
|
|
DeviceInfo getDeviceInfo(const std::string& deviceId)
|
|
```
|
|
|
|
##### Callbacks
|
|
```cpp
|
|
void setDeviceCallback(std::function<void(const DeviceInfo&)> callback)
|
|
void setDataCallback(std::function<void(const std::string&, const std::vector<uint8_t>&)> callback)
|
|
```
|
|
|
|
##### Statistics
|
|
```cpp
|
|
DeviceStats getStats()
|
|
```
|
|
|
|
#### DeviceInfo Structure
|
|
```cpp
|
|
struct DeviceInfo {
|
|
std::string id;
|
|
std::string name;
|
|
DeviceType type;
|
|
DeviceStatus status;
|
|
std::string mac_address;
|
|
std::string ip_address;
|
|
int port;
|
|
double signal_strength;
|
|
std::map<std::string, std::string> capabilities;
|
|
std::chrono::steady_clock::time_point last_seen;
|
|
bool is_active;
|
|
};
|
|
```
|
|
|
|
#### DeviceStats Structure
|
|
```cpp
|
|
struct DeviceStats {
|
|
int total_devices;
|
|
int connected_devices;
|
|
int active_streams;
|
|
double average_signal_strength;
|
|
std::chrono::steady_clock::time_point last_update;
|
|
};
|
|
```
|
|
|
|
## Azure Integration API
|
|
|
|
### AzureIntegration Class
|
|
|
|
The `AzureIntegration` class provides GPU computing and AI Foundry resources through Azure services.
|
|
|
|
#### Constructor
|
|
```cpp
|
|
AzureIntegration(const AzureConfig& config)
|
|
```
|
|
|
|
#### AzureConfig Structure
|
|
```cpp
|
|
struct AzureConfig {
|
|
std::string subscription_id;
|
|
std::string resource_group;
|
|
std::string location;
|
|
std::string storage_account;
|
|
std::string container_name;
|
|
std::string compute_cluster;
|
|
std::string ai_workspace;
|
|
std::string tenant_id;
|
|
std::string client_id;
|
|
std::string client_secret;
|
|
};
|
|
```
|
|
|
|
#### Methods
|
|
|
|
##### Authentication
|
|
```cpp
|
|
bool authenticate()
|
|
```
|
|
|
|
##### GPU Resource Management
|
|
```cpp
|
|
bool provisionGPUResource(const std::string& vmName, const std::string& gpuType,
|
|
int gpuCount, int memoryGB)
|
|
bool deprovisionGPUResource(const std::string& vmName)
|
|
std::vector<GPUResource> getAvailableGPUResources()
|
|
```
|
|
|
|
##### AI Foundry Integration
|
|
```cpp
|
|
bool setupAIWorkspace(const std::string& workspaceName)
|
|
bool deployModel(const std::string& workspaceName, const std::string& modelName,
|
|
const std::vector<uint8_t>& modelData)
|
|
```
|
|
|
|
##### Compute Job Management
|
|
```cpp
|
|
std::string submitComputeJob(const std::string& jobType, const std::string& resourceId,
|
|
const std::vector<uint8_t>& inputData,
|
|
std::function<void(const std::vector<uint8_t>&)> callback)
|
|
bool cancelJob(const std::string& jobId)
|
|
std::vector<ComputeJob> getActiveJobs()
|
|
```
|
|
|
|
##### Monitoring
|
|
```cpp
|
|
bool startMonitoring()
|
|
void stopMonitoring()
|
|
```
|
|
|
|
##### Callbacks
|
|
```cpp
|
|
void setJobCompletionCallback(std::function<void(const std::string&, const std::vector<uint8_t>&)> callback)
|
|
void setResourceStatusCallback(std::function<void(const std::string&, const std::string&)> callback)
|
|
```
|
|
|
|
##### Statistics
|
|
```cpp
|
|
AzureStats getStats()
|
|
```
|
|
|
|
#### GPUResource Structure
|
|
```cpp
|
|
struct GPUResource {
|
|
std::string vm_id;
|
|
std::string vm_name;
|
|
std::string gpu_type;
|
|
int gpu_count;
|
|
int memory_gb;
|
|
std::string status;
|
|
std::chrono::steady_clock::time_point last_used;
|
|
bool is_available;
|
|
};
|
|
```
|
|
|
|
#### AIFoundryResource Structure
|
|
```cpp
|
|
struct AIFoundryResource {
|
|
std::string workspace_id;
|
|
std::string workspace_name;
|
|
std::string compute_target;
|
|
std::string model_registry;
|
|
std::vector<std::string> available_models;
|
|
std::string status;
|
|
int active_jobs;
|
|
int max_jobs;
|
|
};
|
|
```
|
|
|
|
#### ComputeJob Structure
|
|
```cpp
|
|
struct ComputeJob {
|
|
std::string job_id;
|
|
std::string job_type;
|
|
std::string resource_id;
|
|
std::string status;
|
|
std::chrono::steady_clock::time_point start_time;
|
|
std::chrono::steady_clock::time_point end_time;
|
|
double progress;
|
|
std::vector<uint8_t> input_data;
|
|
std::vector<uint8_t> output_data;
|
|
std::function<void(const std::vector<uint8_t>&)> callback;
|
|
};
|
|
```
|
|
|
|
## SLAM Processing API
|
|
|
|
### SLAMProcessor Class
|
|
|
|
The `SLAMProcessor` class handles real-time SLAM processing with RF-vision fusion.
|
|
|
|
#### Methods
|
|
```cpp
|
|
void start()
|
|
void stop()
|
|
void setParameters(const SLAMParameters& params)
|
|
SLAMResult getCurrentPose()
|
|
std::vector<MapPoint> getMapPoints()
|
|
```
|
|
|
|
#### SLAMParameters Structure
|
|
```cpp
|
|
struct SLAMParameters {
|
|
double map_scale;
|
|
int update_rate;
|
|
bool enable_rf_fusion;
|
|
bool enable_vision_slam;
|
|
bool enable_nerf_rendering;
|
|
};
|
|
```
|
|
|
|
#### SLAMResult Structure
|
|
```cpp
|
|
struct SLAMResult {
|
|
Eigen::Matrix4d pose;
|
|
double confidence;
|
|
std::chrono::steady_clock::time_point timestamp;
|
|
std::vector<double> uncertainties;
|
|
};
|
|
```
|
|
|
|
## UI/UX API
|
|
|
|
### HolodeckUI Class
|
|
|
|
The `HolodeckUI` class provides a comprehensive PyQt6-based user interface.
|
|
|
|
#### Constructor
|
|
```python
|
|
HolodeckUI()
|
|
```
|
|
|
|
#### Methods
|
|
|
|
##### Device Management
|
|
```python
|
|
def start_device_discovery(self)
|
|
def stop_device_discovery(self)
|
|
def connect_to_device(self)
|
|
def disconnect_from_device(self)
|
|
```
|
|
|
|
##### SLAM Processing
|
|
```python
|
|
def start_slam_processing(self)
|
|
def stop_slam_processing(self)
|
|
```
|
|
|
|
##### Azure Integration
|
|
```python
|
|
def connect_to_azure(self)
|
|
def disconnect_from_azure(self)
|
|
def provision_gpu_resource(self)
|
|
def deprovision_gpu_resource(self)
|
|
def setup_ai_workspace(self)
|
|
def deploy_model(self)
|
|
```
|
|
|
|
##### View Management
|
|
```python
|
|
def set_view(self, view_type: str)
|
|
def update_camera_position(self)
|
|
```
|
|
|
|
##### Configuration
|
|
```python
|
|
def open_configuration(self)
|
|
def save_configuration(self)
|
|
def calibrate_camera(self)
|
|
```
|
|
|
|
#### View Types
|
|
- `"3D"` - 3D OpenGL visualization
|
|
- `"2D"` - 2D map view
|
|
- `"NeRF"` - Neural Radiance Fields rendering
|
|
|
|
### HolodeckGLWidget Class
|
|
|
|
OpenGL widget for 3D visualization.
|
|
|
|
#### Methods
|
|
```python
|
|
def set_view_type(self, view_type: str)
|
|
def set_camera_position(self, x: float, y: float, z: float)
|
|
def update(self)
|
|
```
|
|
|
|
## Configuration API
|
|
|
|
### Configuration File Format
|
|
|
|
The system uses JSON configuration files for all settings.
|
|
|
|
#### Main Configuration
|
|
```json
|
|
{
|
|
"holodeck": {
|
|
"enabled": true,
|
|
"view_type": "3D",
|
|
"update_rate": 30
|
|
},
|
|
"device_management": {
|
|
"discovery_port": 8080,
|
|
"max_connections": 100,
|
|
"discovery_interval": 5,
|
|
"connection_timeout": 30
|
|
},
|
|
"azure_integration": {
|
|
"subscription_id": "your-subscription-id",
|
|
"resource_group": "nowyouseeme-rg",
|
|
"location": "eastus",
|
|
"tenant_id": "your-tenant-id",
|
|
"client_id": "your-client-id",
|
|
"client_secret": "your-client-secret"
|
|
},
|
|
"slam_processing": {
|
|
"map_scale": 1.0,
|
|
"update_rate": 30,
|
|
"enable_rf_fusion": true,
|
|
"enable_vision_slam": true,
|
|
"enable_nerf_rendering": true
|
|
},
|
|
"ui_settings": {
|
|
"window_width": 1600,
|
|
"window_height": 1200,
|
|
"theme": "dark",
|
|
"auto_save": true
|
|
}
|
|
}
|
|
```
|
|
|
|
## Network Protocols
|
|
|
|
### Device Communication Protocol
|
|
|
|
#### Connection Establishment
|
|
1. Device discovery via UDP broadcast on port 8080
|
|
2. TCP connection establishment on device-specific port
|
|
3. Handshake protocol for capability negotiation
|
|
4. Data streaming with protocol headers
|
|
|
|
#### Message Format
|
|
```
|
|
[Header: 8 bytes][Payload: variable length]
|
|
Header: [MessageType: 2 bytes][Length: 4 bytes][Flags: 2 bytes]
|
|
```
|
|
|
|
#### Message Types
|
|
- `0x0001` - Device Info
|
|
- `0x0002` - Data Stream
|
|
- `0x0003` - Control Command
|
|
- `0x0004` - Status Update
|
|
- `0x0005` - Error Report
|
|
|
|
### Azure REST API Integration
|
|
|
|
#### Authentication
|
|
- OAuth 2.0 client credentials flow
|
|
- Bearer token authentication
|
|
- Automatic token refresh
|
|
|
|
#### GPU Resource Management
|
|
- Azure Compute REST API
|
|
- VM provisioning with GPU extensions
|
|
- Real-time status monitoring
|
|
|
|
#### AI Foundry Integration
|
|
- Azure Machine Learning REST API
|
|
- Model deployment and management
|
|
- Workspace and compute target management
|
|
|
|
## Error Handling
|
|
|
|
### Error Codes
|
|
- `0x0001` - Device not found
|
|
- `0x0002` - Connection failed
|
|
- `0x0003` - Authentication failed
|
|
- `0x0004` - Resource unavailable
|
|
- `0x0005` - Invalid parameters
|
|
- `0x0006` - Timeout
|
|
- `0x0007` - Network error
|
|
- `0x0008` - Azure API error
|
|
|
|
### Exception Handling
|
|
```cpp
|
|
try {
|
|
device_manager.connectToDevice(device_id);
|
|
} catch (const DeviceException& e) {
|
|
spdlog::error("Device error: {}", e.what());
|
|
} catch (const NetworkException& e) {
|
|
spdlog::error("Network error: {}", e.what());
|
|
} catch (const AzureException& e) {
|
|
spdlog::error("Azure error: {}", e.what());
|
|
}
|
|
```
|
|
|
|
## Performance Considerations
|
|
|
|
### Multi-threading
|
|
- Device discovery runs in separate thread
|
|
- Data reception uses thread pool
|
|
- UI updates on main thread
|
|
- Azure operations use async/await pattern
|
|
|
|
### Memory Management
|
|
- Device connections use smart pointers
|
|
- Data buffers are pre-allocated
|
|
- GPU memory is managed by Azure
|
|
- UI elements are properly disposed
|
|
|
|
### Network Optimization
|
|
- Connection pooling for multiple devices
|
|
- Data compression for large payloads
|
|
- Heartbeat mechanism for connection monitoring
|
|
- Automatic reconnection on failure
|
|
|
|
## Security
|
|
|
|
### Authentication
|
|
- Azure AD integration for cloud resources
|
|
- Device-specific authentication tokens
|
|
- Encrypted communication channels
|
|
- Certificate-based device verification
|
|
|
|
### Data Protection
|
|
- End-to-end encryption for sensitive data
|
|
- Secure storage of configuration files
|
|
- Access control for device management
|
|
- Audit logging for all operations
|
|
|
|
## Examples
|
|
|
|
### Basic Device Management
|
|
```cpp
|
|
DeviceManager manager(8080, 100);
|
|
manager.setDeviceCallback([](const DeviceInfo& device) {
|
|
std::cout << "Found device: " << device.name << std::endl;
|
|
});
|
|
|
|
manager.startDiscovery();
|
|
manager.connectToDevice("device-001");
|
|
```
|
|
|
|
### Azure GPU Provisioning
|
|
```cpp
|
|
AzureConfig config;
|
|
config.subscription_id = "your-subscription-id";
|
|
// ... set other config parameters
|
|
|
|
AzureIntegration azure(config);
|
|
azure.authenticate();
|
|
azure.provisionGPUResource("gpu-vm-001", "V100", 1, 32);
|
|
```
|
|
|
|
### UI Integration
|
|
```python
|
|
ui = HolodeckUI()
|
|
ui.start_device_discovery()
|
|
ui.connect_to_azure()
|
|
ui.start_slam_processing()
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
1. **Device Discovery Fails**
|
|
- Check network permissions
|
|
- Verify firewall settings
|
|
- Ensure discovery port is available
|
|
|
|
2. **Azure Authentication Fails**
|
|
- Verify credentials in configuration
|
|
- Check Azure AD permissions
|
|
- Ensure subscription is active
|
|
|
|
3. **SLAM Processing Errors**
|
|
- Check camera calibration
|
|
- Verify sensor data quality
|
|
- Monitor system resources
|
|
|
|
4. **UI Performance Issues**
|
|
- Reduce update rate
|
|
- Disable unnecessary features
|
|
- Monitor GPU usage
|
|
|
|
### Debug Mode
|
|
Enable debug logging by setting environment variable:
|
|
```bash
|
|
export NOWYOUSEE_DEBUG=1
|
|
```
|
|
|
|
### Log Files
|
|
- Device logs: `/var/log/nowyouseeme/device.log`
|
|
- Azure logs: `/var/log/nowyouseeme/azure.log`
|
|
- SLAM logs: `/var/log/nowyouseeme/slam.log`
|
|
- UI logs: `/var/log/nowyouseeme/ui.log` |