Sankofa/docs/fairness-audit/OUTPUT_WEIGHTS.md

Output Weight Guidelines

Overview

Each output type in the fairness audit orchestration has a weight that represents the relative effort required to generate it. These weights are used to calculate the total output load (O) in the orchestration formula.

Weight Calculation

Weights are determined by:

1. Processing Complexity: How much computation is required?
2. Data Volume: How much data needs to be processed/generated?
3. Format Complexity: How complex is the output format?
4. Dependencies: Does it depend on other outputs?

Output Types and Weights

Lightweight Outputs (0.5 - 1.5 units)

Metrics Export (1.0 units)

• Type: metrics-export
• Weight: 1.0
• Description: Statistical parity difference, true positive rate, false positive rate metrics
• Complexity: Low - Simple calculations, small output
• Dependencies: None
• Format: JSON/CSV

Alert Configuration (0.8 units)

• Type: alerts-config
• Weight: 0.8
• Description: Automated alert rules for ongoing monitoring
• Complexity: Low - Rule generation, small output
• Dependencies: Metrics export
• Format: YAML/JSON

Detailed Report JSON (1.2 units)

• Type: detailed-report-json
• Weight: 1.2
• Description: Machine-readable detailed fairness analysis
• Complexity: Medium - Structured data, moderate size
• Dependencies: All metrics
• Format: JSON

Medium Outputs (1.5 - 2.0 units)

Flagged Cases CSV (1.5 units)

• Type: flagged-cases-csv
• Weight: 1.5
• Description: Export of cases flagged for potential bias issues
• Complexity: Medium - Data filtering, CSV generation
• Dependencies: Fairness evaluation
• Format: CSV

Executive Summary Slides (2.0 units)

• Type: exec-summary-slides
• Weight: 2.0
• Description: Executive presentation slides with key findings
• Complexity: Medium-High - Data aggregation, slide generation
• Dependencies: All metrics, summary analysis
• Format: PowerPoint/PDF

Dashboard Export (1.8 units)

• Type: dashboard-export
• Weight: 1.8
• Description: Interactive dashboard with fairness metrics
• Complexity: Medium - Dashboard generation, visualization
• Dependencies: All metrics
• Format: HTML/Interactive

Heavy Outputs (2.0 - 3.0 units)

Fairness Audit PDF (2.5 units)

• Type: fairness-audit-pdf
• Weight: 2.5
• Description: Comprehensive fairness audit report in PDF format
• Complexity: High - Full report generation, PDF formatting
• Dependencies: All analyses, metrics, findings
• Format: PDF

Compliance Report (2.2 units)

• Type: compliance-report
• Weight: 2.2
• Description: Regulatory compliance documentation
• Complexity: High - Regulatory formatting, documentation
• Dependencies: All analyses, audit trail
• Format: PDF/DOCX

Weight Rationale

Why These Weights?

1. Metrics Export (1.0): Baseline weight

   • Simple calculations
   • Small output size
   • Fast generation

2. Alert Configuration (0.8): Lighter than baseline

   • Minimal processing
   • Small output
   • Can reuse metrics

3. Flagged Cases CSV (1.5): 50% more than baseline

   • Requires filtering logic
   • Moderate data volume
   • CSV generation overhead

4. Detailed Report JSON (1.2): Slightly above baseline

   • Structured data compilation
   • Moderate complexity
   • JSON serialization

5. Executive Summary Slides (2.0): 2× baseline

   • Data aggregation required
   • Slide generation complexity
   • Visual formatting

6. Dashboard Export (1.8): Between medium and heavy

   • Dashboard framework overhead
   • Visualization generation
   • Interactive components

7. Fairness Audit PDF (2.5): 2.5× baseline

   • Comprehensive report
   • PDF formatting complexity
   • Large output size

8. Compliance Report (2.2): Slightly less than PDF

   • Regulatory formatting
   • Documentation requirements
   • Structured output

Weight Adjustment Guidelines

When to Increase Weight

• Output requires significant computation
• Large data volumes
• Complex formatting requirements
• Multiple dependencies
• Real-time processing needed

When to Decrease Weight

• Simple calculations
• Small output size
• Reusable components
• Cached results available
• Parallel processing possible

Example Scenarios

Scenario 1: Minimal Outputs

• Metrics Export (1.0)
• Alert Configuration (0.8)
• Total: 1.8 units

Scenario 2: Standard Audit

• Fairness Audit PDF (2.5)
• Metrics Export (1.0)
• Flagged Cases CSV (1.5)
• Total: 5.0 units

Scenario 3: Comprehensive Audit

• All 8 outputs
• Total: 13.0 units

Weight Validation

Design Target Check

For input load I:

• Target Output: O ≈ 1.2 × I
• Warning Threshold: O > 1.5 × (1.2 × I)
• Example: If I = 100, target O = 120, warn if O > 180

Total Load Check

• Expected: Total ≈ 3.2 × I
• Warning: Total > 1.3 × (3.2 × I)
• Example: If I = 100, expected total = 320, warn if total > 416

Future Considerations

1. Dynamic Weights: Adjust based on actual performance
2. Context-Aware: Weights vary by dataset size
3. Machine Learning: Learn optimal weights from history
4. Parallel Processing: Reduce effective weights for parallel outputs

Related Documentation

• Orchestration Engine
• Orchestration Design