09 - Data Management Guide

09 - Data Management Guide

💾 Backup, Export, and Manage Your Data
⏱️ Time Estimate: 15 minutes
📋 What You’ll Learn: Export options, backup strategies, data recovery, storage management

---

Table of Contents

• Export Options Overview
• Individual Transcript Export
• Project-Level Export
• Full Application Backup
• Import Workflows
• Storage Locations
• Disk Space Management
• Data Recovery
• Best Practices

---

Export Options Overview

Feature: F006 & F013 - Export/Backup System

Selfoss provides three levels of export:

Level	Scope	Use Case	Feature
Individual	Single transcript	Share specific meeting	F006
Project	All transcripts in project	Archive completed project	F006
Full Backup	Entire application	Device migration, disaster recovery	F013

Export Formats

Transcript data:

• 📄 JSON: Raw structured data
• 📊 PDF: Professional report
• 📝 TXT: Plain text transcript
• 🖼️ PNG/SVG: Individual visualizations

Audio files:

• 🎵 WebM: Original recording format
• 📦 ZIP: Bundled with transcript

Database:

• 🗄️ SQLite: Complete database backup
• 📦 ZIP: Compressed archive

---

Individual Transcript Export

Exporting Single Transcripts

Step 1: Select Transcript

Click transcript in project list
    ↓
Right-click → Export Options

Step 2: Choose Format

┌─────────────────────────────┐
│ Export Transcript           │
├─────────────────────────────┤
│ ☐ JSON (raw data)           │
│ ☐ PDF (formatted report)    │
│ ☐ TXT (plain text)          │
│ ☐ Audio file (if recording) │
│                             │
│ [Cancel]  [Export]          │
└─────────────────────────────┘

Step 3: Select Destination

• File picker opens
• Choose save location
• Name the file

JSON Export Format

Contains:

{
  "metadata": {
    "title": "Q4 Planning Meeting",
    "date": "2024-10-16",
    "participants": ["John", "Sarah", "Mike"],
    "upload_date": "2024-10-16T14:30:00Z"
  },
  "decisions": [
    {
      "id": "D1",
      "text": "Approve Q4 marketing budget: $500K",
      "timestamp": "00:15:30",
      "flows_to": ["A1", "A2"]
    }
  ],
  "actions": [
    {
      "id": "A1",
      "task": "Finalize vendor contracts",
      "owner": "John",
      "deadline": "2024-10-20",
      "priority": "high",
      "source_decision": "D1"
    }
  ],
  "concepts": [...],
  "temporal_data": {...},
  "processing_metadata": {
    "model_used": "gpt-4o-mini",
    "provider": "openai",
    "tokens_used": 5234,
    "processing_cost": 0.0023
  }
}

Use cases:

• Integration with other tools
• Custom analysis scripts
• Data migration
• Backup reference

PDF Export Format

Contains:

┌────────────────────────────┐
│ Q4 Planning Meeting        │
│ October 16, 2024           │
├────────────────────────────┤
│ Participants:              │
│ • John Smith               │
│ • Sarah Johnson            │
│ • Mike Chen                │
├────────────────────────────┤
│ Decisions:                 │
│ 1. Approve Q4 marketing... │
│                            │
│ [Decision Flowchart PNG]   │
│                            │
│ Actions:                   │
│ [Action Matrix Table]      │
│                            │
│ Concepts:                  │
│ [Mind Map PNG]             │
└────────────────────────────┘

Use cases:

• Sharing with executives
• Meeting minutes documentation
• Archival records
• Stakeholder reports

Exporting Audio Files

For recorded transcripts:

1. Right-click transcript
2. Select “Download Audio”
3. Original WebM file downloads

File details:

• Format: WebM (original)
• Size: ~1MB per minute
• Includes: Full audio recording
• Location: Choose save destination

💡 Pro Tip: Export audio before deleting transcripts if you might need it later.

---

Project-Level Export

Exporting Entire Projects

Step 1: Right-Click Project

Projects Sidebar
    ↓
Right-click project → Export Project

Step 2: Choose Options

┌─────────────────────────────────┐
│ Export Project: Engineering     │
├─────────────────────────────────┤
│ ☑ Transcripts (12 files)        │
│ ☑ Audio recordings (8 files)    │
│ ☑ Visualizations (36 images)    │
│ ☐ Processing metadata           │
│                                 │
│ Format: [ZIP Archive       ▼]   │
│                                 │
│ [Cancel]  [Export Project]      │
└─────────────────────────────────┘

Step 3: Wait for Processing

• Progress bar shows status
• Large projects may take 1-2 minutes
• Audio files are largest component

Project Archive Structure

Engineering_Project_2024-10-16.zip
├── project_metadata.json
├── transcripts/
│   ├── meeting_001.json
│   ├── meeting_002.json
│   └── ...
├── audio_recordings/
│   ├── recording_001.webm
│   ├── recording_002.webm
│   └── ...
└── visualizations/
    ├── meeting_001_flowchart.png
    ├── meeting_001_mindmap.svg
    └── ...

Use Cases

Project completion:

• Archive finished initiatives
• Free up database space
• Create handoff packages

Client deliverables:

• Package all meeting records
• Include audio for reference
• Professional presentation

Compliance:

• Legal record keeping
• Audit trail documentation
• Historical reference

---

Full Application Backup

Feature: F013 - Complete Data Export

Create a complete snapshot of all Selfoss data.

Creating Full Backup

Step 1: Open Settings

Settings → Data Management → Export All Data

Step 2: Configure Backup

┌─────────────────────────────────┐
│ Full Application Backup         │
├─────────────────────────────────┤
│ Include:                        │
│ ☑ All projects (5)              │
│ ☑ All transcripts (47)          │
│ ☑ All audio recordings (23)     │
│ ☑ LLM settings                  │
│ ☑ Application preferences       │
│ ☑ License information           │
│                                 │
│ Estimated size: 2.3 GB          │
│ Destination: Downloads folder   │
│                                 │
│ [Cancel]  [Create Backup]       │
└─────────────────────────────────┘

Step 3: Wait for Completion

• Progress indicator shows stages:
  • Exporting database…
  • Archiving audio files…
  • Compressing data…
  • Saving to disk…

⏱️ Time estimate:

• Small (< 100 transcripts): 30 seconds
• Medium (100-500): 2-3 minutes
• Large (500+): 5-10 minutes

Backup Archive Structure

selfoss_backup_2024-10-16_14-30-00.zip
├── metadata.json
├── projects.json
├── transcripts.json
├── llm_settings.json
├── visualizations.json
└── audio_recordings/
    ├── project_1/
    │   ├── recording_001.webm
    │   └── recording_002.webm
    ├── project_2/
    │   └── recording_003.webm
    └── ...

Backup Metadata

The metadata.json file contains:

{
  "backup_version": "1.0",
  "backup_date": "2024-10-16T14:30:00Z",
  "selfoss_version": "1.5.2",
  "total_projects": 5,
  "total_transcripts": 47,
  "total_audio_files": 23,
  "total_size_bytes": 2400000000,
  "includes_audio": true,
  "includes_settings": true
}

Use cases:

• Validate backup integrity
• Track backup versions
• Verify completeness

---

Import Workflows

Importing Individual Transcripts

Step 1: Upload JSON File

Upload & Process → Select JSON file
    ↓
Selfoss detects JSON format
    ↓
"Import processed transcript?" prompt

Step 2: Choose Import Options

┌─────────────────────────────────┐
│ Import Transcript               │
├─────────────────────────────────┤
│ Title: Q4 Planning Meeting      │
│ Date: Oct 16, 2024              │
│                                 │
│ Destination:                    │
│ ○ New Project: [___________]    │
│ ● Existing: [Engineering   ▼]   │
│                                 │
│ ☐ Overwrite if exists           │
│                                 │
│ [Cancel]  [Import]              │
└─────────────────────────────────┘

Step 3: Verify Import

• Transcript appears in project
• All visualizations recreated
• Metadata preserved

Importing Full Backups

Step 1: Select Backup File

Settings → Data Management → Import Data
    ↓
File picker → Select .zip backup

Step 2: Conflict Resolution

┌─────────────────────────────────┐
│ Import Backup                   │
├─────────────────────────────────┤
│ Backup contains:                │
│ • 5 projects                    │
│ • 47 transcripts                │
│ • 23 audio recordings           │
│                                 │
│ Conflicts detected:             │
│ • 2 projects already exist      │
│ • 8 transcripts already exist   │
│                                 │
│ ☑ Overwrite existing data       │
│ ☐ Skip duplicates               │
│ ☐ Merge (keep both)             │
│                                 │
│ [Cancel]  [Import]              │
└─────────────────────────────────┘

Step 3: Wait for Import

• Progress shows:
  • Extracting archive…
  • Importing projects…
  • Restoring transcripts…
  • Copying audio files…
  • Rebuilding database…

⏱️ Time estimate: Similar to backup time

Step 4: Review Results

✅ Import Complete

   Imported:
   • 5 projects
   • 47 transcripts
   • 23 audio files

   Skipped:
   • 2 duplicate projects
   • 8 duplicate transcripts

   Errors:
   • None

Import Strategies

Overwrite Mode:

• Replaces existing data
• Use for: Device migration, clean restore

Skip Duplicates:

• Keeps existing, ignores conflicts
• Use for: Selective restore, adding new data

Merge Mode:

• Renames imported items (adds suffix)
• Use for: Comparing versions, keeping both

💡 Pro Tip: Test imports on a backup device first to verify data integrity.

---

Storage Locations

Application Data Directory

Windows:

C:\Users\{Username}\AppData\Roaming\selfoss\
├── selfoss.db (SQLite database)
├── audio_recordings/
│   ├── project_1/
│   └── project_2/
└── cache/
    └── whisper_models/

macOS:

~/Library/Application Support/selfoss/
├── selfoss.db
├── audio_recordings/
└── cache/

Linux:

~/.local/share/selfoss/
├── selfoss.db
├── audio_recordings/
└── cache/

Audio File Storage

Structure:

audio_recordings/
├── project_1/
│   ├── recording_2024-10-16_14-30-25.webm
│   └── recording_2024-10-16_15-45-10.webm
├── project_2/
│   └── recording_2024-10-17_09-15-30.webm
└── ...

Naming convention:

recording_YYYY-MM-DD_HH-MM-SS.webm

Whisper Model Cache

Location:

cache/whisper_models/
├── tiny.en.bin (75MB)
├── base.en.bin (140MB)
└── small.en.bin (460MB)

Purpose:

• Local transcription models
• Downloaded via Ollama
• Shared across all transcripts

Database Schema

SQLite file: selfoss.db

Key tables:

• projects - Project metadata
• transcripts - Transcript data and processed JSON
• llm_settings - AI provider configurations
• visualizations - Generated images
• api_usage - Cost tracking (planned)

---

Disk Space Management

Storage Usage Breakdown

Typical 100-transcript installation:

Database (SQLite):        50 MB
Audio recordings:       6 GB (60 hours @ 1MB/min)
Whisper models:         1 GB
Visualizations cache:    200 MB
Total:                  ~7.25 GB

Checking Disk Usage

Windows:

Settings → System → Storage
    ↓
Look for Selfoss in app list

macOS:

du -sh ~/Library/Application\ Support/selfoss

Linux:

du -sh ~/.local/share/selfoss

Cleanup Strategies

1. Delete Old Recordings

Settings → Data Management → Audio Cleanup
    ↓
Select recordings older than: [6 months  ▼]
    ↓
Preview: 45 recordings, 2.7 GB
    ↓
[Delete Selected]

2. Export and Delete Projects

1. Export project archive
2. Verify backup is complete
3. Delete project from Selfoss
4. Move archive to external storage

3. Clean Whisper Models

Keep only models you use:
☑ whisper:base (140MB) ← Keep
☐ whisper:tiny (75MB)  ← Delete
☐ whisper:small (460MB) ← Delete

Saves: 535 MB

4. Database Optimization

Settings → Advanced → Optimize Database
    ↓
Vacuum: Reclaim unused space
Reindex: Improve query performance

Low Disk Space Warnings

Selfoss monitors disk space:

⚠️ Low Disk Space Warning

Available: 850 MB
Required: 2 GB (recommended)

Suggestions:
• Delete old recordings (1.5 GB)
• Export and remove projects (500 MB)
• Move backups to external drive

[Manage Storage]  [Dismiss]

---

Data Recovery

Backup Restoration

Scenario: Device crash, need to restore everything

Steps:

1. Install Selfoss on new device
2. Open Settings → Data Management → Import
3. Select full backup ZIP file
4. Choose “Overwrite” mode
5. Wait for import (5-10 minutes)
6. ✅ All data restored

Partial Data Loss

Scenario: Accidentally deleted project

Steps:

1. Locate most recent backup
2. Import backup with “Skip duplicates” mode
3. Deleted project is restored
4. Existing data unchanged

Database Corruption

Symptoms:

• App won’t open
• SQLite errors
• Missing data

Solutions:

Option 1: Restore from backup

1. Rename corrupted selfoss.db
2. Import latest backup
3. Verify data integrity

Option 2: Database repair

# Linux/macOS
cd ~/.local/share/selfoss/
sqlite3 selfoss.db "PRAGMA integrity_check;"
sqlite3 selfoss.db ".recover" | sqlite3 selfoss_recovered.db

Option 3: Manual export

If database accessible:
1. Export all projects individually
2. Reinstall Selfoss (fresh database)
3. Import exported projects

Audio File Recovery

Scenario: Audio files deleted but database intact

If you have backup:

1. Extract backup ZIP
2. Copy audio_recordings/ folder
3. Paste into Selfoss data directory
4. Restart Selfoss

If no backup:

• Audio is lost
• Transcripts still accessible
• Can re-record if needed

---

Best Practices

Backup Schedule

Recommended:

Daily:   None (auto-save handles this)
Weekly:  Quick export of changed projects
Monthly: Full application backup
Quarterly: Archive old projects, clean storage

Automation (planned):

• Scheduled automatic backups
• Incremental backups (changes only)
• Cloud sync options

Backup Storage

Local:

• ✅ Fast access
• ✅ No internet needed
• ❌ Vulnerable to device failure

External drive:

• ✅ Safe from device issues
• ✅ Large capacity
• ❌ Must remember to backup

Cloud storage:

• ✅ Off-site protection
• ✅ Accessible anywhere
• ❌ Privacy concerns (encrypt first)

Recommended approach:

Primary: Local SSD/HDD
Secondary: External drive (monthly)
Tertiary: Encrypted cloud (quarterly)

Version Control

Manual versioning:

selfoss_backup_2024-10-16.zip  ← Latest
selfoss_backup_2024-10-01.zip  ← Last month
selfoss_backup_2024-09-01.zip  ← Archive

Keep:

• Latest backup (always)
• Monthly backups (last 6 months)
• Quarterly backups (last 2 years)

Testing Backups

Verify backups work:

1. Create test project
2. Add test transcript
3. Export full backup
4. Delete test data
5. Restore from backup
6. Verify test data returned

Frequency: Test quarterly

Privacy Considerations

Before sharing backups:

• Remove sensitive transcripts
• Redact participant names
• Clear API keys (Settings export)
• Encrypt ZIP file with password

Encrypted backups:

# Linux/macOS
zip -e selfoss_backup.zip selfoss_backup/*
# Enter password when prompted

# Extract
unzip selfoss_backup.zip

---

Next Steps

🎉 You’re now a data management expert!

Recommended Actions:

1. 💾 Create first full backup - Test the process
2. 📅 Set backup schedule - Calendar reminders
3. 🗂️ Organize backup storage - External drive or cloud
4. 🧪 Test restore process - Verify backups work
5. 🔒 Consider encryption → 14_PRIVACY_SECURITY_GUIDE.md

Advanced Topics:

• Automated backup scripts
• Database optimization
• Selective sync for large datasets
• Disaster recovery planning

---

💾 Your data, protected and portable.