Admin Bud-E

English

Deutsch

English

Deutsch

Appearance

Maintenance Operations ​

Regular maintenance keeps Admin Bud-E running smoothly and securely. This page covers routine tasks, troubleshooting, and system administration.

Routine Maintenance Schedule ​

Daily Tasks (Automated) ​

  • ✅ Automated backups (see Backups)
  • ✅ Log rotation (if configured)
  • ✅ Health checks (uptime monitoring)

Weekly Tasks (5-10 minutes) ​

  • Check system status
  • Review usage reports for anomalies
  • Verify backup success
  • Monitor disk space

Monthly Tasks (30-60 minutes) ​

  • Review and refill project budgets
  • Rotate old logs (if not automated)
  • Update system packages
  • Test backup restore
  • Review security logs

Quarterly Tasks (1-2 hours) ​

  • Rotate admin passwords
  • Review and update pricing
  • Audit user accounts (deactivate inactive)
  • Update Admin Bud-E (if new version available)
  • Review and update documentation

Annual Tasks (half day) ​

  • Full system audit
  • Disaster recovery drill
  • Review data retention policy
  • Update DPAs with providers
  • Security assessment

Maintenance Page in Admin UI ​

Navigate to Maintenance in the Admin UI to access:

1. Download Backup ​

Creates and downloads a snapshot of the SQLite database.

Use cases:

  • Before major changes (bulk user import, pricing update)
  • Weekly/monthly backups
  • Migration to new server

Steps:

  1. Click Download Backup
  2. Wait for download to start
  3. Save file with clear name: backup_YYYY-MM-DD.db
  4. Verify file size is reasonable (> 0 KB)

2. Restore from Backup ​

Replaces current database with a backup file.

DANGER

This overwrites all current data. Any changes made after the backup was taken will be permanently lost.

Use cases:

  • Recover from data corruption
  • Undo accidental bulk deletion
  • Migrate from another server

Steps:

  1. First: Download current database as a safety backup
  2. Click Restore from Backup
  3. Select backup file (.db file)
  4. Read warning carefully
  5. Confirm
  6. Wait for upload and processing (may take 1-5 minutes for large databases)
  7. Log out and back in to verify

3. Reset Database (Danger Zone) ​

Deletes all data and creates a fresh, empty database.

DANGER

This is irreversible. Only use for:

  • Fresh test environments
  • Complete system reset (e.g., switching organizations)
  • Never for production unless you're certain

What gets deleted:

  • All users
  • All projects
  • All usage logs
  • All providers and routes
  • All pricing rules

What survives:

  • Admin password (usually)
  • Server configuration

Steps:

  1. Create backup first (or accept data loss)
  2. Click Reset Database
  3. Type confirmation phrase (e.g., "DELETE ALL DATA")
  4. Confirm
  5. Re-configure everything from scratch

System Updates ​

Updating Ubuntu ​

Keep your server's OS up to date:

bash
# Update package list
sudo apt update

# Install available updates
sudo apt upgrade -y

# Reboot if kernel was updated
sudo reboot

Frequency: Monthly, or when critical security patches are released.

Updating Admin Bud-E ​

Check the GitHub repository for new releases.

Steps:

1. Backup current database:

bash
cp /opt/bud-e/school-bud-e-middleware/admin_bude.db ~/backup_before_update.db

2. Stop service:

bash
sudo systemctl stop admin-bude.service

3. Pull latest code:

bash
cd /opt/bud-e/school-bud-e-middleware
git pull origin main

4. Update dependencies:

bash
source .venv/bin/activate
pip install -r requirements.txt --upgrade

5. Run migrations (if any):

bash
# Check repository instructions for migration commands
# Example:
python migrate.py

6. Restart service:

bash
sudo systemctl start admin-bude.service
sudo systemctl status admin-bude.service

7. Verify:

  • Log into Admin UI
  • Check version number (if displayed)
  • Test a request from frontend

Updating Python Dependencies ​

Periodically update Python packages for security patches:

bash
cd /opt/bud-e/school-bud-e-middleware
source .venv/bin/activate

# Update pip itself
pip install --upgrade pip

# Update all packages
pip list --outdated
pip install --upgrade <package-name>

# Or update everything (risky, test first):
pip install -r requirements.txt --upgrade

WARNING

Major updates can break compatibility. Test in a staging environment first, or wait for Admin Bud-E official release notes.

Monitoring ​

Check Service Status ​

bash
sudo systemctl status admin-bude.service

Expected output:

● admin-bude.service - Admin Bud-E (middleware)
   Loaded: loaded (/etc/systemd/system/admin-bude.service; enabled)
   Active: active (running) since Mon 2024-12-15 08:00:00 UTC; 3 days ago

If not running:

bash
# View logs
sudo journalctl -u admin-bude.service -n 50

# Restart
sudo systemctl restart admin-bude.service

Check Disk Space ​

bash
df -h

Thresholds:

  • >50% free: Healthy
  • 20-50% free: Monitor
  • <20% free: Clean up or expand storage

What uses space:

  • SQLite database (grows over time)
  • Logs (/var/log/)
  • Backups (if stored locally)

Cleanup:

bash
# Rotate old logs
sudo journalctl --vacuum-time=30d

# Remove old backups (keep last 30 days)
find /opt/bud-e/backups -name "backup_*.db.gz" -mtime +30 -delete

# Check large files
du -h /opt/bud-e/ | sort -h | tail -20

Check Memory Usage ​

bash
free -h

Expected:

  • 2-4 GB typical usage
  • Spikes during heavy traffic

If memory is low:

  • Restart service to free cached memory
  • Consider upgrading to larger instance

Uptime Monitoring ​

Use external service to monitor availability:

Options:

Setup:

  1. Create account
  2. Add monitor:
    • URL: https://admin.yourschool.edu/health (or your admin URL)
    • Interval: 5 minutes
    • Alert: Email or SMS
  3. Test by stopping service briefly

Log Management ​

View Recent Logs ​

systemd logs:

bash
# Last 50 lines
sudo journalctl -u admin-bude.service -n 50

# Follow (live tail)
sudo journalctl -u admin-bude.service -f

# Logs from today
sudo journalctl -u admin-bude.service --since today

Application logs (if separate file):

bash
tail -f /opt/bud-e/school-bud-e-middleware/logs/app.log

Log Rotation ​

Logs can grow large. Rotate them regularly:

systemd journal (automatic):

bash
# Set max size
sudo journalctl --vacuum-size=500M

# Set max age
sudo journalctl --vacuum-time=30d

Custom application logs:

Create /etc/logrotate.d/admin-bude:

/opt/bud-e/school-bud-e-middleware/logs/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    missingok
    postrotate
        systemctl reload admin-bude.service > /dev/null 2>&1 || true
    endscript
}

Security Maintenance ​

Password Rotation ​

Admin password:

  • Change every 90 days
  • Use strong password (16+ chars, mixed case, symbols)
  • Store in password manager

Update password:

bash
# If your Admin Bud-E version supports env variable:
export ADMIN_PASSWORD="new-strong-password"

# Or edit env file:
sudo nano /opt/bud-e/env.sh
# Update ADMIN_PASSWORD line

# Restart service
sudo systemctl restart admin-bude.service

API Key Rotation ​

For compromised or old keys:

  1. Navigate to Users
  2. Click Rotate Key for affected user(s)
  3. Provide new key to user securely
  4. Monitor for unauthorized usage of old key

Security Patches ​

Subscribe to security announcements:

Apply patches promptly:

bash
sudo apt update
sudo apt upgrade -y
sudo reboot # If kernel updated

Firewall Review ​

bash
sudo ufw status verbose

Expected rules:

  • Allow: 22/tcp (SSH)
  • Allow: 80/tcp (HTTP)
  • Allow: 443/tcp (HTTPS)
  • Deny: Everything else

If rules are incorrect:

bash
# Remove bad rule
sudo ufw delete allow 8000/tcp

# Add correct rule
sudo ufw allow 443/tcp

# Reload
sudo ufw reload

Database Maintenance ​

Vacuum (Optimize) Database ​

SQLite databases can become fragmented. Vacuum reclaims space:

bash
# Stop service
sudo systemctl stop admin-bude.service

# Vacuum database
sqlite3 /opt/bud-e/school-bud-e-middleware/admin_bude.db "VACUUM;"

# Restart service
sudo systemctl start admin-bude.service

Frequency: Quarterly, or when database grows significantly.

Benefits:

  • Smaller file size
  • Faster queries
  • Reduced disk I/O

Integrity Check ​

Verify database is not corrupted:

bash
sqlite3 /opt/bud-e/school-bud-e-middleware/admin_bude.db "PRAGMA integrity_check;"

Expected output: ok

If not "ok": Restore from backup immediately.

Performance Tuning ​

Slow Admin UI ​

Possible causes:

  1. Large usage log table
  2. Insufficient server resources
  3. Network latency

Solutions:

1. Archive old logs:

bash
# Export logs older than 90 days to CSV
sqlite3 admin_bude.db <<EOF
.mode csv
.output old_logs.csv
SELECT * FROM usage_logs WHERE timestamp < date('now', '-90 days');
EOF

# Delete old logs from database
sqlite3 admin_bude.db "DELETE FROM usage_logs WHERE timestamp < date('now', '-90 days');"

# Vacuum to reclaim space
sqlite3 admin_bude.db "VACUUM;"

2. Upgrade server:

  • If CPU constantly >80%, add vCPUs
  • If memory consistently >90%, add RAM

3. Add caching (if supported):

  • Check Admin Bud-E docs for caching options
  • Use Redis or similar for session/query caching

Slow Provider Requests ​

Possible causes:

  1. Provider rate limiting
  2. Network latency (server location)
  3. Large requests (images, long prompts)

Solutions:

  • Add failover providers (see Routes)
  • Move server closer to provider region
  • Optimize frontend to send smaller requests

Troubleshooting Common Issues ​

Issue: Service Won't Start ​

Symptoms:

bash
sudo systemctl status admin-bude.service
# Shows: failed

Debugging:

bash
# View detailed logs
sudo journalctl -u admin-bude.service -n 100

# Common errors:
# - "Address already in use" → Another process on port 8000
# - "Database locked" → Old process still running
# - "ModuleNotFoundError" → Missing Python dependency

Solutions:

bash
# Kill process on port 8000
sudo fuser -k 8000/tcp

# Or find and kill
sudo lsof -i :8000
sudo kill <PID>

# Reinstall dependencies
cd /opt/bud-e/school-bud-e-middleware
source .venv/bin/activate
pip install -r requirements.txt

# Try starting manually to see errors
python serve.py --host 0.0.0.0 --port 8000

Issue: Admin UI Login Fails ​

Symptoms: Correct password, but login rejected.

Possible causes:

  1. Password env variable not loaded
  2. Session cookie issue
  3. Database corruption

Solutions:

bash
# Check env file
cat /opt/bud-e/env.sh
# Ensure ADMIN_PASSWORD is set

# Restart service (reloads env)
sudo systemctl restart admin-bude.service

# Clear browser cookies and retry

# If still fails, reset password (requires server access)

Issue: Requests Failing with 5xx Errors ​

Symptoms: Users report "Server error" or 500/503 errors.

Debugging:

  1. Check service status: sudo systemctl status admin-bude.service
  2. View logs: sudo journalctl -u admin-bude.service -n 100
  3. Check provider status (Vertex, Together, etc.)

Common causes:

  • Provider API key expired or invalid
  • Provider service outage
  • Database locked
  • Disk full

Issue: High Server Load ​

Symptoms: Server responds slowly, high CPU/memory.

Check:

bash
# CPU and memory
top

# Disk I/O
iostat -x 1

# Network
iftop

Solutions:

  • Identify heavy process (usually Python)
  • Check for DDoS or abuse (unusual request volume)
  • Scale up server resources
  • Implement rate limiting

Disaster Scenarios ​

Scenario 1: Server Crashed ​

Response:

  1. Provision new server
  2. Install Admin Bud-E
  3. Restore latest backup
  4. Update DNS (if IP changed)
  5. Test functionality

See: Disaster Recovery Plan

Scenario 2: Database Corrupted ​

Response:

  1. Stop service immediately
  2. Attempt integrity check: sqlite3 admin_bude.db "PRAGMA integrity_check;"
  3. If corrupted, restore from latest backup
  4. Investigate cause (disk failure, power loss?)

Scenario 3: API Key Leaked ​

Response:

  1. Rotate compromised key immediately (in Users page)
  2. Check usage logs for unauthorized access
  3. Deduct fraudulent credits (if any)
  4. Inform affected user
  5. Review security practices

Scenario 4: Provider Outage ​

Response:

  1. Check provider status page (Google Cloud Status, etc.)
  2. If prolonged, switch routes to failover provider
  3. Inform users of temporary limitations
  4. Monitor for recovery

Getting Help ​

Self-Help Resources ​

  1. This documentation (start here)
  2. GitHub Issues: school-bud-e-middleware issues
  3. LAION Discord: Community support
  4. Provider documentation:

Reporting Issues ​

When opening a GitHub issue, include:

  1. Description: Clear explanation of the problem
  2. Steps to reproduce: How to trigger the issue
  3. Expected behavior: What should happen
  4. Actual behavior: What actually happens
  5. Environment:
    • Admin Bud-E version
    • Ubuntu version: lsb_release -a
    • Python version: python --version
  6. Logs:
    • Relevant log excerpts (redact sensitive data)
    • sudo journalctl -u admin-bude.service -n 100
  7. Screenshots: If UI issue

WARNING

Never include:

  • API keys
  • Passwords
  • User personal data
  • Full database dumps

Maintenance Checklist Template ​

Print or bookmark this checklist:

Weekly (10 min):

  • [ ] Check service status: sudo systemctl status admin-bude.service
  • [ ] Review usage for anomalies (Usage page)
  • [ ] Verify last backup exists and is non-zero size
  • [ ] Check disk space: df -h (>20% free)

Monthly (1 hour):

  • [ ] Test backup restore in staging environment
  • [ ] Review and refill project budgets
  • [ ] Check for system updates: sudo apt update && sudo apt list --upgradable
  • [ ] Review security logs
  • [ ] Archive or delete old usage logs (>90 days)

Quarterly (2 hours):

  • [ ] Rotate admin password
  • [ ] Update Admin Bud-E (if new version)
  • [ ] Audit user accounts (deactivate inactive)
  • [ ] Review pricing (check provider rates)
  • [ ] Vacuum database

Annually (half day):

  • [ ] Full disaster recovery drill
  • [ ] Security audit
  • [ ] Review data retention policy
  • [ ] Update DPAs with providers
  • [ ] Review and update documentation

Next Steps ​

Admin Bud-E / Bud-E / School Bud-E are developed by LAION e. V. (non-profit). Useful links: Hosted frontend · Middleware repo · Frontend repo · LAION blog