EB_Dashboard/DOCUMENTATION/DOCUMENTATION_98_USER_GUIDE.md

# Endobest Dashboard - User Guide

**Document Version:** 1.0
**Last Updated:** 2025-11-08
**Audience:** End Users, Clinical Staff, Data Analysts
**Language:** English

---

## Quick Start

The Endobest Dashboard collects patient inclusion data and generates reports. It supports three modes:

### Mode 1: Normal Collection (Full Process)
```bash
python eb_dashboard.py
```
- Authenticates with server
- Collects data from all organizations
- Runs quality checks
- Generates JSON files
- Exports to Excel (if configured)
- Takes 5-30 minutes depending on data volume

### Mode 2: Excel Export Only (Fast)
```bash
python eb_dashboard.py --excel-only
```
- Skips data collection
- Uses existing JSON data files
- Generates Excel reports
- Takes 5-15 seconds
- **Useful for:** Reconfiguring reports, testing templates, regenerating with new rules

### Mode 3: Quality Checks Only
```bash
python eb_dashboard.py --check-only
```
- Loads existing JSON data
- Runs quality checks
- No export
- **Useful for:** Verifying data integrity, debugging

---

## What Happens When You Run the Script

### Normal Mode Flow

```
1. Login (provides credentials)
   ↓
2. Configuration Loading (reads Excel config file)
   ↓
3. Data Collection (fetches from servers)
   ├─ Progress bar shows collection status
   ├─ Logs timing and results
   └─ Can be cancelled with Ctrl+C
   ↓
4. Quality Checks (validates data consistency)
   ├─ Coherence checks (API counts vs actual data)
   ├─ Regression checks (compares with previous run)
   └─ May ask for confirmation if issues found
   ↓
5. JSON Export (writes data to files)
   ├─ endobest_inclusions.json
   └─ endobest_organizations.json
   ↓
6. Excel Export (generates workbooks)
   ├─ Reads configuration
   ├─ Applies filters/sorts
   └─ Creates Excel files
   ↓
7. Done (shows elapsed time)
```

### --excel-only Mode Flow

```
1. Load existing JSON files
   ↓
2. Configuration Loading (reads Excel config)
   ↓
3. Excel Export (very fast)
   ↓
4. Done
```

---

## What Gets Created

### Output Files

**JSON Files (always created):**
- `endobest_inclusions.json` - Patient inclusion data
- `endobest_organizations.json` - Organization statistics

**Excel Files (if configured and enabled):**
- Named according to configuration (e.g., `Endobest_Output_2025-01-15.xlsx`)
- Multiple files if multiple workbooks configured
- Created in current directory

**Log File:**
- `dashboard.log` - Detailed execution log

### File Locations

All files created in the directory where you run the script:
```
current_directory/
├── endobest_inclusions.json
├── endobest_organizations.json
├── endobest_organizations.json_old (backup)
├── Endobest_Output_2025-01-15.xlsx
├── Statistics_2025-01.xlsx
└── dashboard.log
```

---

## Configuration

Configuration is done in Excel (no code changes needed):

**File:** `config/Endobest_Dashboard_Config.xlsx`

**Two tables to configure Excel export:**

### Table 1: Excel_Workbooks
Defines WHAT files to create:

| Column | Example | Meaning |
|--------|---------|---------|
| workbook_name | Endobest_Output | Identifier for this workbook |
| template_path | templates/Endobest_Template.xlsx | Which template to use |
| output_filename | {workbook_name}_{extract_date_time}.xlsx | What to name the file |
| output_exists_action | Increment | What to do if file already exists |

**output_exists_action options:**
- `Overwrite`: Replace existing file
- `Increment`: Create _1, _2, etc. if file exists
- `Backup`: Rename existing to _backup_1, etc.

### Table 2: Excel_Sheets
Defines HOW to fill each sheet:

| Column | Example | Meaning |
|--------|---------|---------|
| workbook_name | Endobest_Output | Which workbook (must match Table 1) |
| sheet_name | Inclusions | Which sheet in the template |
| source_type | Inclusions | Where data comes from (Inclusions / Organizations / Variable) |
| target | DataTable | Where to put data (cell or named range) |
| column_mapping | {"col_id": "patient_id", "col_name": "name"} | Column definitions (JSON) |
| filter_condition | {"status": "active"} | Filter data (JSON) |
| sort_keys | [{"field": "date", "order": "asc"}] | Sort data (JSON) |
| value_replacement | [{"type": "bool", "true": "Yes", "false": "No"}] | Replace values (JSON) |

---

## Common Tasks

### Task 1: Run Collection and Export
```bash
python eb_dashboard.py
# Full run: collect → checks → JSON → Excel
```

### Task 2: Re-export Excel with New Configuration
```bash
# Edit config in Excel
python eb_dashboard.py --excel-only
# Fast! Only regenerates Excel files
```

### Task 3: Check Data Quality
```bash
python eb_dashboard.py --check-only
# Just run quality checks on existing data
```

### Task 4: Debug Issues
```bash
python eb_dashboard.py --debug
# Shows verbose logging
# Check dashboard.log for details
```

---

## FAQ

### Q: How long does data collection take?
**A:** Depends on organizations and data volume:
- Small (50 organizations): 5-10 minutes
- Medium (100+ organizations): 10-20 minutes
- Large (200+ organizations): 20-30 minutes

### Q: Can I stop the script?
**A:** Yes, press Ctrl+C to cancel. Already-collected data is not lost.

### Q: What if the script fails?
**A:** Check `dashboard.log` for error details. Most common issues:
- Network connection lost (retry)
- Invalid credentials (re-enter)
- Configuration error (fix Excel config file)

### Q: Where are the Excel templates?
**A:** In `config/templates/` folder. Configure which template to use in the Excel config file.

### Q: Can I modify the Excel files after generation?
**A:** Yes! But running the script again may overwrite them. Use "Backup" or "Increment" action to prevent overwriting.

### Q: How do I filter the data?
**A:** Set `filter_condition` in the Excel_Sheets table. Example:
```json
{"status": "active", "visit_type": "inclusion"}
```
Only includes items matching ALL conditions.

### Q: How do I sort the data?
**A:** Set `sort_keys` in the Excel_Sheets table. Example:
```json
[
  {"field": "date_visit", "order": "desc"},
  {"field": "patient_name", "order": "asc"}
]
```

### Q: What's "extract_date_time"?
**A:** Timestamp when data was extracted, automatically inserted into filenames:
```
Endobest_Output_2025-01-15T14-30-45.xlsx
```
Use `{extract_date_time}` in your output filename template.

### Q: Excel formulas aren't calculating. Why?
**A:** Formulas only recalculate if win32com is installed. Data is correct; formulas just aren't auto-updating. To fix:
- Install: `pip install pywin32`
- Run script again

### Q: Can I have multiple workbooks?
**A:** Yes! Add multiple rows to Excel_Workbooks table with different configurations.

---

## Troubleshooting

### "Login failed"
- Check credentials (username/password)
- Check internet connection
- Check firewall/proxy settings

### "Template file not found"
- Check template file exists in `config/templates/`
- Check path spelling matches exactly

### "Named range not found"
- Check range name in template (Excel: Formulas → Name Manager)
- Check spelling matches exactly

### "No data in Excel"
- Check filter_condition isn't too restrictive
- Check filter_condition is valid JSON
- Run `--check-only` to verify data exists

### "Excel file won't open"
- Check if file is open in another program
- Try deleting and regenerating (using --excel-only mode)
- Check dashboard.log for errors

---

## Performance Tips

**Slow Collection?**
- Increase `Number of threads` (up to 20, default 12)
- Close other programs using network

**Slow Excel Generation?**
- Smaller datasets = faster
- Use filters to reduce data volume
- Disable formula recalculation (remove win32com requirement)

---

## Support

### Check Log File
```bash
# View last 50 lines
tail -n 50 dashboard.log

# View errors only
grep ERROR dashboard.log
```

### Get Help
1. Check this document (FAQ/Troubleshooting sections)
2. Check dashboard.log for details
3. Run with `--debug` for more information

---

## Important Notes

- **Data Privacy:** Patient data in JSON files - handle appropriately
- **Backup Files:** Script creates `.json_old` backups - you can delete if needed
- **Configuration:** Changes in Excel config take effect on next run
- **Templates:** Modify Excel templates in `config/templates/` folder (admin only)

---

**For technical details and architecture, see DOCUMENTATION_13_EXCEL_EXPORT.md**