# 📊 Endobest Dashboard - Visual Flowcharts & Diagrams

**Visual Reference for Understanding System Flow**

---

## 1️⃣ Main Execution Flow

```
START
  │
  ├─→ [USER INPUT]
  │   ├─ Credentials (login)
  │   ├─ Number of threads (1-20)
  │   └─ Execution mode (normal/excel_only/check_only)
  │
  ├─→ [AUTHENTICATION]
  │   ├─ IAM Login: POST /api/auth/ziwig-pro/login
  │   ├─ Token Exchange: POST /api/auth/config-token
  │   └─ ✅ Get access_token + refresh_token
  │
  ├─→ [CONFIGURATION LOADING]
  │   ├─ Load Endobest_Dashboard_Config.xlsx
  │   ├─ Parse 5 sheets:
  │   │  ├─ Inclusions_Mapping
  │   │  ├─ Organizations_Mapping
  │   │  ├─ Excel_Workbooks
  │   │  ├─ Excel_Sheets
  │   │  └─ Regression_Check
  │   └─ ✅ Validate all configs
  │
  ├─→ [PHASE 2: ORG + COUNTERS] (5-8 sec)
  │   ├─ GET /api/inclusions/getAllOrganizations
  │   ├─ POST /api/inclusions/inclusion-statistics (20 workers)
  │   ├─ Optional: Load eb_org_center_mapping.xlsx
  │   └─ ✅ Organizations with counts ready
  │
  ├─→ [PHASE 3: DATA COLLECTION] (2-4 min)
  │   ├─ Outer Loop: 20 workers per organization
  │   │  ├─ GET /api/inclusions/search
  │   │  └─ For each patient (sequential):
  │   │     ├─ POST /api/records/byPatient
  │   │     ├─ POST /api/surveys/filter/with-answers
  │   │     ├─ Submit: GET /api/requests/by-tube-id (async)
  │   │     └─ Process: Field mapping + transformation
  │   └─ ✅ All inclusion data collected
  │
  ├─→ [PHASE 4: QUALITY CHECKS] (10-15 sec)
  │   ├─ Coherence Check:
  │   │  └─ Compare API stats vs actual data
  │   ├─ Non-Regression Check:
  │   │  └─ Compare current vs previous run
  │   └─ Decision Point:
  │      ├─ ✅ No critical issues → Continue
  │      └─ ❌ Critical issues → Prompt user
  │
  ├─→ [USER CONFIRMATION]
  │   └─ "Critical issues detected. Write anyway? [Y/N]"
  │      ├─ YES → Continue to export
  │      └─ NO → Exit (files NOT modified)
  │
  ├─→ [PHASE 5: EXPORT]
  │   ├─ Backup old files
  │   ├─ Write JSON outputs
  │   │  ├─ endobest_inclusions.json
  │   │  └─ endobest_organizations.json
  │   ├─ Generate Excel (if configured)
  │   │  ├─ Load templates
  │   │  ├─ Apply filters/sorts
  │   │  ├─ Fill data
  │   │  └─ Recalculate formulas
  │   └─ ✅ All files written
  │
  ├─→ [COMPLETION]
  │   ├─ Display elapsed time
  │   ├─ Log summary
  │   └─ Press Enter to exit
  │
  END
```

---

## 2️⃣ Data Collection Detail (Phase 3)

```
[PHASE 3: DATA COLLECTION]
  │
  ├─ Setup: Thread Pools
  │  ├─ Main Pool: ThreadPoolExecutor(N workers, max 20)
  │  └─ Async Pool: ThreadPoolExecutor(40 workers)
  │
  ├─ Outer Loop: FOR each Organization (20 workers)
  │  │
  │  ├─ [1] GET /api/inclusions/search
  │  │  └─ Response: list of patients for this org
  │  │
  │  ├─ For Each Patient (Sequential):
  │  │  │
  │  │  ├─ [2] POST /api/records/byPatient
  │  │  │  └─ Response: clinical_record
  │  │  │
  │  │  ├─ [3] POST /api/surveys/filter/with-answers ⚡ OPTIMIZED
  │  │  │  └─ Response: all questionnaires + answers in 1 call
  │  │  │
  │  │  ├─ [4] Submit async task (Async Pool):
  │  │  │  │   GET /api/requests/by-tube-id/{tubeId}
  │  │  │  │   └─ Fetched in background while main thread processes
  │  │  │  │
  │  │  │  └─ [CONTINUE] Process Field Mappings:
  │  │  │     │
  │  │  │     ├─ For Each Field in Config:
  │  │  │     │  ├─ Determine Source
  │  │  │     │  │  ├─ q_id/q_name/q_category → Find questionnaire
  │  │  │     │  │  ├─ record → Use clinical record
  │  │  │     │  │  ├─ inclusion → Use patient data
  │  │  │     │  │  ├─ request → Use lab request
  │  │  │     │  │  └─ calculated → Execute function
  │  │  │     │  │
  │  │  │     │  ├─ Extract Value (JSON path + wildcard support)
  │  │  │     │  ├─ Check Condition (if defined)
  │  │  │     │  ├─ Apply Transformations:
  │  │  │     │  │  ├─ true_if_any (convert to boolean)
  │  │  │     │  │  ├─ value_labels (map to text)
  │  │  │     │  │  ├─ field_template (format)
  │  │  │     │  │  └─ List joining (flatten arrays)
  │  │  │     │  │
  │  │  │     │  └─ Store: output_inclusion[group][field] = value
  │  │  │     │
  │  │  │     ├─ Wait: Lab request from async pool
  │  │  │     └─ Final: Complete inclusion record ready
  │  │  │
  │  │  ├─ [5] Update Progress Bars
  │  │  │  ├─ Organization progress
  │  │  │  ├─ Overall progress
  │  │  │  └─ (Thread-safe with lock)
  │  │  │
  │  │  └─ [NEXT PATIENT]
  │  │
  │  └─ [NEXT ORGANIZATION]
  │
  └─ Combine All Inclusions
     └─ ✅ All data collected
```

---

## 3️⃣ Field Processing Pipeline

```
[INPUT: Configuration field entry]
         │
         ├─ Field Name: "Patient_Age"
         ├─ Source Type: "calculated"
         ├─ Field Path: ["extract_age_from_birthday", "Patient_Birthday"]
         ├─ Condition: undefined
         ├─ Value Labels: undefined
         ├─ Field Template: undefined
         └─ True If Any: undefined
         │
         ▼
[STEP 1: Determine Source]
         │
         ├─ Is source "calculated"?
         │  └─ YES → Call custom function
         │          execute_calculated_field(
         │            "extract_age_from_birthday",
         │            ["Patient_Birthday"]
         │          )
         │
         ▼
[STEP 2: Extract Raw Value]
         │
         ├─ Get function result
         ├─ raw_value = 49 (calculated)
         │
         ▼
[STEP 3: Check Condition]
         │
         ├─ Condition field: undefined
         ├─ Action: Skip condition check
         │
         ▼
[STEP 4: Apply Transformations]
         │
         ├─ true_if_any: undefined (skip)
         ├─ value_labels: undefined (skip)
         ├─ field_template: undefined (skip)
         ├─ List joining: N/A (not a list)
         │
         ▼
[STEP 5: Format Score Dictionary]
         │
         ├─ Is value dict with {total, max}?
         │  └─ NO → Keep as 49
         │
         ▼
[OUTPUT: final_value = 49]
         │
         └─ Store: output_inclusion["Patient_Identification"]["Patient_Age"] = 49
```

---

## 4️⃣ Questionnaire Optimization Impact

```
SCENARIO: Patient with 15 questionnaires

❌ OLD APPROACH (SLOW):
   │
   ├─ Loop: for each questionnaire ID
   │  ├─ API Call 1: GET /api/surveys/qcm_1/answers?subject=patient_id
   │  ├─ API Call 2: GET /api/surveys/qcm_2/answers?subject=patient_id
   │  ├─ API Call 3: GET /api/surveys/qcm_3/answers?subject=patient_id
   │  └─ ... (15 calls total)
   │
   ├─ Per Patient: 15 API calls
   ├─ Total (1200 patients): 18,000 API calls
   ├─ Estimated Time: 15-30 minutes
   └─ Result: SLOW ⏳
   │

✅ NEW APPROACH (FAST - OPTIMIZED):
   │
   ├─ Single API Call:
   │  │
   │  └─ POST /api/surveys/filter/with-answers
   │     {
   │       "context": "clinic_research",
   │       "subject": patient_id
   │     }
   │     ↓
   │     Response: [
   │       {questionnaire: {...}, answers: {...}},
   │       {questionnaire: {...}, answers: {...}},
   │       ... (all 15 questionnaires in 1 response)
   │     ]
   │
   ├─ Per Patient: 1 API call
   ├─ Total (1200 patients): 1,200 API calls
   ├─ Estimated Time: 2-5 minutes
   └─ Result: FAST ⚡ (4-5x improvement)
```

---

## 5️⃣ Multithreading Architecture

```
MAIN THREAD
    │
    ├─────────────────────────────────────────────────┐
    │  PHASE 2: Counter Fetching (Sequential Wait)   │
    │                                                 │
    │  Thread Pool: 20 workers                        │
    │  ├─ Worker 1: Fetch counters for Org 1         │
    │  ├─ Worker 2: Fetch counters for Org 2         │
    │  ├─ Worker 3: Fetch counters for Org 3         │
    │  └─ ...                                         │
    │  │                                              │
    │  └─ Wait: tqdm.as_completed() [All done]       │
    │                                                 │
    └─────────────────────────────────────────────────┘
           │ (Sequential barrier)
           ▼
    ┌─────────────────────────────────────────────────┐
    │  PHASE 3: Inclusion Collection (Nested)        │
    │                                                 │
    │  Outer Thread Pool: 20 workers (Orgs)          │
    │  │                                              │
    │  ├─ Worker 1: Process Org 1                    │
    │  │  ├─ GET /api/inclusions/search             │
    │  │  └─ For each patient:                       │
    │  │     ├─ Process clinical record              │
    │  │     ├─ Process questionnaires               │
    │  │     └─ [ASYNC] Submit lab fetch:            │
    │  │        │                                     │
    │  │        └─ Inner Pool (40 workers)           │
    │  │           ├─ Worker A: Fetch lab for Pat1   │
    │  │           ├─ Worker B: Fetch lab for Pat2   │
    │  │           └─ ...                            │
    │  │                                              │
    │  ├─ Worker 2: Process Org 2 [same pattern]    │
    │  ├─ Worker 3: Process Org 3 [same pattern]    │
    │  └─ ...                                        │
    │  │                                              │
    │  └─ Wait: tqdm.as_completed() [All done]       │
    │                                                 │
    └─────────────────────────────────────────────────┘
           │ (Sequential barrier)
           ▼
    ┌─────────────────────────────────────────────────┐
    │  PHASE 4: Quality Checks (Sequential)          │
    │  ├─ Coherence check (single-threaded)          │
    │  ├─ Non-regression check (single-threaded)     │
    │  └─ Results ready                              │
    │                                                 │
    └─────────────────────────────────────────────────┘
           │
           ▼
    ┌─────────────────────────────────────────────────┐
    │  PHASE 5: Export (Sequential)                  │
    │  ├─ Write JSON files                           │
    │  ├─ Generate Excel (single-threaded)           │
    │  └─ Done                                       │
    │                                                 │
    └─────────────────────────────────────────────────┘
```

---

## 6️⃣ Quality Checks Logic

```
[INPUT: Current data, Previous data, Config rules]
         │
         ▼
    ┌─────────────────────────────────────────┐
    │  COHERENCE CHECK                        │
    ├─────────────────────────────────────────┤
    │                                         │
    │  For each organization:                │
    │    ├─ API Count = from statistics      │
    │    ├─ Actual Count = from inclusions   │
    │    └─ Compare:                         │
    │       ├─ Δ ≤ 10% → Warning            │
    │       └─ Δ > 10% → CRITICAL ⚠️        │
    │                                         │
    │  Result: has_coherence_critical        │
    │                                         │
    └─────────────────────────────────────────┘
             │
             ▼
    ┌─────────────────────────────────────────┐
    │  NON-REGRESSION CHECK                   │
    ├─────────────────────────────────────────┤
    │                                         │
    │  For each regression rule:             │
    │    ├─ Load previous data                │
    │    ├─ Build field selection (pipeline)  │
    │    ├─ Find key field for matching       │
    │    └─ For each inclusion:               │
    │       ├─ Match with previous record     │
    │       ├─ Check transitions              │
    │       ├─ Apply exceptions               │
    │       └─ Report violations:             │
    │          ├─ ⚠️  Warning                 │
    │          └─ 🔴 CRITICAL                │
    │                                         │
    │  Result: has_regression_critical       │
    │                                         │
    └─────────────────────────────────────────┘
             │
             ▼
    ┌─────────────────────────────────────────┐
    │  DECISION POINT                         │
    ├─────────────────────────────────────────┤
    │                                         │
    │  If has_coherence_critical OR          │
    │     has_regression_critical:           │
    │    │                                    │
    │    └─→ [PROMPT USER]                   │
    │        "Critical issues detected!       │
    │         Write results anyway? [Y/N]"   │
    │        ├─ YES → Proceed to export     │
    │        └─ NO → Cancel, exit            │
    │  Else:                                 │
    │    └─→ [PROCEED] to export             │
    │                                         │
    └─────────────────────────────────────────┘
             │
             ▼
    [EXPORT PHASE]
```

---

## 7️⃣ Excel Export Pipeline

```
[INPUT: Inclusions data, Organizations data, Config]
         │
         ▼
    ┌──────────────────────────────────────────────────┐
    │  LOAD CONFIGURATION                              │
    ├──────────────────────────────────────────────────┤
    │                                                  │
    │  Excel_Workbooks table:                         │
    │  ├─ workbook_name: "Endobest_Output"            │
    │  ├─ template_path: "templates/Endobest.xlsx"    │
    │  ├─ output_filename: "{name}_{date_time}.xlsx"  │
    │  └─ output_exists_action: "Increment"           │
    │                                                  │
    │  Excel_Sheets table:                            │
    │  ├─ workbook_name: "Endobest_Output"            │
    │  ├─ sheet_name: "Inclusions"                    │
    │  ├─ source_type: "Inclusions"                   │
    │  ├─ target: "DataTable"                         │
    │  ├─ column_mapping: JSON                        │
    │  ├─ filter_condition: JSON                      │
    │  ├─ sort_keys: JSON                             │
    │  └─ value_replacement: JSON                     │
    │                                                  │
    └──────────────────────────────────────────────────┘
             │
             ▼
    ┌──────────────────────────────────────────────────┐
    │  FOR EACH WORKBOOK                               │
    ├──────────────────────────────────────────────────┤
    │                                                  │
    │  [1] Load template from Excel file               │
    │  │                                               │
    │  [2] FOR EACH SHEET IN WORKBOOK                 │
    │  │   │                                           │
    │  │   ├─ Determine data source                   │
    │  │   │  ├─ Inclusions → Use patient data       │
    │  │   │  ├─ Organizations → Use org stats       │
    │  │   │  └─ Variable → Use template variables    │
    │  │   │                                           │
    │  │   ├─ [FILTER] Apply AND conditions          │
    │  │   │  └─ Keep only rows matching all filters  │
    │  │   │                                           │
    │  │   ├─ [SORT] Apply multi-key sorting          │
    │  │   │  ├─ Primary: Organization Name (desc)    │
    │  │   │  ├─ Secondary: Date (asc)                │
    │  │   │  └─ Tertiary: Patient Name (asc)         │
    │  │   │                                           │
    │  │   ├─ [REPLACE] Apply value transformations   │
    │  │   │  ├─ Boolean: true→"Yes", false→"No"      │
    │  │   │  ├─ Status codes: 1→"Active", 0→"Inactive" │
    │  │   │  └─ Enum values: Mapped to display text   │
    │  │   │                                           │
    │  │   ├─ [FILL] Write data to sheet              │
    │  │   │  ├─ Load column mapping                  │
    │  │   │  ├─ For each row in filtered+sorted data│
    │  │   │  │  └─ Write to row in Excel            │
    │  │   │  └─ Fill target (cell or named range)    │
    │  │   │                                           │
    │  │   └─ [NEXT SHEET]                           │
    │  │                                               │
    │  [3] Handle file conflicts                      │
    │  │   ├─ Overwrite: Replace existing file        │
    │  │   ├─ Increment: Create _1, _2, _3 variants   │
    │  │   └─ Backup: Rename existing to _backup      │
    │  │                                               │
    │  [4] Save workbook (openpyxl)                   │
    │  │                                               │
    │  [5] Recalculate formulas (win32com) [OPTIONAL] │
    │  │   ├─ If win32com available                   │
    │  │   ├─ Open in Excel                           │
    │  │   ├─ Force recalculation                      │
    │  │   ├─ Save                                     │
    │  │   └─ Close                                    │
    │  │                                               │
    │  └─ [NEXT WORKBOOK]                             │
    │                                                  │
    └──────────────────────────────────────────────────┘
             │
             ▼
    [OUTPUT: Excel files created ✅]
```

---

## 8️⃣ Error Handling & Recovery

```
[API CALL]
    │
    ▼
[TRY] Execute HTTP request
    │
    ├─ SUCCESS (200-299)
    │  └─→ Return response
    │
    └─ FAILURE
       │
       ├─ HTTP 401 (Unauthorized)
       │  │
       │  ├─→ Lock acquired (token_refresh_lock)
       │  ├─→ new_token(): Refresh token
       │  │   POST /api/auth/refreshToken
       │  │   └─ Update global tokens
       │  ├─→ Lock released
       │  └─→ RETRY the original request
       │
       ├─ Network Error (timeout, connection refused, etc.)
       │  │
       │  ├─→ Log warning with attempt number
       │  ├─→ RETRY after WAIT_BEFORE_RETRY seconds
       │  └─→ Increment attempt counter
       │
       └─ Other HTTP Error (4xx, 5xx)
          │
          ├─→ Log warning with attempt number
          ├─→ RETRY after WAIT_BEFORE_RETRY seconds
          └─→ Increment attempt counter

[LOOP: Repeat for ERROR_MAX_RETRY times (10 attempts)]
    │
    ├─ Attempt 1, 2, 3, ..., 9
    │  └─→ If any succeeds: Return response
    │
    └─ Attempt 10
       │
       ├─ Still failing?
       │  │
       │  └─→ Log CRITICAL error
       │      └─→ Raise exception (propagate to main)
       │
       └─ Main catches exception
          ├─→ Display error message
          ├─→ Log to dashboard.log
          └─→ Exit gracefully
```

---

## 9️⃣ Configuration-Driven Execution

```
┌─────────────────────────────────────────────┐
│  ENDOBEST_DASHBOARD_CONFIG.XLSX             │
├─────────────────────────────────────────────┤
│                                             │
│  Sheet 1: Inclusions_Mapping                │
│  ├─ Row 1: Headers                          │
│  ├─ Row 2+: Field definitions               │
│  │  ├─ field_group: "Patient_Identification" │
│  │  ├─ field_name: "Patient_Id"             │
│  │  ├─ source_id: "q_name"                  │
│  │  ├─ source_value: "Demographics"         │
│  │  ├─ field_path: ["patient_id"]           │
│  │  └─ ... (more transformations)           │
│  │                                          │
│  Sheet 2: Organizations_Mapping              │
│  │  ├─ Defines org fields                    │
│  │  └─ Rarely modified                       │
│  │                                          │
│  Sheet 3: Excel_Workbooks                    │
│  │  ├─ Workbook metadata                     │
│  │  ├─ Template references                   │
│  │  └─ Output filename templates             │
│  │                                          │
│  Sheet 4: Excel_Sheets                       │
│  │  ├─ Sheet configurations                  │
│  │  ├─ Data transformation rules             │
│  │  └─ Filters, sorts, replacements          │
│  │                                          │
│  Sheet 5: Regression_Check                   │
│  │  ├─ Quality check rules                   │
│  │  ├─ Field selection pipelines             │
│  │  ├─ Transition patterns                   │
│  │  └─ Severity levels                       │
│  │                                          │
└─────────────────────────────────────────────┘
         │
         │ [APPLICATION LOADS]
         ▼
    ┌─────────────────────────────────────────┐
    │  LOADED IN MEMORY                       │
    ├─────────────────────────────────────────┤
    │                                         │
    │  INCLUSIONS_MAPPING_CONFIG = [...]      │
    │  REGRESSION_CHECK_CONFIG = [...]        │
    │  EXCEL_EXPORT_CONFIG = {...}            │
    │                                         │
    │  [USED BY]                              │
    │  ├─ Field extraction (all fields)        │
    │  ├─ Quality checks (regression rules)    │
    │  └─ Excel generation (workbook config)   │
    │                                         │
    └─────────────────────────────────────────┘
         │
         │ [CHANGES = AUTOMATIC ON NEXT RUN]
         │
         └─→ NO CODE RECOMPILATION NEEDED ✅
            └─→ NO RESTART NEEDED ✅
```

---

## 🔟 File I/O & Backup Strategy

```
[INITIAL STATE]
  ├─ endobest_inclusions.json (today's data)
  ├─ endobest_inclusions_old.json (yesterday's data)
  ├─ endobest_organizations.json (today's stats)
  └─ endobest_organizations_old.json (yesterday's stats)

[EXECUTION STARTS]
  │
  ├─ [Collect new data]
  ├─ [Run quality checks]
  └─ [If quality checks passed]
     │
     └─→ [Backup old → old.old]
         ├─ endobest_inclusions.json → endobest_inclusions_old.json
         └─ endobest_organizations.json → endobest_organizations_old.json
         │
         └─→ [Write new files]
             ├─ NEW endobest_inclusions.json (written)
             └─ NEW endobest_organizations.json (written)
             │
             └─→ [COMPLETION]

[FINAL STATE]
  ├─ endobest_inclusions.json (NEW data)
  ├─ endobest_inclusions_old.json (TODAY's data, now old)
  ├─ endobest_organizations.json (NEW stats)
  └─ endobest_organizations_old.json (TODAY's stats, now old)

[IF CRITICAL ISSUES]
  │
  ├─ [Skip backup & write]
  ├─ [Old files preserved]
  ├─ [User prompted: write anyway?]
  │  ├─ YES → Write new files (override)
  │  └─ NO → Abort, keep old files
  │
  └─ [No data loss]
```

---

## File Format Examples

### endobest_inclusions.json Structure
```
[
  {
    "Patient_Identification": {
      "Organisation_Id": "...",
      "Organisation_Name": "...",
      "Center_Name": "..." (from mapping),
      "Patient_Id": "...",
      "Pseudo": "...",
      "Patient_Name": "...",
      "Patient_Birthday": "...",
      "Patient_Age": ...
    },
    "Inclusion": {
      "Consent_Signed": true/false,
      "Inclusion_Date": "...",
      "Inclusion_Status": "...",
      ...
    },
    "Extended_Fields": {
      "Custom_Field_1": "...",
      "Custom_Field_2": ...,
      ...
    },
    "Endotest": {
      "Request_Sent": true/false,
      "Diagnostic_Status": "...",
      ...
    }
  }
]
```

### endobest_organizations.json Structure
```
[
  {
    "id": "org-uuid",
    "name": "Hospital Name",
    "Center_Name": "HOSP-A" (from mapping),
    "patients_count": 45,
    "preincluded_count": 8,
    "included_count": 35,
    "prematurely_terminated_count": 2
  }
]
```

---

**All diagrams above show the complete system flow from start to finish.**

**For detailed implementation, see technical documentation files.**