# IGF Admin - Universal MongoDB Data Management UI

A powerful, fully adaptive Streamlit-based admin interface for managing data in MongoDB. The system is designed to work with **any JSON structure**, **any collection**, and **any field schema** without code modifications.

## 🌟 Key Features

### Universal JSON Support
- **Any JSON Structure**: Handles deeply nested objects, arrays, mixed types - unlimited depth
- **No Hardcoded Fields**: Works with ANY field names automatically
- **Automatic Flattening**: Converts nested JSON to flat key-value pairs for tabular editing
- **Lossless Processing**: Every field from source JSON is preserved - zero data loss

### Fully Adaptive Architecture
- **Dynamic Collection Discovery**: Automatically detects available collections
- **Dynamic Field Detection**: Discovers all fields across records automatically
- **Configurable Business Keys**: Define matching rules per collection via YAML config
- **Multiple Matching Strategies**: Primary keys, alternative keys, or combined matching

### Complete Data Control
- **Preview Before Import**: Review all records before any database writes
- **Field-Level Editing**: Edit individual fields in a spreadsheet-like interface
- **Per-Record Decisions**: Import, skip, or mark pending for each record
- **Confirmation Required**: No automatic writes - user must confirm all operations

## 🏗️ Architecture

```
igf-admin/
├── app.py                    # Main Streamlit application
├── config/
│   └── app_config.yaml       # Collection and matching configuration
├── services/
│   ├── json_flattener.py     # Universal JSON flattening (any depth)
│   ├── json_loader.py        # JSON file loading from directories
│   ├── diff_engine.py        # Record comparison and matching
│   ├── import_preview_engine.py  # Import preview and editing
│   ├── data_service.py       # MongoDB data operations
│   ├── mongo.py              # MongoDB connection management
│   ├── settings_service.py   # Configuration management
│   ├── schema_manager.py     # Schema discovery and management
│   ├── update_engine.py      # Record update operations
│   ├── backup_service.py     # Data backup functionality
│   └── foreign_key_validator.py  # Foreign key validation
└── ui/
    └── tabs/
        ├── json_vs_db.py     # JSON vs MongoDB comparison UI
        └── settings.py       # Settings and configuration UI
```

## 🔧 Core Components

### JSONFlattener
The heart of the universal JSON support:
- Recursively flattens ANY nested structure
- Uses dot notation for paths: `parent.child.0.field`
- Handles arrays of objects and primitives
- Supports unflatten for MongoDB insertion
- **No field name assumptions** - works with any schema

### DiffEngine
Intelligent record comparison:
- **Exact Matching**: SHA256 hash comparison
- **Business Key Matching**: Configurable primary keys
- **Alternative Key Matching**: Fallback matching strategy
- **Field-by-Field Comparison**: Identifies all differences

### ImportPreviewEngine
Safe import workflow:
- Builds editable DataFrames from any record structure
- Tracks user decisions (import/skip/pending)
- Prepares records for MongoDB insertion
- Removes metadata fields automatically

## 📋 Configuration

### Collection Configuration (app_config.yaml)

```yaml
collections:
  your_collection:
    enabled: true
    json_folder: folder_name        # Folder in JSON source
    business_keys:
      - field1                      # Primary matching fields
      - field2
    alternative_keys:
      - alt_field1                  # Fallback matching fields
    matching_strategy: primary_or_alternative
    allow_manual_import: true
    description: Your collection description
```

### Matching Strategies
- `primary_only`: Match only by business_keys
- `primary_or_alternative`: Try business_keys first, then alternative_keys
- `alternative_only`: Match only by alternative_keys

## 🚀 Getting Started

### Prerequisites
- **Python 3.10+** (required)
- **Streamlit 1.36.0+** (for button width support)
- MongoDB instance
- JSON data files

### Installation

```bash
# Clone the repository
git clone <repository-url>
cd igf-admin

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your MongoDB URI
```

### Configuration

1. **Set MongoDB URI** in `.env`:
   ```
   MONGO_URI=mongodb://localhost:27017
   ```

2. **Configure collections** in `config/app_config.yaml`:
   - Add your collections
   - Define business keys for duplicate detection
   - Set JSON folder mappings

3. **Place JSON files** in the configured source directory

### Running (Local)

```bash
python -m streamlit run app.py
```

### Running with Docker

```bash
# Build the image
docker build -t igf-admin .

# Run the container
docker run -d -p 8501:8501 \
  -e MONGO_URI=mongodb://your-host:27017 \
  -v $(pwd)/config:/app/config \
  -v $(pwd)/idf-json-sample:/app/idf-json-sample \
  igf-admin
```

Access the application at: http://localhost:8501

## 📊 Usage Workflow

### JSON vs DB Comparison

1. **Select Source**: Choose JSON source folder (e.g., tokyo_2020)
2. **Select Collection**: Choose target collection (filtered by available data)
3. **Load & Compare**: System compares JSON records with MongoDB
4. **Review Results**:
   - **Exact Matches**: Records identical in both sources
   - **Duplicates**: Records with same business key but different values
   - **New Records**: Records only in JSON, not in MongoDB
5. **Make Decisions**: Mark records for import or skip
6. **Apply Imports**: Confirm and execute the import

### Settings

- Configure business keys per collection
- Set alternative keys for fallback matching
- Define record limits for comparison
- Configure foreign key relationships

## 🔒 Safety Features

- **No Automatic Writes**: All imports require explicit user confirmation
- **Preview Everything**: See exactly what will be imported
- **Reversible Decisions**: Change import/skip decisions before applying
- **Backup Support**: Create backups before major operations
- **Foreign Key Validation**: Validate references before import

## 🎯 Adaptability Examples

### Example 1: Simple Flat JSON
```json
{"id": "001", "name": "John", "age": 30}
```
Automatically handled - fields appear as columns.

### Example 2: Nested Objects
```json
{
  "person": {
    "name": {"first": "John", "last": "Doe"},
    "contact": {"email": "john@example.com"}
  }
}
```
Flattened to: `person.name.first`, `person.name.last`, `person.contact.email`

### Example 3: Arrays of Objects
```json
{
  "participants": [
    {"id": "001", "score": 95},
    {"id": "002", "score": 87}
  ]
}
```
Flattened to: `participants.0.id`, `participants.0.score`, `participants.1.id`, etc.

### Example 4: Mixed Complex Structure
```json
{
  "event": {
    "name": "Olympics",
    "athletes": [
      {
        "person": {"name": "John"},
        "results": [{"round": 1, "score": 9.5}]
      }
    ]
  }
}
```
All fields preserved with full path notation.

## 📝 Adding New Collections

1. Add collection to `app_config.yaml`:
   ```yaml
   collections:
     new_collection:
       enabled: true
       json_folder: new_folder
       business_keys:
         - unique_field
       allow_manual_import: true
   ```

2. Place JSON files in `{json_source}/{json_folder}/`

3. Restart the application - new collection appears automatically

**No code changes required!**

## 🛠️ Technical Details

### Data Flow
1. JSON files → JSONLoader → Flattened records
2. MongoDB documents → DataService → Flattened records
3. DiffEngine compares both sets
4. ImportPreviewEngine builds editable preview
5. User makes decisions
6. Records unflattened and inserted to MongoDB

### Type Handling
- ObjectId → String (for display)
- DateTime → ISO String
- Nested Dict → Flattened with dot notation
- Arrays → Indexed with numbers (e.g., `items.0.name`)
- Null values → Preserved

## 📄 License

[Your License Here]

## 🤝 Contributing

Contributions welcome! The system is designed for extensibility:
- Add new services in `/services`
- Add new UI tabs in `/ui/tabs`
- Extend configuration in `app_config.yaml`