o ג[i^+@sZdZddlZddlZddlmZddlmZmZmZm Z m Z ddl m Z GdddZ dS)aZ JSON Loader Service This module loads JSON records from the ODF directory structure. It respects folder hierarchy and extracts records relevant to selected collections. Key principles: - NO normalization - preserve all fields as-is - NO field removal - keep all keys - Support multiple JSON files per collection - Respect ODF folder structure N)Path)ListDictAnyOptionalTuple) JSONFlattenerc @s eZdZdZdedeeeffddZdeefddZ d ede efd d Z d edeefd d Z de de eeeffddZdeeefd ede deeeffddZd ededeeeeffddZd ededefddZd edeeeffddZd ededeeeffddZdS) JSONLoaderz Service for loading JSON records from ODF directory structure. Loads JSON files while preserving all original fields and structure. All mappings are read from configuration. base_pathcollections_configcCst||_||_t|_dS)z Initialize JSON loader with base path and collections config. Args: base_path: Base path to JSON files (e.g., 'idf-json-sample') collections_config: Collections configuration from app_config.yaml N)rr r r flattener)selfr r r0/var/www/html/IGF-ODF-V3/services/json_loader.py__init__#s  zJSONLoader.__init__returncCs>|jsgSg}|jD] }|r||jqt|S)z Get list of available JSON source directories. Returns: List of directory names (e.g., ['tokyo_2020', 'paris_2024']) )r existsiterdiris_dirappendnamesorted)r sourcesitemrrrget_available_sources/s  z JSONLoader.get_available_sourcescollection_namecCs|j|i}|dS)z Get the ODF folder name for a collection from config. Args: collection_name: Name of the collection Returns: Folder name or None if not configured json_folderr get)r rcollection_configrrrget_collection_folder@s z JSONLoader.get_collection_foldercCsgS)a Get expected document types for a collection. NOTE: With folder-based validation, we no longer track specific document types. This method now returns an empty list, indicating validation is folder-based. Args: collection_name: Name of the collection Returns: Empty list (validation is now folder-based, not type-based) r)r rrrrget_expected_document_typesMs z&JSONLoader.get_expected_document_types file_pathc Cs|z t|ddd}t|WdWS1swYWdSty=}ztd|d|WYd}~dSd}~ww)z Load and parse a single JSON file. Args: file_path: Path to JSON file Returns: Parsed JSON data or None if error rzutf-8)encodingNzError loading z: )openjsonload Exceptionprint)r r"ferrrload_json_file\s (zJSONLoader.load_json_file json_datac Cs\|j|i}|d}|sdS|jj}||kr,d|jd|d|d|d }d|fSdS) ab Validate that JSON file is in the correct folder for the target collection. NEW APPROACH: Validation based on folder structure, not document_type. Files in the correct folder are automatically valid, regardless of document_type. This makes the system flexible and independent of specific document type values. Args: json_data: Parsed JSON data collection_name: Target collection name file_path: Path to JSON file (for validation) Returns: Tuple of (is_valid, error_message) r)TzVALIDATION ERROR: File 'z' is in folder 'z' but collection 'z' expects files from folder 'z7'. This file should NOT be loaded into this collection.F)r rparentr)r r-rr"rexpected_folder actual_folder error_msgrrrvalidate_document_typems z!JSONLoader.validate_document_type source_namecCsP||}|dur gS|j||}|r|sgSg}g}g}|dD]T}||} | r{|| ||\} } | sR|| ||jt d|jd|q'|j j | dd} | D]} |j| d<t || d<|| d <|| d <d| d <|| q\q'|rt d |d t dt |dt dt |d|D] }t d|q|S)a| Load all JSON records for a collection from a source directory. This method: 1. Finds the appropriate ODF folder 2. Loads all JSON files in that folder 3. Flattens ALL nested fields recursively 4. Returns flattened records with ALL fields preserved CRITICAL: NO fields are lost. ALL nested data is flattened. Args: collection_name: Name of the collection source_name: Name of the source directory (e.g., 'tokyo_2020') Returns: List of flattened records with ALL fields from JSON N*.jsonu⚠️ SKIPPED: z - Wrong document type for T)extract_arrays _source_file _source_path_source_competition_target_collection_validation_passedu ⚠️ VALIDATION SUMMARY for :z - Loaded: z recordsz - Skipped: z files (wrong document type)z - )r r rrglobr,r3rrr)r flatten_json_filestrlen)r rr4 folder_name folder_path all_recordsvalidation_errors skipped_filesr"datais_validr2flattened_recordsrecorderrorrrrload_collection_from_sourcesD       z&JSONLoader.load_collection_from_sourcecCsJ||}|dur dS|j||}|r|sdStt|dS)a Count JSON files for a collection in a source directory. Args: collection_name: Name of the collection source_name: Name of the source directory Returns: Number of JSON files found Nrr5)r r rrr@listr=r rr4rArBrrrcount_files_in_sources z JSONLoader.count_files_in_sourcecCs>|j|i}|d}|||du|rd|ddSddS)z Get validation information for a collection. Args: collection_name: Name of the collection Returns: Dictionary with validation rules rNzOnly files from folder 'z/' will be loadedzNo folder validation) collectionr0validation_enabled descriptionr)r rrrArrrget_validation_infos zJSONLoader.get_validation_infocCsL||}|durddddS|j||}d|t|||||dS)a  Get information about a collection in a source. Args: collection_name: Name of the collection source_name: Name of the source directory Returns: Dictionary with collection info NFr) supportedfolder file_countT)rSrTpathrrU)r r r?rrNrMrrrget_collection_infos  zJSONLoader.get_collection_infoN)__name__ __module__ __qualname____doc__r?rrrrrrr r!rr,tupleboolr3rKintrNrRrWrrrrr s<   *) L  r )r[r&ospathlibrtypingrrrrrservices.json_flattenerrr rrrrs