ג[i^+hdZddlZddlZddlmZddlmZmZmZm Z m Z ddl m Z GddZ dS)aZ JSON Loader Service This module loads JSON records from the ODF directory structure. It respects folder hierarchy and extracts records relevant to selected collections. Key principles: - NO normalization - preserve all fields as-is - NO field removal - keep all keys - Support multiple JSON files per collection - Respect ODF folder structure N)Path)ListDictAnyOptionalTuple) JSONFlattenerc neZdZdZdedeeeffdZdeefdZ dede efdZ dedeefd Z d e de eeeffd Zd eeefded e deeeffd ZdededeeeeffdZdededefdZdedeeeffdZdededeeeffdZdS) JSONLoaderz Service for loading JSON records from ODF directory structure. Loads JSON files while preserving all original fields and structure. All mappings are read from configuration. base_pathcollections_configcbt||_||_t|_dS)z Initialize JSON loader with base path and collections config. Args: base_path: Base path to JSON files (e.g., 'idf-json-sample') collections_config: Collections configuration from app_config.yaml N)rr r r flattener)selfr r s 0/var/www/html/IGF-ODF-V3/services/json_loader.py__init__zJSONLoader.__init__#s(i"4&returnc|jsgSg}|jD]0}|r||j1t |S)z Get list of available JSON source directories. Returns: List of directory names (e.g., ['tokyo_2020', 'paris_2024']) )r existsiterdiris_dirappendnamesorted)rsourcesitems rget_available_sourcesz JSONLoader.get_available_sources/sr~$$&& IN**,, * *D{{}} *ty)))grcollection_namecb|j|i}|dS)z Get the ODF folder name for a collection from config. Args: collection_name: Name of the collection Returns: Folder name or None if not configured json_folderr get)rrcollection_configs rget_collection_folderz JSONLoader.get_collection_folder@s1!377LL $$]333rcgS)a Get expected document types for a collection. NOTE: With folder-based validation, we no longer track specific document types. This method now returns an empty list, indicating validation is folder-based. Args: collection_name: Name of the collection Returns: Empty list (validation is now folder-based, not type-based) )rrs rget_expected_document_typesz&JSONLoader.get_expected_document_typesMs  r file_pathc t|dd5}tj|cdddS#1swxYwYdS#t$r }t d|d|Yd}~dSd}~wwxYw)z Load and parse a single JSON file. Args: file_path: Path to JSON file Returns: Parsed JSON data or None if error rzutf-8)encodingNzError loading z: )openjsonload Exceptionprint)rr)fes rload_json_filezJSONLoader.load_json_file\s iw777 $1y|| $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $    393333 4 4 444444 s2A5 A9A9A A, A''A, json_datac |j|i}|d}|sdS|jj}||krd|jd|d|d|d }d|fSdS) ab Validate that JSON file is in the correct folder for the target collection. NEW APPROACH: Validation based on folder structure, not document_type. Files in the correct folder are automatically valid, regardless of document_type. This makes the system flexible and independent of specific document type values. Args: json_data: Parsed JSON data collection_name: Target collection name file_path: Path to JSON file (for validation) Returns: Tuple of (is_valid, error_message) r!)TzVALIDATION ERROR: File 'z' is in folder 'z' but collection 'z' expects files from folder 'z7'. This file should NOT be loaded into this collection.F)r r#parentr)rr5rr)r$expected_folder actual_folder error_msgs rvalidate_document_typez!JSONLoader.validate_document_typems"!377LL+// >> 8 "(-  O + $H9>HH=HH#2HHQ`HHH  )# #xr source_namecv||}|gS|j|z |z }|r|sgSg}g}g}|dD]}||} | r|| ||\} } | sJ|| ||jtd|jd||j | d} | D]B} |j| d<t|| d<|| d <|| d <d| d <|| C|rjtd |d tdt|dtdt|d|D]}td||S)a| Load all JSON records for a collection from a source directory. This method: 1. Finds the appropriate ODF folder 2. Loads all JSON files in that folder 3. Flattens ALL nested fields recursively 4. Returns flattened records with ALL fields preserved CRITICAL: NO fields are lost. ALL nested data is flattened. Args: collection_name: Name of the collection source_name: Name of the source directory (e.g., 'tokyo_2020') Returns: List of flattened records with ALL fields from JSON N*.jsonu⚠️ SKIPPED: z - Wrong document type for T)extract_arrays _source_file _source_path_source_competition_target_collection_validation_passedu ⚠️ VALIDATION SUMMARY for :z - Loaded: z recordsz - Skipped: z files (wrong document type)z - )r%r rrglobr4r<rrr1rflatten_json_filestrlen)rrr= folder_name folder_path all_recordsvalidation_errors skipped_filesr)datais_validr;flattened_recordsrecorderrors rload_collection_from_sourcez&JSONLoader.load_collection_from_sources9.00AA  In{2[@ !!## ;+=+=+?+? I  %))(33 / /I&&y11D /&*&A&A$Yb&c&c#)%,,Y777!((888iY^iiXgiijjj%)N$D$DTZ^$D$_$_!0//F-6^F>*-0^^F>*4?F013BF/037F/0&&v....  ' G_GGG H H H <#k"2"2<<< = = = S3}#5#5SSS T T T* ' 'oeoo&&&&rc||}|dS|j|z |z }|r|sdSt t |dS)a Count JSON files for a collection in a source directory. Args: collection_name: Name of the collection source_name: Name of the source directory Returns: Number of JSON files found Nrr?)r%r rrrJlistrGrrr=rKrLs rcount_files_in_sourcez JSONLoader.count_files_in_sources00AA  1n{2[@ !!## ;+=+=+?+? 14 ((2233444rc|j|i}|d}|||du|rd|dnddS)z Get validation information for a collection. Args: collection_name: Name of the collection Returns: Dictionary with validation rules r!NzOnly files from folder 'z/' will be loadedzNo folder validation) collectionr9validation_enabled descriptionr")rrr$rKs rget_validation_infozJSONLoader.get_validation_infosi!377LL'++M:: **"-T"9XcTkTTTTi    rc||}|ddddS|j|z |z }d|t|||||dS)a  Get information about a collection in a source. Args: collection_name: Name of the collection source_name: Name of the source directory Returns: Dictionary with collection info NFr) supportedfolder file_countT)r`rapathrrb)r%r rIrrYrXs rget_collection_infozJSONLoader.get_collection_infos00AA  "  n{2[@ ! $$!((**44_kRR    rN)__name__ __module__ __qualname____doc__rIrrrrrrr%r(rr4tupleboolr<rUintrYr^rdr'rrr r s )# )4S> ) ) ) )tCy" 4S 4Xc] 4 4 4 4 3 49    $sCx.1I"'S#X'QT'ae'joptvypyjz''''RJJJ d38n  JJJJX555 55556 3 4S>    (       c3h       rr )rhr.ospathlibrtypingrrrrrservices.json_flattenerr r r'rrrps   33333333333333111111\ \ \ \ \ \ \ \ \ \ r