o ג[i6 @sdZddlmZmZmZmZmZddlZGdddZ ddeee efeefde d ee effd d Z dd ee efde d ee effd dZ dS)aQ Universal JSON Flattener This module provides recursive JSON flattening functionality to convert deeply nested JSON structures into flat key-value pairs suitable for tabular display and editing. Key principles: - NO data loss - ALL fields at ANY depth are preserved - NO hardcoded field names - works with ANY JSON structure - Recursive parsing of nested objects and arrays - Dot notation for nested paths (e.g., "parent.child.0.field") - Type preservation for all values CRITICAL: This is a data integrity critical component. Every field from source JSON MUST appear in flattened output. )DictAnyListUnionOptionalNc @s&eZdZdZd defddZ  d!deeeefe efd ed e d eeeffd d Z de ed ed e d eeeffddZ deeefd eeeffddZ de eeefd e efddZ d"de eeefde d e eeeffddZ d"deeefde d e eeeffddZdS)# JSONFlattenera Universal JSON flattener that converts nested structures to flat key-value pairs. Handles: - Nested dictionaries (unlimited depth) - Arrays of primitives - Arrays of objects - Mixed nested structures - Null values - All primitive types . separatorcCs ||_dS)z Initialize flattener. Args: separator: Character to use for path separation (default: ".") N)r )selfr r 3/var/www/html/IGF-ODF-V3/services/json_flattener.py__init__$s zJSONFlattener.__init__Fdata parent_keypreserve_arraysreturncCsi}t|trC|D]5\}}|r||j|n|}t|tr,|||||q t|tr<|||||q |||<q |St|trT||||||S|r\|||<|S||d<|S)a Recursively flatten a nested JSON structure. Args: data: Input data (dict or list) parent_key: Current path prefix preserve_arrays: If True, keep primitive arrays as-is instead of flattening Returns: Flattened dictionary with dot-notation keys Examples: Input: {"a": {"b": {"c": 1}}} Output: {"a.b.c": 1} Input: {"items": [{"id": 1}, {"id": 2}]} Output: {"items.0.id": 1, "items.1.id": 2} Input: {"tags": ["red", "blue"]} Output: {"tags.0": "red", "tags.1": "blue"} (or {"tags": ["red", "blue"]} if preserve_arrays=True) _value) isinstancedictitemsr updateflattenlist _flatten_list)r rrr flattenedkeyvaluenew_keyr r r r-s$      zJSONFlattener.flattenrcCsi}|r|rtdd|Dr|||<|St|D]1\}}||j|}t|tr6|||||qt|trF|||||q|||<q|S)a Flatten a list into indexed keys. Args: items: List to flatten parent_key: Current path prefix preserve_arrays: If True, keep primitive arrays as-is Returns: Flattened dictionary css |] }t|ttf VqdSN)rrr).0itemr r r {sz.JSONFlattener._flatten_list..) all enumerater rrrrrr)r rrrrindexr!rr r r rgs   zJSONFlattener._flatten_list flat_datac Csi}|D]\}}|dr|dvr|||<q||j}|}t|ddD]d\}}|re|dkr:||dnd} | rDt|tsD t|} t|tsOg}t || kr`| it || ksU|| }q(||vr|dt |krw||dnd} | r| rg||<ni||<||}q(|d} | rt| } t|tsg}t || kr| dt || ks||| <qt|t r||| <q|S)a6 Reconstruct nested structure from flattened data. Args: flat_data: Flattened dictionary with dot-notation keys Returns: Nested dictionary/list structure Note: This is used when writing back to MongoDB or JSON. _)_id _source_file _source_path_source_competitionNr) r startswithsplitr r$isdigitrrintlenappendr) r r&resultflat_keyrpartscurrentipartrr% next_part final_keyr r r unflattensL               zJSONFlattener.unflattenrecordscCsFt}|D] }||qdd|D}tdd|D}||S)z Extract all unique keys from a list of flattened records. Args: records: List of flattened dictionaries Returns: Sorted list of all unique keys cSsg|] }|dr|qSr'r.r kr r r z.JSONFlattener.get_all_keys..cSsg|] }|ds|qSr>r?r@r r r rBrC)setrkeyssorted)r r=all_keysrecord metadata_keys data_keysr r r get_all_keyss  zJSONFlattener.get_all_keysT fill_missingcCsT|sgS||}|s |Sg}|D]}i}|D] }||d||<q||q|S)a, Ensure all records have the same keys (columns). Args: records: List of flattened dictionaries fill_missing: If True, add missing keys with None values Returns: List of normalized records with consistent keys N)rKgetr3)r r=rLrG normalizedrHnormalized_recordrr r r normalize_recordss  zJSONFlattener.normalize_records json_dataextract_arrayscs||s|gSg}|}i}|D]f}|j}t|D]Y\}} | rzj|d|} t| } |dt|krKj||ddnd} | |vrUi|| <| || vrai|| | <| rn|||| | | <n |||| | d<nq!q|rt |fddd} t || D]$}|| | }| D]\}}| | js|||<q||q|S|g}|S)a Flatten a JSON file that may contain arrays of records. This is specifically designed for ODF JSON files where the actual records are nested inside arrays within the structure. Args: json_data: Parsed JSON data extract_arrays: If True, extract array items as separate records Returns: List of flattened records Example: Input: { "odf_body": { "competition": { "participant": [ {"code": 1, "name": "A"}, {"code": 2, "name": "B"} ] } } } Output: [ { "odf_body.competition.participant.0.code": 1, "odf_body.competition.participant.0.name": "A" }, { "odf_body.competition.participant.1.code": 2, "odf_body.competition.participant.1.name": "B" } ] Nr-rrcs |jSr)countr )xr r r ds z1JSONFlattener.flatten_json_file..)r)rrEr/r r$r0joinr1r2maxrFcopyrr.r3)r rQrRr=flat array_pathsrr6r8r9 array_base array_index array_suffix deepest_arrayr%rHrr rUr flatten_json_filesD)    ,   zJSONFlattener.flatten_json_fileNr)rF)T)__name__ __module__ __qualname____doc__strr rrrrboolrrr<rKrPr`r r r r rsR    :  ")"I % rrrr rcCt|}||S)z Convenience function to flatten JSON data. Args: data: Input JSON data separator: Path separator (default: ".") Returns: Flattened dictionary )rr)rr flattenerr r r flatten_jsonx rjr&cCrh)z Convenience function to unflatten data. Args: flat_data: Flattened dictionary separator: Path separator (default: ".") Returns: Nested structure )rr<)r&r rir r r unflatten_jsonrkrlra) retypingrrrrrjsonrrfrjrlr r r r s,e