ג[i6 dZddlmZmZmZmZmZddlZGddZ d deee efeefde d ee effd Z d d ee efde d ee effd Z dS)aQ Universal JSON Flattener This module provides recursive JSON flattening functionality to convert deeply nested JSON structures into flat key-value pairs suitable for tabular display and editing. Key principles: - NO data loss - ALL fields at ANY depth are preserved - NO hardcoded field names - works with ANY JSON structure - Recursive parsing of nested objects and arrays - Dot notation for nested paths (e.g., "parent.child.0.field") - Type preservation for all values CRITICAL: This is a data integrity critical component. Every field from source JSON MUST appear in flattened output. )DictAnyListUnionOptionalNc eZdZdZddefdZ ddeeeefe efded e d eeeffd Z d e eded e d eeeffd Z deeefd eeeffdZ de eeefd e efdZ dde eeefde d e eeeffdZ ddeeefde d e eeeffdZdS) JSONFlattenera Universal JSON flattener that converts nested structures to flat key-value pairs. Handles: - Nested dictionaries (unlimited depth) - Arrays of primitives - Arrays of objects - Mixed nested structures - Null values - All primitive types . separatorc||_dS)z Initialize flattener. Args: separator: Character to use for path separation (default: ".") N)r )selfr s 3/var/www/html/IGF-ODF-V3/services/json_flattener.py__init__zJSONFlattener.__init__$s#Fdata parent_keypreserve_arraysreturnc0i}t|tr|D]\}}|r ||j|n|}t|tr+|||||Vt|t r+||||||||<nMt|t r+|||||n |r|||<n||d<|S)a Recursively flatten a nested JSON structure. Args: data: Input data (dict or list) parent_key: Current path prefix preserve_arrays: If True, keep primitive arrays as-is instead of flattening Returns: Flattened dictionary with dot-notation keys Examples: Input: {"a": {"b": {"c": 1}}} Output: {"a.b.c": 1} Input: {"items": [{"id": 1}, {"id": 2}]} Output: {"items.0.id": 1, "items.1.id": 2} Input: {"tags": ["red", "blue"]} Output: {"tags.0": "red", "tags.1": "blue"} (or {"tags": ["red", "blue"]} if preserve_arrays=True) _value) isinstancedictitemsr updateflattenlist _flatten_list)r rrr flattenedkeyvaluenew_keys rrzJSONFlattener.flatten-sD8 dD ! ! +"jjll / / UBLUZ>>>>>RUeT**/$$T\\%/%R%RSSSSt,,/$$T%7%7w%X%XYYYY*/Ig&& /d # # +   T//j/RR S S S S +(, *%%&* (#rrci}|r"|r td|Dr|||<|St|D]\}}||j|}t|tr+|||||Rt|tr+||||||||<|S)a Flatten a list into indexed keys. Args: items: List to flatten parent_key: Current path prefix preserve_arrays: If True, keep primitive arrays as-is Returns: Flattened dictionary c3PK|]!}t|ttf V"dSN)rrr).0items r z.JSONFlattener._flatten_list..{s4,b,bTXD4,1O1O-O,b,b,b,b,b,br) all enumerater rrrrrr)r rrrrindexr'r"s rrzJSONFlattener._flatten_listgs"   u ,b,b\a,b,b,b)b)b $)Ij ! %U++ * *KE4#22EG$U3B3Z00 , ,4<<>>,/01u!>qs$J!*Wd*C*CIIE>44%"$g,,%/+r***g,,%/+&enGG7*/23A#E 2B$LE!A#JJ $/):):)<)</,.GDMM,.GDM%dmGGb I  "" /I!'400! G'lle+)NN4((('lle+)!&gt,,/).GI& rrecordsct}|D])}||*d|D}td|D}||zS)z Extract all unique keys from a list of flattened records. Args: records: List of flattened dictionaries Returns: Sorted list of all unique keys c<g|]}|d|Sr.r5r&ks r z.JSONFlattener.get_all_keys..s)BBBq S0A0ABBBBrc<g|]}|d|SrGrHrIs rrKz.JSONFlattener.get_all_keys..s)III!q||C7H7HIAIIIr)setrkeyssorted)r rDall_keysrecord metadata_keys data_keyss r get_all_keyszJSONFlattener.get_all_keyssv55 + +F OOFKKMM * * * *CBHBBB IIxIIIJJ y((rT fill_missingc|sgS||}|s|Sg}|D]7}i}|D]}||d||<||8|S)a, Ensure all records have the same keys (columns). Args: records: List of flattened dictionaries fill_missing: If True, add missing keys with None values Returns: List of normalized records with consistent keys N)rTgetr:)r rDrUrP normalizedrQnormalized_recordr s rnormalize_recordszJSONFlattener.normalize_recordss I$$W-- N  1 1F "  ? ?)/C)>)>!#&&   / 0 0 0 0r json_dataextract_arraysc.|s|gSg}|}i}|D]}|j}t |D]\}} | rj|d|} t| } |dzt|kr%j||dzdnd} | |vri|| <| || vr i|| | <| r|||| | | <n|||| | d<nِ|rt|fd} t|| D]s}|| | }| D]'\}}| | jzs|||<(||tn|g}|S)a Flatten a JSON file that may contain arrays of records. This is specifically designed for ODF JSON files where the actual records are nested inside arrays within the structure. Args: json_data: Parsed JSON data extract_arrays: If True, extract array items as separate records Returns: List of flattened records Example: Input: { "odf_body": { "competition": { "participant": [ {"code": 1, "name": "A"}, {"code": 2, "name": "B"} ] } } } Output: [ { "odf_body.competition.participant.0.code": 1, "odf_body.competition.participant.0.name": "A" }, { "odf_body.competition.participant.1.code": 2, "odf_body.competition.participant.1.name": "B" } ] Nr4rrc8|jSr%)countr )xr s rz1JSONFlattener.flatten_json_file..ds!''$.BYBYr)r )rrNr6r r*r7joinr8r9maxrOcopyrr5r:)r r[r\rDflat array_pathsr r=r?r@ array_base array_index array_suffix deepest_arrayr+rQr!s` rflatten_json_filezJSONFlattener.flatten_json_fileseR -LL++, ,||I&& 99;;  CIIdn--E%U++  4<<>>!%!4!4U2A2Y!?!?J"%d))KGHsSQVZZGW#_4>#6#6uQqSTT{#C#C#C]_L!4524 J/"+j*AAB?A J/ <$SMQRUY J/ <\JJIMc J/ >-$.*HII,&+s v&&&& 'fGrNr )rF)T)__name__ __module__ __qualname____doc__strrrrrrboolrrrCrTrZrkrrr r s  ####### % 88DcNDI-.88 8 c3h 8888t'Cy'' ' c3h ''''RG4S>Gd38nGGGGR)Dc3h$8)T#Y))))2"  d38n%   d38n     J $ddS>dd d38n  ddddddrr r rr rcJt|}||S)z Convenience function to flatten JSON data. Args: data: Input JSON data separator: Path separator (default: ".") Returns: Flattened dictionary )r r)rr flatteners r flatten_jsonrvxs%i((I   T " ""rr,cJt|}||S)z Convenience function to unflatten data. Args: flat_data: Flattened dictionary separator: Path separator (default: ".") Returns: Nested structure )r rC)r,r rus runflatten_jsonrxs%i((I   y ) ))rrl) rptypingrrrrrjsonr rqrvrxrsrrr{s$43333333333333 ^^^^^^^^F ## S#XS ) *## #s(^####(**CH~** #s(^******r