o ג[i+@sNdZddlmZmZmZmZddlZddlmZddl m Z GdddZ dS)a5 Import Preview Engine This module builds editable preview tables for import decisions. It tracks user decisions per row (import/skip) without performing any writes. Key principles: - NO database writes - User decisions tracked in memory - Field-level editing supported - All changes reversible until apply )ListDictAnyOptionalN)datetime) JSONFlattenerc@s2eZdZdZddZ d deeeefdede j fdd Z d eeeefde j fd d Z d e j deeee ffddZ d!d e j deeeefdee dedeeeeff ddZdeeee fdedeeeffddZd e j de j fddZd e j de j fddZd e j deeeffddZdS)"ImportPreviewEnginez Service for building and managing import preview data. Allows users to review, edit, and make decisions about records before importing them to MongoDB. cCs t|_dS)z!Initialize import preview engine.N)r flattener)selfr :/var/www/html/IGF-ODF-V3/services/import_preview_engine.py__init__s zImportPreviewEngine.__init__newrecords record_typereturnc Cs|stSg}t|D]l\}}|dkr|di}nN|dkrP|di}|di}|di} |}|dd|d<|d d |d <| d d |d<n|dkri|di}|did d |d <ni}||d<d|d<||q t|} | t| d} | S)a Build a DataFrame for preview and editing. Args: records: List of records to preview record_type: Type of records ('new', 'duplicate', 'exact') Returns: DataFrame with preview data r json_record duplicate db_record comparison match_typeunknown _match_type_id_db_iddifference_countr_difference_countexact _row_indexpending_import_decisionN)pd DataFrame enumerategetcopyappendwherenotnull) r rr preview_datairecordrow_datajson_recdb_recrdfr r r build_preview_dataframe s0     z+ImportPreviewEngine.build_preview_dataframe duplicatesc Cs$|stSg}t|D]u\}}|di}|di}|di}|di}t|t|B} | D]J} | dkr=q6|| } || } | |vrR|| d} nd} | dur\t| nd}| durft| nd}|||d d t|dd | ||| d d q6q t|}|t |d}|S)z Build a side-by-side comparison DataFrame for duplicates. Args: duplicates: List of duplicate records with comparison data Returns: DataFrame with JSON vs DB comparison rrr differencesrstatusmatchNrrrr )rrrfield json_valuedb_valuer4r!) r"r#r$r%setkeysstrr'r(r))r r2comparison_datar+dupr.r/rr3 all_fieldsr6r7r8r4json_value_str db_value_strr0r r r $build_duplicate_comparison_dataframe]sB          z8ImportPreviewEngine.build_duplicate_comparison_dataframe preview_dfcCs|jsd|jvrgggdSgggd}|D]'\}}|dd}|d|}||vr6||t|q|dt|q|S)z Extract user import decisions from preview DataFrame. Args: preview_df: DataFrame with user decisions Returns: Dictionary with lists of row indices by decision r!)importskipr r r)emptycolumnsiterrowsr%r'int)r rB decisionsidxrowdecision row_indexr r r extract_import_decisionss    z,ImportPreviewEngine.extract_import_decisionsForiginal_recordsimport_indices unflattenc Csg}|D]@}|t|kr q|j|}|}gd} | D]} || dqdd|D}dd|D}|r?|j|}||q|S)a Prepare records for import based on user decisions and edits. Args: preview_df: DataFrame with user edits (flattened columns) original_records: Original record list import_indices: List of row indices to import unflatten: If True, convert flattened records back to nested structure Returns: List of records ready for import (flattened or nested) ) rr!rrr _source_file _source_path_source_competition_target_collection_validation_passed_document_type_expected_typesNcSs i|] \}}|ds||qS)_) startswith.0kvr r r  zBImportPreviewEngine.prepare_records_for_import..cSs i|] \}}t|r||qSr )r"notnar[r r r r_r`)lenilocto_dictpopitemsr rQr') r rBrOrPrQrecords_to_importrJrKr,metadata_fieldsr6r r r prepare_records_for_imports    z.ImportPreviewEngine.prepare_records_for_importrIc CsH|tdd|Dt|dgt|dgt|dg|dS)z Build a summary of import decisions. Args: decisions: Dictionary with decision lists record_type: Type of records Returns: Summary dictionary css|]}t|VqdS)N)rb)r\r^r r r sz;ImportPreviewEngine.build_import_summary..rCrDr )r total_records to_importto_skipr rI)sumvaluesrbr%)r rIrr r r build_import_summarysz(ImportPreviewEngine.build_import_summarycCd|jvr d|d<|S)z Mark all records in preview for import. Args: preview_df: Preview DataFrame Returns: Updated DataFrame r!rCrFr rBr r r mark_all_for_import z'ImportPreviewEngine.mark_all_for_importcCrq)z Mark all records in preview to skip. Args: preview_df: Preview DataFrame Returns: Updated DataFrame r!rDrrrsr r r mark_all_for_skip(ruz%ImportPreviewEngine.mark_all_for_skipcs|jriSgdfdd|jD}i}|D]/}||}||}t|t|t|dkr@t|t|ddndd||<q|S)z Get statistics about fields in preview data. Args: preview_df: Preview DataFrame Returns: Dictionary with field statistics )rr!rrrcsg|]}|vr|qSr r )r\colrhr r Ssz.rd)non_null_count null_countcoverage_percentage)rErFrarnisnarHrbround)r rB data_fields field_statsr6r|r}r rxr get_field_statistics:s $ z(ImportPreviewEngine.get_field_statisticsN)r)F)__name__ __module__ __qualname____doc__r rrr;rr"r#r1rArHrNboolrirprtrvrr r r r rsn = < * @     r) rtypingrrrrpandasr"rservices.json_flattenerrrr r r r s