ג[i+\dZddlmZmZmZmZddlZddlmZddl m Z GddZ dS)a5 Import Preview Engine This module builds editable preview tables for import decisions. It tracks user decisions per row (import/skip) without performing any writes. Key principles: - NO database writes - User decisions tracked in memory - Field-level editing supported - All changes reversible until apply )ListDictAnyOptionalN)datetime) JSONFlattenerceZdZdZdZ ddeeeefdede j fdZ deeeefde j fd Z d e j deeee ffd Z dd e j d eeeefdee dedeeeeff dZdeeee fdedeeeffdZd e j de j fdZd e j de j fdZd e j deeeffdZdS)ImportPreviewEnginez Service for building and managing import preview data. Allows users to review, edit, and make decisions about records before importing them to MongoDB. c,t|_dS)z!Initialize import preview engine.N)r flattener)selfs :/var/www/html/IGF-ODF-V3/services/import_preview_engine.py__init__zImportPreviewEngine.__init__s&newrecords record_typereturnc|stjSg}t|D][\}}|dkr*|di}n|dkr|di}|di}|di} |}|dd|d<|d d |d <| d d |d<n^|dkrV|di}|did d |d <ni}||d<d|d<||]tj|} | tj| d} | S)a Build a DataFrame for preview and editing. Args: records: List of records to preview record_type: Type of records ('new', 'duplicate', 'exact') Returns: DataFrame with preview data r json_record duplicate db_record comparison match_typeunknown _match_type_id_db_iddifference_countr_difference_countexact _row_indexpending_import_decisionN)pd DataFrame enumerategetcopyappendwherenotnull) r rr preview_datairecordrow_datajson_recdb_recrdfs rbuild_preview_dataframez+ImportPreviewEngine.build_preview_dataframe s "<>> ! "7++ * *IAve# !::mR88==?? + !::mR88K44#ZZ b99 $==??+1**\9*M*M'%+ZZr%:%:"0:?QST0U0U,--' !::mR88==??%+ZZ R%@%@%D%DUB%O%O""&'H\ ",5H' (    ) ) ) )\, ' 'XXbjnnd + + r duplicatesc |stjSg}t|D]e\}}|di}|di}|di}|di}t |t |z} | D]} | dkr || } || } | |vr|| d} nd} | t | nd}| t | nd}|||d d t |dd | ||| d d gtj|}|tj |d}|S)z Build a side-by-side comparison DataFrame for duplicates. Args: duplicates: List of duplicate records with comparison data Returns: DataFrame with JSON vs DB comparison rrr differencesrstatusmatchNrrrr$)r#rrfield json_valuedb_valuer9r%) r&r'r(r)setkeysstrr+r,r-)r r6comparison_datar/dupr2r3rr8 all_fieldsr;r<r=r9json_value_str db_value_strr4s r$build_duplicate_comparison_dataframez8ImportPreviewEngine.build_duplicate_comparison_dataframe]s "<>> ! ++# # FAsww}b11HWW["--Fr22J$..;;KX]]__--FKKMM0B0BBJ#  E>%\\%00 !::e,,K'%(/9FF$F5?!TZPT08Ns8}}}$ &&"##&77<#C#C!&**UB"7"788""0 ,$(1 ( (    # 8\/ * * XXbjnnd + + r preview_dfct|js d|jvrgggdSgggd}|D]\}}|dd}|d|}||vr)||t |^|dt ||S)z Extract user import decisions from preview DataFrame. Args: preview_df: DataFrame with user decisions Returns: Dictionary with lists of row indices by decision r%)importskipr$r$r#)emptycolumnsiterrowsr)r+int)r rG decisionsidxrowdecision row_indexs rextract_import_decisionsz,ImportPreviewEngine.extract_import_decisionss   19KK    #++-- < >::::)$++C NN;;;;rForiginal_recordsimport_indices unflattencg}|D]}|t|kr|j|}|}gd} | D]} || dd|D}d|D}|r|j|}|||S)a Prepare records for import based on user decisions and edits. Args: preview_df: DataFrame with user edits (flattened columns) original_records: Original record list import_indices: List of row indices to import unflatten: If True, convert flattened records back to nested structure Returns: List of records ready for import (flattened or nested) ) r#r%rrr! _source_file _source_path_source_competition_target_collection_validation_passed_document_type_expected_typesNcDi|]\}}|d||S)_) startswith.0kvs r zBImportPreviewEngine.prepare_records_for_import..s/OOOtq!Q\\#=N=NOaOOOrcBi|]\}}tj|||S)r&notnarcs rrgzBImportPreviewEngine.prepare_records_for_import..s+EEEtq!!EaEEEr)lenilocto_dictpopitemsr rWr+) r rGrUrVrWrecords_to_importrPrQr0metadata_fieldsr;s rprepare_records_for_importz.ImportPreviewEngine.prepare_records_for_imports&!' -' -Cc*oo% /#&C[[]]F   O) ( ( 5$''''POv||~~OOOFFEv||~~EEEF :11&99  $ $V , , , ,  rrOc ,|td|Dt|dgt|dgt|dg|dS)z Build a summary of import decisions. Args: decisions: Dictionary with decision lists record_type: Type of records Returns: Summary dictionary c34K|]}t|VdS)N)rk)rdrfs r z;ImportPreviewEngine.build_import_summary..s( D DAQ D D D D D DrrIrJr$)r total_records to_importto_skipr$rO)sumvaluesrkr))r rOrs rbuild_import_summaryz(ImportPreviewEngine.build_import_summarys ' D D1A1A1C1C D D DDDY]]8R88999==44559==B7788"    rc"d|jvrd|d<|S)z Mark all records in preview for import. Args: preview_df: Preview DataFrame Returns: Updated DataFrame r%rIrLr rGs rmark_all_for_importz'ImportPreviewEngine.mark_all_for_imports% !3 3 6-5J) *rc"d|jvrd|d<|S)z Mark all records in preview to skip. Args: preview_df: Preview DataFrame Returns: Updated DataFrame r%rJr}r~s rmark_all_for_skipz%ImportPreviewEngine.mark_all_for_skip(s% !3 3 4-3J) *rc |jriSgdfd|jD}i}|D]}||}||}t |t |t |dkr#t|t |z dzdndd||<|S)z Get statistics about fields in preview data. Args: preview_df: Preview DataFrame Returns: Dictionary with field statistics )r#r%rrr!cg|]}|v| Sriri)rdcolrqs r z.Ss#WWWsC!>!L>!tCH~.>!S >!  >! d38n  >!>!>!>!@ T#Y'   c3h     0L $L $'L' c3h''''''rr ) rtypingrrrrpandasr&rservices.json_flattenerrr rirrrs  -,,,,,,,,,,,111111MMMMMMMMMMr