o ג[ig7@sRdZddlmZmZmZmZmZmZddlZddl Z ddl m Z GdddZ dS)ad Diff Engine Service This module compares JSON records against MongoDB records to identify: - Exact matches - Potential duplicates (by business keys OR alternative keys) - New records - Field differences Key principles: - Config-driven matching rules - Support for multiple matching strategies - No automatic decisions - Preserve all comparison metadata )ListDictAnySetTupleOptionalN) JSONFlattenerc @seZdZdZdeeeffddZdeeefdefddZdeeefd e ede e fd d Z d e eeefde efd dZdeeefdeeefdeeeffddZdede eeefde eeefdeeeffddZdede efddZdS) DiffEnginez Service for comparing JSON records with MongoDB records. Identifies matches, duplicates, and new records using configurable business key matching rules with support for alternative keys. collections_configcCs||_t|_dS)z Initialize diff engine with collections configuration. Args: collections_config: Collections configuration from app_config.yaml N)r r flattener)selfr r 0/var/www/html/IGF-ODF-V3/services/diff_engine.py__init__s zDiffEngine.__init__recordreturncCs4dd|D}tj|dtd}t|S)z Compute hash of a record for exact matching. Args: record: Record dictionary (flattened or nested) Returns: SHA256 hash of record cSs i|] \}}|ds||qS_ startswith).0kvr r r 4s z,DiffEngine._compute_hash..T) sort_keysdefault)itemsjsondumpsstrhashlibsha256encode hexdigest)r r clean_record record_strr r r _compute_hash)s zDiffEngine._compute_hashkeyscCsDg}|D]}||}|durdS|t|q|r t|SdS)z Extract business key values from a record. Args: record: Record dictionary keys: List of key field names Returns: Tuple of key values or None if any key is missing N)getappendrtuple)r rr'valueskeyvaluer r r_extract_business_key9s z DiffEngine._extract_business_keyrecordscCs"t}|D] }||q|S)z Get union of all fields across records. Args: records: List of records Returns: Set of all field names )setupdater')r r/ all_fieldsrr r r_get_all_fieldsVs zDiffEngine._get_all_fields json_record db_recordc Cs|||g}i}g}g}|D]B}||}||} |dkr q||vr2||d|dd||<q||vr?dd| d||<q|| krLd|| d||<q||q|||t|dS)z Compare two records field by field. Args: json_record: Record from JSON db_record: Record from MongoDB Returns: Dictionary with comparison results _idnewN)status json_valuedb_valuemissing_in_json different) differences new_fieldsmatching_fieldsdifference_count)r3r(r)len) r r4r5r2r=r>r?fieldr9r:r r r_compare_recordsgs>      zDiffEngine._compare_recordscollection_name json_records db_recordsc& Csg}|D]}tdd|Dr||q|j|}||q|j|i}|dg}|dg} |dd} |sYggggddt|ddddgg| t|t||d Si} |D] }||} || | <q]i} i}|D]+}| ||}|r}|| |<| r| d vr| || }|r||vrg||<|||qog}g}g}t }|D]}||}|| vr||| |qd }d }d }d }| d vr| ||}|r|| vrd}d}| |}|}|s| d vr| r| || }|r||vr||}t|dkr d}d}|d}|}n |r+|r+| ||}||||||d| |q| ||}|||dq| |}| |} dd|D}!dd| D}"|!|"}#tdd|D}$tdd|D}%|||t|#t|t|t|t|#|$|%d|| | t|t||d S)a Compare JSON records against MongoDB records with multiple matching strategies. This method: 1. Flattens MongoDB records if needed 2. Identifies exact matches (by hash) 3. Identifies potential duplicates (by primary OR alternative keys) 4. Identifies new records 5. Identifies new fields (ALL flattened keys) Matching Strategies: - primary_only: Match only by business_keys - primary_or_alternative: Try business_keys first, then alternative_keys - alternative_only: Match only by alternative_keys Args: collection_name: Name of the collection json_records: Flattened records from JSON files db_records: Records from MongoDB (will be flattened) Returns: Dictionary with comparison results including match_method for each duplicate css|]}d|vVqdS).Nr )rr,r r r sz1DiffEngine.compare_collections.. business_keysalternative_keysmatching_strategy primary_onlyr)exact_match_countduplicate_countnew_record_countnew_field_countprimary_key_matchesalternative_key_matches) exact_matches duplicates new_recordsr>summary matching_keysrJrK json_countdb_countrD)primary_or_alternativealternative_onlyFN)rLrZT primary_keyalternative_key)r4r5 comparison matching_key match_method)r4 business_keycSh|] }|ds|qSrrrfr r r Nz1DiffEngine.compare_collections..cSrcrrrdr r rrfOrgcs"|] }|ddkrdVqdS)rar\r]Nr(rdr r rrHS csrh)rar^r]Nrirjr r rrHTrl)anyr'r)r flattenr r(rAr&r.r0addrCr3sumlist)&r rDrErFflattened_db_recordsr5flatcollection_configrIrJrK db_by_hash record_hashdb_by_primary_keydb_by_alternative_keyr\alt_keyrSrTrUmatched_json_hashesr4 json_hashmatchedramatched_db_record matched_keyjson_primary_key json_alt_key alt_matchesr_ json_fields db_fieldsjson_data_fieldsdb_data_fieldsall_new_fieldsprimary_matchesalternative_matchesr r rcompare_collectionss                       zDiffEngine.compare_collectionscCs|j|i}|dgS)z Get the business key matching rules for a collection from config. Args: collection_name: Name of the collection Returns: List of field names used for matching rI)r r()r rDrtr r rget_matching_keysks zDiffEngine.get_matching_keysN)__name__ __module__ __qualname____doc__rrrrr&rrrr.rr3rCrrr r r rr s<   "    >  Gr ) rtypingrrrrrrr rservices.json_flattenerrr r r r rs