o ג[i7@s^dZddlZddlmZmZmZmZmZddl m Z ddl m Z ddl m Z GdddZdS) u Data Service Layer This module handles loading and saving tabular data from MongoDB collections. It provides the bridge between MongoDB documents and pandas DataFrames for UI editing. Key responsibilities: - Load collection documents as normalized DataFrames - Handle schema normalization (missing fields → None) - Convert ObjectId to string for UI display - Track changes between original and edited data - Apply changes back to MongoDB with user confirmation N)DictListAnyTupleOptional)ObjectId)datetime) JSONFlattenerc @s"eZdZdZddZdeeefdeeeffddZde eeefde efd d Z dd ede de de j fddZ  dd ede de de eeeffddZde j de j dee ee ee effddZde j de j deeeffddZd ede j de j deeeffddZdS) DataServicez Service for loading and saving collection data in tabular format. This service normalizes MongoDB documents into pandas DataFrames, tracks changes, and applies updates back to MongoDB. cCs||_t|_dS)z Initialize data service with MongoDB connection. Args: mongo_service: MongoService instance N)mongor flattener)self mongo_servicer1/var/www/html/IGF-ODF-V3/services/data_service.py__init__s zDataService.__init__docreturncCszi}|D]4\}}t|trt|||<qt|tr"|||<qt|tr,|||<qt|tr6|||<q|||<q|S)a  Normalize a MongoDB document for DataFrame display. Converts ObjectId to string and handles nested structures. Args: doc: MongoDB document Returns: Normalized document with ObjectId as string )items isinstancerstrr isoformatdictlist)r r normalizedkeyvaluerrr_normalize_document(s        zDataService._normalize_document documentscCsHt}|D] }||qt|}d|vr"|d|dd|S)z Get union of all fields across documents. Args: documents: List of MongoDB documents Returns: Sorted list of unique field names _idr)setupdatekeyssortedremoveinsert)r r all_fieldsrfieldsrrr_get_all_fieldsHs   zDataService._get_all_fieldsrcollection_namelimitskipcsbj|||}|stjdgdSfdd|D}|}tj||d}|t|d}|S)a Load MongoDB collection as a normalized pandas DataFrame. All documents are normalized to have the same columns. Missing fields are filled with None. ObjectId fields are converted to strings. Args: collection_name: Name of the collection limit: Maximum number of documents to load skip: Number of documents to skip Returns: DataFrame with normalized documents r)columnscsg|]}|qSr)r).0rr rr ~sz.N)r sample_documentspd DataFramer(wherenotnull)r r*r+r,rnormalized_docsr&dfrr/rload_collection_as_dataframeas z(DataService.load_collection_as_dataframec Cs|j|||}|s gSg}|D]@}d|vr%t|dtr%t|d|d<|j|}|D]\}} t| tr?t| ||<q/t| trJ| ||<q/| |q|S)a0 Load MongoDB records and flatten them for comparison with JSON records. This method: 1. Loads documents from MongoDB 2. Flattens each document using JSONFlattener 3. Converts ObjectId to string 4. Returns list of flattened dictionaries Args: collection_name: Name of the collection limit: Maximum number of documents to load skip: Number of documents to skip Returns: List of flattened document dictionaries r) r r1rrrr flattenrrrappend) r r*r+r,rflattened_recordsr flattenedrrrrrload_flattened_recordss      z"DataService.load_flattened_records original_df edited_dfcCsg}g}g}t}d|jvrt|dt}t}d|jvr,t|dt}t||}|D]\}} | } dd| D} | d} | rW| dksWt | rc| dd| | q6t| |vru| dd| | q6||d| k} | js| jd} dd| D} d}i}| D]#\}}|dkrq| |}t |rt |rq||krd }|||<q| D]}|dkr|| vrd }d||<q|r| | |d q6|||fS) a Identify new, updated, and deleted documents. Args: original_df: Original DataFrame from MongoDB edited_df: Edited DataFrame from UI Returns: Tuple of (new_docs, updated_docs, deleted_ids) rcS i|] \}}t|r||qSrr2notnar.kvrrr  z1DataService._identify_changes..NrcSrArrBrDrrrrGrHFT)rupdates)r r-dropnaastyperriterrowsto_dictrgetr2isnapopr;emptyilocr")r r?r@new_docs updated_docs deleted_ids original_ids edited_idsidxrowrow_dictdoc_id original_row original_dictchangedrJrroriginal_valuerrr_identify_changess`            zDataService._identify_changescCs0|||\}}}t|t|t||||dS)z Preview changes between original and edited DataFrames. Args: original_df: Original DataFrame edited_df: Edited DataFrame Returns: Dictionary with change summary ) new_count updated_count deleted_countrTrUrV)ralen)r r?r@rTrUrVrrrpreview_changess zDataService.preview_changesc CsD|j|}|durdddS|||\}}}ddddgd}z|rPz||} t| j|d<WntyO} z|d d t| WYd} ~ nd} ~ ww|D]h} zE| d } | d } t | trft | } d d| D}dd| D}i}|r||d<|r||d<|r| d | i||dd7<WqRty} z|d d| dt| WYd} ~ qRd} ~ ww|D]=} zt | trt | } | d | i|dd7<Wqty} z|d d| dt| WYd} ~ qd} ~ wwW|Sty!} zd|d<|d dt| WYd} ~ |Sd} ~ ww)a  Apply changes from edited DataFrame back to MongoDB. This method: 1. Identifies new, updated, and deleted documents 2. Inserts new documents 3. Updates existing documents (only changed fields) 4. Deletes removed documents Args: collection_name: Name of the collection original_df: Original DataFrame from MongoDB edited_df: Edited DataFrame from UI Returns: Dictionary with operation results NFz0Collection not found or not connected to MongoDB)successerrorTr)rginsertedupdateddeletederrorsrirlzInsert error: rrJcSsi|] \}}|dur||qS)NrrDrrrrGwz-DataService.apply_changes..cSsi|] \}}|dur|dqS)NrIrrDrrrrGxrmz$setz$unsetrjzUpdate error for z: rkzDelete error for rgzGeneral error: )r get_collectionra insert_manyre inserted_ids Exceptionr;rrrr update_one delete_one)r r*r?r@ collectionrTrUrVresults insert_resulte update_docr\rJ set_updates unset_updates update_oprrr apply_changes7s   $ * *"zDataService.apply_changesN)r)r)r9r)__name__ __module__ __qualname____doc__rrrrrrr(intr2r3r8r>rrarfr}rrrrr sd" "  . 2 ]   r )rpandasr2typingrrrrrbsonrrservices.json_flattenerr r rrrrs