o Vh P@s`ddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddlZddlmZmZmZmZddlZddlZddlmZmZmZmZejejdZd d Zd d Zd edefddZ d-de!de"de#fddZ$de!defddZ%de!defddZ&de!defddZ'd e!de!fd!d"Z(d e!defd#d$Z)d.d'd(Z*d)d*Z+d+d,Z,dS)/N)settings) JsonResponse)require_http_methods)DevelopmentApplication)EXTRACT_STRUCTURED_DATA_PROMPTTITLE_SEARCH_EXTRACTION_PROMPT"APPLICATION_FORM_EXTRACTION_PROMPT APPLICATION_FORM_FALLBACK_PROMPT)RateLimitErrorAPIErrorAPIConnectionErrorAPITimeoutError)api_keycCs&zt|WStyYdSw)N)pd to_datetimedate Exception)date_strr8/home/shobhit/Desktop/shao (1)/shao/shaoApp/functions.py parse_dates  rcCs*t|s |dks |durdSt|S)z7Safely get value, handling None, NaN, and empty stringsN)risnastrstrip)valrrrsafe_gets rdatareturnc Cst|tsiSddddddd}i}|D]M\}}|||}|drNz|dus.|dkr3d||<n ttt|||<Wqtt fyMd||<Yqw|dusV|dkr[d||<qt| ||<q|S)z7Validate and clean structured data from OpenAI responserr)development_typenumber_of_dwellingsnumber_of_storeysnumber_of_placesnumber_of_unitsnumber_of_lots number_of_N) isinstancedictitemsget startswithintfloatr ValueError TypeErrorr)rexpected_fieldsvalidated_datafield default_valuevaluerrrvalidate_structured_data"s2       r4?prompt max_retries base_delayc Cst|D]}ztjjjdd|dgddd}|jdjj}|dfWSt ys}zB||d kr[|d |t dd }t d |d d |d d|dt |WYd}~qdd|dt|fWYd}~Sd}~wtttfy}zB||d kr|d |t dd }t d|d d |d d|dt |WYd}~qdd|dt|fWYd}~Sd}~wty}zddt|fWYd}~Sd}~wwdS)z@Make OpenAI API request with exponential backoff and retry logiczgpt-4o-2024-05-13user)rolecontentg?i)modelmessages temperature max_tokensrNzRate limit hit, retrying in z.2fz seconds... (attempt /)zRate limit exceeded after z attempts: zAPI error, retrying in zAPI error after zUnexpected error: )NzMax retries exceeded)rangeclientchat completionscreatechoicesmessager<rr randomuniformprinttimesleeprr r r r)r7r8r9attemptresponser<edelayrrrmake_openai_request_with_retryIs>   "  ( "  ("rU descriptionc Cs$|r|siSd}t||kr|d|d}tj|d}t|\}}|r/td|iS|s7tdiSz|drItdd |}| d}t |}t |}|WSt j yw}ztd |td |iWYd}~Sd}~wty}ztd |iWYd}~Sd}~ww) zCExtract structured data from description with proper error handlingiN...)rVOpenAI request failed: Empty response from OpenAI``` ^```[a-z]*\n?rzJSON decoding error: Raw content: z%Unexpected error in data processing: )rlenrformatrUrNr*resubrstripjsonloadsr4JSONDecodeErrorr)rV max_lengthr7r<errorrr0rSrrrextract_structured_dataqs<       rgpdf_pathc Cs^zt|}t|j}d}d}|jD]}|}|r-t|dkr-|d7}|t|7}q|dkr6||nd}|dkr@||nd}|dkrO|dkrOd} d} n|dkr\|d kr\d} d } n|dkri|dkrid } d } nd } d} | | ||t|dt|d|dWdWS1swYWdSty} zdd t| ddddddWYd} ~ Sd} ~ ww)z>Detect if PDF is digital (text-based) or scanned (image-based)r rAgffffff?ddigitalhighg333333?2mediummixedlowscannedr5)type confidence total_pagespages_with_text text_densityavg_text_per_pagetotal_text_lengthNunknown)rrrsrfrtrurvrwrx) pdfplumberopenr]pages extract_textrroundrr) rhpdfrtrurxpage page_textrvrwpdf_typersrSrrrdetect_pdf_typesZ    ('rc Csz`t|}|ddkr|ddkrdddddWSd}t|}|jD]}|}|r2||d7}q$Wd n1s=wY|}|rY||d|dt||d d WSdddd dWSty{}zdd dt|dWYd }~Sd }~ww)zEExtract text from PDF with enhanced error handling and type detectionrrrqrsrlrz>PDF appears to be scanned/image-based with no extractable text)textrrsreason Nru)rrrs text_lengthpages_processedz5No text could be extracted despite PDF type detectionrfrp)rrrsrf) rrzr{r|r}rr]rr)rhpdf_inforrrrextracted_textrSrrrextract_text_from_pdfsL    rrcsx|gd}gd}tfdd|D}tfdd|D}|dkr(dS|dkr.dS||kr4dS||kr:dSd S) z.Detect document type based on content analysis) z title searchtitlezcertificate of titlezland descriptionzregistered proprietor encumbranceszactivity in the last 125 dayszadministrative noticesztitle referencevolumefolio) zapplication formzplanning permitz proposed userVz for what usezdevelopment or other matterzpermit requiredzonline form applicationzplanning permit applicationc3|] }|vrdVqdSrANr.0keyword text_lowerrr z'detect_document_type..c3rrrrrrrr rr5 title_searchapplication_formry)lowersum)rtitle_search_keywordsapplication_form_keywordstitle_search_matchesapplication_form_matchesrrrdetect_document_type src Cs~|r|s ddiSd}t||kr|d|d}t|}|dkr)tj|d}n|dkr4tj|d}ntj|d}t|\}}|rIdd |iS|sOdd iSz+|d rat d d |}| d }t |}t|tspddiWSd|vrx||d<|WSt jy}ztd|td|ddt|iWYd}~Sd}~wty}zddt|iWYd}~Sd}~ww)z@Extract structured data from PDF text with proper error handlingrfzEmpty or invalid text inputi@NrWr)rrrXrYrZr[rzInvalid response format document_type detected_typez'JSON decoding error in PDF extraction: r\z!Failed to parse OpenAI response: z)Unexpected error in PDF data processing: )rr]rrr^rr rUr*r_r`rarbrcr&r'rdrNrr)rrerr7r<rfrrSrrr!extract_structured_data_from_text/sF         rri@c Csg}tdt||D]_}||||}g}|D]/}z ||} || WqtyG} ztd| |dt| iWYd} ~ qd} ~ ww||||t|kritd||dd|dt|q |S) z9Process items in batches with delays to avoid rate limitsrzError processing item: rfNzProcessed batch rAz , waiting z seconds...) rEr]appendrrNrextendrOrP) r( process_func batch_sizedelay_between_batchesresultsibatch batch_resultsitemresultrSrrrprocess_batch_with_delayjs&  rc Cszddlm}m}m}|jj||ddd\}}|jj||tj || dd| dd| d d | d d| d d|rE| d sEdnd|rR| d rR| d ndd } |r | d s | di} |jjd3id| d| ddd| ddd| ddd| ddd| ddd| ddd| ddd| ddd| ddd| ddd| ddd | d dd!| d!dd"| d"dd#| d#dd$| d$dd%| d%dd&| d&dd'| d'dd(| d(dd)| d)dd*| d*dd+|} d,|| j | j d-d.WS|j y'd/|d0d1YStyD} zd/|d2t| d1WYd} ~ Sd} ~ ww)4a| Save PDF extraction data to the database Args: application_id (str): The application ID pdf_path (str): Path to the PDF file extraction_result (dict): Result from extract_text_from_pdf function extracted_data (dict): Structured data extracted from the PDF Returns: dict: Result with success status and any errors r)r PDFDocumentExtractedPDFDataUnknown)application_id council_name)rdefaultsrrryrsrprrrfsuccessfailedN) application file_path file_namerrrsrrextraction_status error_messagedevelopment_summary pdf_documentland_descriptionrregistered_proprietorractivity_last_125_daysadministrative_notices proposed_userVapplicant_name contact_namecontact_address contact_email contact_phoneapplicant_addressapplicant_emailapplicant_phonelot_size site_coverage total_areaground_floor_areafirst_floor_areapossposraw_extracted_dataTzPDF data saved successfully)rrpdf_document_idrrKFz!Development application not found)rrrfzDatabase error: r)shaoApp.modelsrrrobjects get_or_createrIospathbasenamer)idr DoesNotExistrr) rrhextraction_resultextracted_datarrrrcreatedrrextracted_pdf_datarSrrrsave_pdf_data_to_databases                                 "&  rc CszKt|}|ds$t|||d|ddi}||d|dd|dWSt|d}t||||}||d|d|d |d d |d d ||d WStyo}z||dt|ddt|ddWYd}~Sd}~ww)a  Process a PDF file and save the extracted data to the database Args: application_id (str): The application ID pdf_path (str): Path to the PDF file Returns: dict: Result with processing status and database save status rrfrzNo text extracted from PDFF)rrhextraction_successr database_saveTrrsrrr) rrhrrrsrrrrzProcessing error: )rrf)rrhrrfrN)rr)rrrr)rrhr save_resultrrSrrrprocess_and_save_pdf_datasV        r)r5r6)rir)-rrbpandasrrOrL django.confr django.httprdjango.views.decorators.httprrropenaishaoApp.promptsrrrr r_rzr r r r OpenAIOPENAI_API_KEYrFrrr'r4rr+r,rUrgrrrrrrrrrrrs6    '(073$ ; g