hQSSKrSSKrSSKrSSKrSSKrSSKJr SSKJ r SSK J r SSK J r SSKrSSKJrJrJrJr SSKrSSKrSSKJrJrJrJr \R4"\R6S9rS rS rS \S \4S jr SS\!S\"S\#4Sjjr$S\!S \4Sjr%S\!S \4Sjr&S\!S \4Sjr'S\!S \!4Sjr(S\!S \4Sjr)SSjr*Sr+Sr,g)N)settings) JsonResponse)require_http_methods)DevelopmentApplication)EXTRACT_STRUCTURED_DATA_PROMPTTITLE_SEARCH_EXTRACTION_PROMPT"APPLICATION_FORM_EXTRACTION_PROMPT APPLICATION_FORM_FALLBACK_PROMPT)RateLimitErrorAPIErrorAPIConnectionErrorAPITimeoutError)api_keycl[R"U5R5$![a gf=f)N)pd to_datetimedate Exception)date_strs 6/home/developer/Desktop/shao/shao/shaoApp/functions.py parse_daters1~~h',,.. s #& 33c~[R"U5(d US:XdUcg[U5R5$)z7Safely get value, handling None, NaN, and empty stringsN)risnastrstrip)vals rsafe_getrs- wws||sbyCK s8>> datareturnc &[U[5(d0$SSSSSSSS.n0nUR5GH7up4URX45nUS:XaU(dURS5n[U[5(aU(aUVs/sH[n[U[5(dMURS5(dM2[ URSS55R 5PM] nnU(aSRU5OUnURS5(a+US;a[[[ U555OSX#'GMU(a[ U5R 5OSX#'GM: U$s snf![[4a SX#'GM[f=f) z7Validate and clean structured data from OpenAI responserr)development_typeland_usenumber_of_dwellingsnumber_of_storeysnumber_of_placesnumber_of_unitsnumber_of_lotsr$ land_usesz, number_of_)Nr) isinstancedictitemsgetlistrrjoin startswithintfloat ValueError TypeError) r expected_fieldsvalidated_datafield default_valuevaluer*itemextracted_usess rvalidate_structured_datar>"sr dD ! !  ON / 5 5 7. J -I)T**y!*" )!$-:26((:2F:CR01779 )" 6D .1   L ) ) *BGzBYE#e*,=(>_`%;@CJ$4$4$6RN !+!8. !" * *()% *s$ E2%E2=-E2'E77FFprompt max_retries base_delayc j[U5Hgn[RRR SSUS./SSS9nUR SR RR5nUS4s $ g![a|nX1S - :aUUS U--[R"SS 5-n[S US S US -SUS35 [R"U5 SnAMSSUS[U534sSnAs $SnAf[ ["[$4a}nX1S - :aVUS U--[R"SS 5-n[SUS S US -SUS35 [R"U5 SnAGMcSSUS[U534sSnAs $SnAf[&anSS[U534sSnAs $SnAff=f)z@Make OpenAI API request with exponential backoff and retry logiczgpt-4o-2024-05-13user)rolecontentg?i)modelmessages temperature max_tokensrNzRate limit hit, retrying in z.2fz seconds... (attempt /)zRate limit exceeded after z attempts: zAPI error, retrying in zAPI error after zUnexpected error: )NzMax retries exceeded)rangeclientchat completionscreatechoicesmessagerErr randomuniformprinttimesleeprr r rr)r?r@rAattemptresponserEedelays rmake_openai_request_with_retryr^Ms% 7{{..55)#)f=> 6H&&q)1199??AGD= &D (- [q("a7l3fnnQ6JJ4U3K?TU\_`U`Taabcnboopqr 5!9+kRUVWRXQYZZZ,o> Qq("a7l3fnnQ6JJ/c{:OPWZ[P[}\]^i]jjklm 5!/ }KAxPPP 7-c!fX66 6 7sPA"A88 F2AC9C91F29F2AF /F F2 F2F-%F2-F2 descriptioncU(aUR5(d0$Sn[U5U:aUSUS-n[R"US9n[ U5up4U(a[ SU35 0$U(d [ S5 0$UR S5(a)[R"SS U5nURS5n[R"U5n[ S U5 [U5n[ S U5 U$![Ra(n[ S U35 [ S U35 0sSnA$SnAf[an[ SU35 0sSnA$SnAff=f)zCExtract structured data from description with proper error handlingiN...)r_OpenAI request failed: Empty response from OpenAI``` ^```[a-z]*\n?rzdata================r8zJSON decoding error: Raw content: z%Unexpected error in data processing: )rlenrformatr^rWr2resubrstripjsonloadsr>JSONDecodeErrorr)r_ max_lengthr?rEerrorr r8r\s rextract_structured_datarqusM k//11 J ;*$!+:.6 + 2 2{ KF4F;NG  'w/0  *+    e $ $ff-r7;GnnU+Gzz'" $T*1$7 ~.    %aS)* gY'(  5aS9: s1A9C??ED60E6 EEEEpdf_pathc [R"U5n[UR5nSnSnURHNnUR 5nU(dM[UR 55S:dM;US- nU[U5- nMP US:aX2- OSnUS:aXB- OSnUS:a US:aSn Sn O&US:a US :aSn S n OUS:a US:aS n S n OS n Sn U U UU[ US5[ US5US.sSSS5 $!,(df  g=f![an SS [U 5SSSSSS.sSn A $Sn A ff=f)z>Detect if PDF is digital (text-based) or scanned (image-based)r rJgffffff?ddigitalhighg333333?2mediummixedlowscanned)type confidence total_pagespages_with_text text_densityavg_text_per_pagetotal_text_lengthNunknown)r~rrprrrrr) pdfplumberopenrgpages extract_textrroundrr) rrpdfrrrpage page_textrrpdf_typerr\s rdetect_pdf_typersk2 __X &#cii.KO !   --/ 9Y__%6!7"!<#q(O%Y7% "=H!O?8QRLCNQR? 1 ?XY s"'83'>$# $):R)?$% !&7"&<"" $# !(*#2 %lA 6%*+DD9BD< D DDD D?!D:4D?:D?c[U5nUSS:XaUSS:XaSSSSS.$Sn[R"U5nURH#nUR 5nU(dMX%S-- nM% S S S 5 UR 5nU(aUUSUS[ U5US S .$SSSS S.$!,(df  NF=f![anSS S[U5S.sS nA$S nAff=f)zEExtract text from PDF with enhanced error handling and type detectionr~r|rrwrz>PDF appears to be scanned/image-based with no extractable text)textrrreason Nr)rrr text_lengthpages_processedz5No text could be extracted despite PDF type detectionrpr{)rrrrp) rrrrrrrgrr)rrpdf_inforrrrextracted_textr\s rextract_text_from_pdfrs. "8, F y (Xl-Cv-M%$Z   __X &#  --/ 9,,D"'  &$V,&|4">2#+,=#>  %$Q  %' &2  V    sE#CC%B2' B228C+C2 C<C C' C"C'"C'rc^UR5m/SQn/SQn[U4SjU55n[U4SjU55nUS:agUS:agX4:agXC:agg) z.Detect document type based on content analysis) z title searchtitlezcertificate of titlezland descriptionzregistered proprietor encumbranceszactivity in the last 125 dayszadministrative noticesztitle referencevolumefolio) zapplication formzplanning permitz proposed user_z for what usezdevelopment or other matterzpermit requiredzonline form applicationzplanning permit applicationc34># UHoT;dM Sv M g7frJN.0keyword text_lowers r 'detect_document_type..&s_0EWT^I^qq0E c34># UHoT;dM Sv M g7frrrs rrr's"g4M\fQf114Mrr} title_searchapplication_formr)lowersum)rtitle_search_keywordsapplication_form_keywordstitle_search_matchesapplication_form_matchesrs @rdetect_document_typerssJ!_0E__""g4M"ggq  !Q &!  8 ! 8!rc>U(aUR5(dSS0$Sn[U5U:aUSUS-n[U5nUS:Xa[R"US9nO/US:Xa[ R"US9nO[ R"US9n[U5upEU(aSS U30$U(dSS 0$URS 5(a)[R"S S U5nURS 5n[R"U5n[U[5(dSS0$SU;aX&S'U$![R a6n[#SU35 [#SU35 SS[%U530sSnA$SnAf[&anSS[%U530sSnA$SnAff=f)z@Extract structured data from PDF text with proper error handlingrpzEmpty or invalid text inputi@Nrar)rrrbrcrdrerzInvalid response format document_type detected_typez'JSON decoding error in PDF extraction: rfz!Failed to parse OpenAI response: z)Unexpected error in PDF data processing: )rrgrrrhr r r^r2rirjrkrlrmr,r-rnrWrr)rrorr?rErpr r\s r!extract_structured_data_from_textr6s tzz||677J 4y:KZ 5().M&/66DA , ,3::E288dC4F;NG 25':;; 566O   e $ $ff-r7;GnnU+Gzz'"$%%67 7 $ &$1 !   G 7s;< gY'(CJ & $Q]Q%6$7zBWAXXcd e JJ, -#.& N 8/s34$$gs1v%677 8sB C(*CCcHSSKJnJnJn URR UUSS.S9upxURR UU[RRU5URSS5URSS5URS S 5URS S5URS S5U(aURS 5(dSOSU(a'URS 5(aURS 5OSS9 n U(GaURS 5(GdURS05n URR U URSS5URSS5URSS5URSS5URSS5URSS5URSS5URSS5URSS5URSS5URSS5URSS5URS S5URS!S5URS"S5U RS#S5U RS$S5U RS%S5U RS&S5U RS'S5U RS(S5U RS)S5US*9n S+UU RU RS,S-.$!WRa S.US/S0.s$[an S.US1[U 53S0.sSn A $Sn A ff=f)2aP Save PDF extraction data to the database Args: application_id (str): The application ID pdf_path (str): Path to the PDF file extraction_result (dict): Result from extract_text_from_pdf function extracted_data (dict): Structured data extracted from the PDF Returns: dict: Result with success status and any errors r)r PDFDocumentExtractedPDFDataUnknown)application_id council_name)rdefaultsrrrrr{rrrpsuccessfailedN) application file_path file_namerrrrrextraction_status error_messagedevelopment_summaryland_descriptionrregistered_proprietorractivity_last_125_daysadministrative_notices proposed_user_applicant_name contact_namecontact_address contact_email contact_phoneapplicant_addressapplicant_emailapplicant_phonelot_size site_coverage total_areaground_floor_areafirst_floor_areaposspos) pdf_documentrrrrrrr_rrrrrrrrrrrrrrrraw_extracted_dataTzPDF data saved successfully)rrpdf_document_idrrTFz!Development application not found)rrrpzDatabase error: )shaoApp.modelsrrrobjects get_or_createrRospathbasenamer/idr DoesNotExistrr) rrrextraction_resultextracted_datarrrrcreatedrrextracted_pdf_datar\s rsave_pdf_data_to_databasers>W XX 6==KK)"0 ) L  #**11#gg&&x0(,,_iH&**:yA(,,\5A)--mQ?-112CQG+9.BTBTU\B]B]ick9GNL^L^_fLgLg.,,W5mq2   ."4"4W"="="0"4"45JB"O !1!9!9!@!@)"0!3!34F!K&4&8&89PRT&U+//C'5'9'9:RTV'W'5'9'9:RTV'W,//C*..}bA .112BBG+//C . 2 23Db I,00"E,00"E"0"4"45H""M . 2 23Db I . 2 23Db I-00R@155orJ.22<D"5"9"9:Mr"R!4!8!89KR!P'++E26(,,VR8$2E"A#" L,+!-!?!?4    " . . ,8   ,'Ax0   s$K!K$$L!< L!LL!L!c [U5nURS5(d8[UUUSURSS505nUUSURSS5US.$[US5n[UUUU5nUUSURS5URS 5URS S 5URS S 5UUS . $![a*nUUS[ U5SS[ U53S.S.sSnA$SnAff=f)z Process a PDF file and save the extracted data to the database Args: application_id (str): The application ID pdf_path (str): Path to the PDF file Returns: dict: Result with processing status and database save status rrprzNo text extracted from PDFF)rrrextraction_successr database_saveTrrrrr) rrrrrrrrrrzProcessing error: )rrp)rrrrrprN)rr/rrrr)rrrr save_resultrr\s rprocess_and_save_pdf_datars:6 1(; $$V,,3!+//:VWX K#1$&++//:VW!,  ;;LV;TU0      - "&)--j9+// =,00B0445FJ,(     , "'V -c!fX6    s%ACA%C C5 C0*C50C5)r}g?)rtg@)-rrlpandasrrXrU django.confr django.httprdjango.views.decorators.httprrropenaishaoApp.promptsrrr r rirr r r rOpenAIOPENAI_API_KEYrOrrr-r>rr3r4r^rqrrrrrrrrrrrs  $=1 aa PP x66 7(4(D(V%(3%(S%(RW%(P000f4 c4 d4 n0 C0 D0 f!s!s!H8OC8OD8Ov4d NA r