hSSKrSSKrSSKrSSKJrJrJrJr SSKr SSK r SSK J r SSK JrJrJrJrJrJr SSKJr SSKJr SSKJrJr SSKJr SS KJr SS KJ r SS K!J"r" S S K#J$r$J%r%J&r& "SS5r'g)N)ListDictAnyOptional)settings)QSumCountAvgMaxMin)SentenceTransformer) ChatOpenAI) HumanMessage SystemMessage)RecursiveCharacterTextSplitter)HuggingFaceEmbeddings)FAISS)Document)DevelopmentApplication PDFDocumentExtractedPDFDatac\rSrSrSrS\\4SjrSrSr SS\ S\ S\\4S jjr S\ S \\S\ 4S jr S \ S\\ \44S jrS \ S\4SjrS \ S\\ \44SjrS\\ \44SjrS\\ \44SjrS\\ \44SjrS\\ \44SjrS\\ \44SjrS\\ \44SjrS\\ \44SjrS \ S\4SjrS \ S\\ \44SjrS\\ \44SjrS\\ \44SjrS\\ \44SjrSr g)RAGChatbotServicec[S5Ul[SS[RS9Ul[ RR[RS5Ul SUl [SS[S9Ulg) Nall-MiniLM-L6-v2zgpt-4o-2024-05-13g?)model temperatureapi_key faiss_index) chunk_size chunk_overlaplength_function)rembedding_modelrrOPENAI_API_KEYllmospathjoin MEDIA_ROOTvector_store_path vector_storerlen text_splitter)selfs )/var/www/html/shao/shaoApp/rag_service.py__init__RAGChatbotService.__init__sh23EF%++  "$h.A.A=!Q ; returnc /n[RR5RS5R 5R S5nUGHnUR bSUR S3OSnSR/SPUR=(d S PS PUR=(d S PS PUR=(d S PS PUR=(d S PS PUR=(d S PSPUPSPUR=(d S PSPUR=(d S PSPUR=(d S PSPUR =(d S PSPUR"=(d S PSPUR$=(d S PSPUR&=(d S PSPUR(=(d S PSPUR*=(d SPSP5nUR-[/USURURSS.S95 UR0R 5GH4nUR2nU(GaSR/SPUR4=(d S PSPUR6=(d S PS PUR8=(d SPS!PUR:=(d SPS"PUR<=(d SPS#PUR>=(d SPS$PUR@=(d SPS%PURB=(d SPS&PURD=(d SPS'PURF=(d SPS(PURH=(d SPS)PURJ=(d SPS*PURL=(d SPS+PURN=(d SPS,PURP=(d SPS-PURR=(d SPS.PURT=(d SPS/PURV=(d SPS0PURX=(d SPS1P5nUR-[/US2URUR4UR6S3S4.S95 GM4GM7 GM U$![Za n Sn A GMQSn A ff=f)5z4Create documents from the database for vectorizationpdf_documents__extracted_dataapplication_idN$,.2f Not specifiedu **Development Application Summary** **Application Details:** • **Application ID:** Unknownu • **Council:** u • **Decision:** u% • **Lodgement Date:** u& • **Determined Date:** u# • **Project Cost:** uX **Development Specifications:** • **Category:** u • **Land Use:** u' • **Development Type:** u* • **Number of Dwellings:** u( • **Number of Storeys:** u' • **Number of Places:** u+ • **Number of Apartments:** u- • **Number of Subdivisions:** u[ **Additional Information:** • **Application URL:** z Not availablez application_metadatametadata)sourcer; council_nametype) page_contentrBu **PDF Document Information** **Document Details:** • **File Name:** u0 • **Document Type:** u~ **Property Information:** • **Land Description:** u8 • **Registered Proprietor:** u/ • **Encumbrances:** uy **Development Details:** • **Proposed Use:** u. • **Description:** uv **Contact Information:** • **Applicant:** u1 • **Contact Person:** u2 • **Contact Address:** u0 • **Contact Email:** u0 • **Contact Phone:** uu **Site Specifications:** • **Lot Size:** u0 • **Site Coverage:** u- • **Total Area:** u4 • **Ground Floor Area:** u3 • **First Floor Area:** u5 • **Private Open Space:** u> • **Secluded Private Open Space:** z pdf_extracted_dataextracted_data)rCr;pdf_file document_typerE).robjectsselect_relatedprefetch_relatedallorder_bycostr-r;rDdecisionlodgement_datedetermined_datecategoryland_usedevelopment_typenumber_of_dwellingsnumber_of_storeysnumber_of_placesnumber_of_apartmentsnumber_of_subdivisionsapplication_urlappendr pdf_documentsrH file_namerJland_descriptionregistered_proprietor encumbrances proposed_use descriptionapplicant_name contact_namecontact_address contact_email contact_phonelot_size site_coverage total_areaground_floor_areafirst_floor_areaposspos Exception) r3 documents applicationsappcost_str app_metadatapdf_doc extracted pdf_contentes r4create_documents_from_database0RAGChatbotService.create_documents_from_database#s .55DDFWW + #%)*  C/2hh.B388D/*H %&)%7%7%D9$E F  "..;)<  =  #||8y9  :% &)%7%7%D9$EF&'*&9&9&FY%GH#$,*- #||8y9: #||8y9:'(+';';'Hy&IJ*+.*A*A*NY)OP( ),(=(=(J'K! L'!"(+';';'Hy&I#"J+#$,/+C+C+Py*Q%$R-%&.1-G-G-T9,U'&V&','*&9&9&L_%M-,N -L2   X)4&)&8&8$'$4$4&  ,,0021 ' 6 6I ''+,'-4,=,=,J+K 'L0 ' 180E0E0R/S ' T3 '4=3M3M3`Q`2a'b8'9B8W8W8j[j7k'l/'09/E/E/X.Y'Z/'09/E/E/X.Y'Z.'/8.C.C.V-W'X,'"-6,D,D,W+X#'"Y1#'$2;1G1G1Z?0[%'$\2%'&3<2K2K2^1_''&`0''(1:0G0G0Z?/[)'(\0)'*1:0G0G0Z?/[+'*\++'0,5+=+=+P*Q1'0R01'21:0G0G0Z?/[3'2\-3'4.7-A-A-T_,U5'4V45'65>4O4O4bSb3c7'6d37'84=3M3M3`Q`2a9'8b59':6?]]5Uo4V;':W>;'<?Hnn>_P_=`='<a=' B"(()4*>252D2D,3,=,=181F1F(8 & * G!3S z !s+H,Q&& Q<7Q<c[S5 UR5nU(d [S5 g[S[U5S35 URR U5n[S[U5S35 [ SS S 0S 9n[ R"X#5Ul[R"URS S 9 URRUR5 [SUR35 g)z%Build and save the FAISS vector storez&Building vector store from database...zNo documents found in databaseNz Splitting z documents into chunks...zCreated z chunksrdevicecpu model_name model_kwargsT)exist_okzVector store saved to ) printr{r1r2split_documentsrrfrom_documentsr0r+makedirsr/ save_local)r3rr split_docs embeddingss r4build_vector_store$RAGChatbotService.build_vector_stores 67779  2 3   3y>**CDE''77 B  Z)12+)"E* "00H D**T: $$T%;%;< &t'='=&>?@r7c[RRUR5(a=[ SSS0S9n[ R "URUSS9Ul[S5 gg) z$Load the existing FAISS vector storerr~rrT)allow_dangerous_deserializationz Vector store loaded successfullyF) r+r,existsr/rr load_localr0r)r3rs r4load_vector_store#RAGChatbotService.load_vector_storese 77>>$00 1 1.-&.J!& 0 0&&04!D  4 5r7querykc^^UR(d%UR5(dUR5 UR(d/$UR5m[ U4SjS55(aSnO[ U4SjS55(aSnURR XS9nU$)z0Search for similar documents in the vector storec3*># UH oT;v M g7fN).0word query_lowers r4 =RAGChatbotService.search_similar_documents..sr0qt{"0q) how manycounttotal statisticssummaryoverviewlist allshow allall applicationapplication idsr#c3*># UH oT;v M g7frr)r indicatorrs r4rrsLf EM   !F8_ b%  s5&D"&D"D"-B0D"D"" E ,E>E E c^UR5m/SQn[U4SjU55(ag/SQn[U4SjU55$)z8Check if the question is asking for statistics or countsrc3*># UH oT;v M g7frr)rr message_lowers r4r=RAGChatbotService._is_statistical_question..YsO;NiM);NrF)rrrrrrrsprojects developmentscouncilsrrrzall applicationsrc3*># UH oT;v M g7frrrkeywordrs r4rrdsP;Om+;Orrr)r3rproperty_indicatorsstatistical_keywordsrs @r4r*RAGChatbotService._is_statistical_questionLsI    O;NO O O P;OPPPr7cUR5nSU;d SU;dSU;aUR5$SU;dSU;aUR5$SU;dSU;aUR5$SU;dS U;aUR 5$S U;d S U;dS U;aUR 5$UR 5$![anS [U53/SS.sSnA$SnAff=f)z9Handle statistical questions with direct database querieszapplication idrrcouncilrdevelopment typeztypes of developmentrQapprovalrPvaluebudgetz4I encountered an error while retrieving statistics: rrN) r_get_all_application_ids_get_council_statistics _get_development_type_statistics_get_decision_statistics_get_cost_statistics_get_comprehensive_statisticsrqrr3rrrzs r4r.RAGChatbotService._handle_statistical_questionfs #MMOM =04E4VZdhuZu4466m+z]/J3355#}48NR_8_<<>>}, m0K4466=(G},DTaHa002299;; RSVWXSYRZ[%  s:1B:B:B:,B:!B:*B:: C CC C c[RR5n[RR5n[RR5n[RR SSS9R 5n[UVs/sH oU(dM UPM sn5n[RR SSS9R 5n[UVs/sH o(dM UPM sn5n [RR SSS9R 5n [U V s/sH o(dM U PM sn 5n [RRSS9R5n [RRSS9R[S5S 9S =(d S nS US SUSUS SUS SU SU SU S SUSS3nUSSSSS./SS.$s snfs snfs sn f)z%Get comprehensive database statisticsrDTflatrVrQF cost__isnullrPrrrud📊 **Development Applications Database Overview** 🏗️ **📋 Total Applications:** **,uE** development applications **🏛️ Council Coverage:** **uD** councils in the database **📄 Document Processing:** **u?** PDF documents processed **📊 Data Extraction:** **ur** documents with extracted data **🏗️ Development Diversity:** • **Development Types:** **uJ** different types of developments • **Decision Categories:** **u<** different decision outcomes • **Cost Data:** **uN** applications with cost information • **Total Project Value:** **$r=u** across all projects **💡 What you can explore:** • Ask about specific councils (e.g., "Tell me about City of Darebin") • Query development types (e.g., "Show me residential projects") • Check decision trends (e.g., "What are the recent approvals?") • Explore high-value projects (e.g., "Show me projects over $1 million") This database contains comprehensive planning information to help you understand development patterns and trends! 🚀zDatabase Statistics All CouncilsStatistical Summarydatabase_queryrrr) rrKrrr values_listdistinctr1filter aggregater )r3total_applications total_pdfstotal_extractedrc council_countdevelopment_typesdtdevelopment_type_count decisionsddecision_countapplications_with_cost total_costrs r4r /RAGChatbotService._get_comprehensive_statisticss3;;AAC ((..0 *2288:)11==nSW=Xaac61AQ67 2::FFGY`dFennp!$3D%K3DRb3D%K!L*22>>zPT>U^^` 8Aaa89!7!?!?!F!FTY!F!Z!`!`!b+33:::NXX_bci_jXklstyxy ((:1'=>))68))3A7%%4Q$78&'=%=>((6'784Q78))3D(9:B.!+@R`tIZjkl   C7&L9s$ GG G#,G#. G(;G(c[RRS5R[ S5S9R SS9R S5nSnUSUR5S 3- nUS - nUS S HnUS USSUSSS3- nM UR5S :aUSUR5S - S3- nUSSSSS./SS.$)zGet council-specific statisticsrDidrFcouncil_name__isnull-countu%🏛️ **Council Statistics** 📊 **Total Councils:** rz**Applications by Council:** N • **:** rr applications ... and z more councilszCouncil StatisticsrrrrrrrrKvaluesannotater rrOr)r3 council_statsrrs r4r)RAGChatbotService._get_council_statisticss.66==nMVV+W &e& ,XXh-? =*=+>+>+@*AFF44$Sb)G &!8 9gg>Nq=QQ`a aH*    2 % *]%8%8%:R%?$@O OH!+?Q_sHYijk   r7c[RRS5R[ S5S9R SS9R S5nSnUSUR5S 3- nUS - nUS S HnUS USSUSSS3- nM UR5S :aUSUR5S - S3- nUSSSSS./SS.$)zGet development type statisticsrVr)r*Fdevelopment_type__isnullr-u.🏗️ **Development Type Statistics** 📊 **Total Development Types:** rz#**Most Common Development Types:** Nr/r0r1rrr2r3z more typeszDevelopment Type Statisticsrrrrrrr4)r3dev_type_statsrdev_types r4r2RAGChatbotService._get_development_type_statisticss/77>>?QR[[+\ &%& 0(1C F3N4H4H4J3K4PP::&s+H &*d8GCTUVBWWfg gH,    !B & *^%9%9%;b%@$AM MH!+HZh|Qbrst   r7c[RRS5R[ S5S9R SS9R S5nSnUSUR5S 3- nUS - nUS S HnUS USSUSSS3- nM UR5S :aUSUR5S - S3- nUSSSSS./SS.$)zGet decision statisticsrQr)r*Fdecision__isnullr-u#📋 **Decision Statistics** 📊 z**Total Decision Categories:** rz**Decision Outcomes:** Nr/r0r1rrr2r3z more decision typeszDecision Statisticsrrrrrrr4)r3decision_statsrrQs r4r*RAGChatbotService._get_decision_statisticss/77>>zJSS+T &%& ((); ;5n6J6J6L5MTRR..&s+H &*!5 6d8G;LQ:O_ _H,    !B & *^%9%9%;b%@$AAUV VH!+@R`tIZjkl   r7c[RRSS9nUR[ S5S9S=(d SnUR[ S5S9S=(d SnUR[ S5S 9S =(d SnUR[S5S 9S =(d SnS nUSUR5SS3- nUS- nUSUSS3- nUSUSS3- nUSUSS3- nUSUSS3- nUSSSSS./SS.$)zGet cost statisticsFrrPrrr)avgrF)maxrG)minrHu💰 **Cost Statistics** 📊 z!**Applications with Cost Data:** rrz**Financial Overview:** u• **Total Project Value:** $r= u• **Average Project Cost:** $u• **Highest Project Cost:** $u• **Lowest Project Cost:** $zCost Statisticsrrrrrr) rrKrrr r r r r)r3 cost_statsr&avg_costmax_costmin_costrs r4r&RAGChatbotService._get_cost_statisticssY+33:::N ))F )r@)r)rrQrPrEr3d. **r)z** | rErIc3(# UH oSv M g7f)rNrrrts r4r=RAGChatbotService._get_all_application_ids..<sC]cy>]c3(# UH oSv M g7f)rQNrrVs r4rrW=sE} O}rXc3(# UH oSv M g7f)rENrrVs r4rrW>sA=CK=rXu! **📈 Comprehensive Summary:** u• **Total Applications:** u• **Unique Councils:** u• **Decision Types:** u• **Development Types:** u• **First Application:** zN/Au• **Last Application:** u **💡 Next Steps:** u4• Ask about specific applications: "Tell me about z D-101-2023z" uC• Get council statistics: "Show me City of Darebin applications" uD• Check development types: "What residential projects are there?" u1• Explore costs: "Show me high-value projects" zAll ApplicationsrzComplete Application Databaserrrz=I encountered an error while retrieving all application IDs: rN)rrKrLrexcluderOrrangerPrDrQrVr]r; enumeratesetr1rqr)r3rs total_countr batch_size all_apps_dataibatchrtrurrQr>app_datarr" dev_typesrzs r4r*RAGChatbotService._get_all_application_idss\P 199HHJQQ',Rg!h'(  ',,.Ka!Y!"'YH 9+aH[\ \H = =HJM1k6$q~6 C69hh388D/2OH!..;)G"||8yH"33FH!((!00#*$, ( ( * !7" ): fD$(8x?O>PPRSSr6Q;$H ;C]CCHE}EEIA=AAI = >H 6s=7I!6LBO OH 3CM?"E EH 23y>2B"E EH 5c)n5ERH HH 5P]mA6Ft6Lch5iikl lH 4P]]25Ft5Lch4iikl lH 2 3H OjwP]^_P`aePf~JPKKOP PH ` aH a bH N OH%/ASauTeuvw$   [\_`a\b[cd%  s,A"I%C&IC4I I*I%I*%I*c[RR5n[RR5n[RR5n[RR SSS9R 5nUUU[U5[RRUR5S.$![anS[U50sSnA$SnAff=f)z'Get statistics about the knowledge baserDTr )rtotal_pdf_documentstotal_extracted_datarvector_store_existsrN)rrKrrrrrlistr+r,rr/rqr)r3rrrrrzs r4get_statistics RAGChatbotService.get_statistics[s %!7!?!?!E!E!G $,,224J.66<<>O-55AA.W[A\eegH'9'1(7 N')ww~~d6L6L'M   %SV$ $ %sB=C C! CC!C!cV^/SQnUR5m[U4SjU55$)z8Check if the question is asking for bulk listing of data) rrzall ofeveryz complete listz full listzentire databasez all recordsz all entriesz all itemsc3*># UH oT;v M g7frrrs r4r=RAGChatbotService._is_bulk_listing_request..usI=m+=rr)r3r bulk_keywordsrs @r4r*RAGChatbotService._is_bulk_listing_requestns(   I=IIIr7c^UR5nSU;dSU;aUR5$SU;aUR5$SU;dSU;aUR5$SU;aUR 5$UR 5$![ anS[U53/SS .sS nA$S nAff=f) zrzs r4rx1RAGChatbotService._get_all_development_types_lists! .66==>PQZZDk[fef4XXh5G PH 7 8I7J$O OH J JH(A6 fD2D)E(FeHU\L]^_K``opp 7%/BTbvQbrst$   B3q6(K%  rc[RRS5R[ S5S9R SS9R S5nSnUSUR5S 3- nUS - n[US 5Hup4X#S S USSUSSS3- nM USSSSS./SS.$![anS[U53/SS.sSnA$SnAff=f)z"Get all decision types with countsrQr)r*FrAr-u.📋 **All Decision Types in Database** 📊 z**Total Decision Types:** rz,**Decision Types with Application Counts:** rr|rSr}rrr2zDecision TypesrzDecision Databaserrrrz!Error retrieving decision types: rNr~)r3r"rrcrQrzs r4ry)RAGChatbotService._get_all_decisions_lists .66==jIRRDkSfef,XXh-? JH 4Y__5F4GtL LH G GH(A6 fD*)=(>eHWDUVWCXXghh 7%/?Q_sFWghi$   ?AxH%  r)r(r*r2r0r/N)r)!__name__ __module__ __qualname____firstlineno__r5rrr{rrrintrrrrrboolrrr rrrrrrmrrrwrxry__static_attributes__rr7r4rrs  fXfPA< cc$x.2- s- $x.- S- ^ICIDcNIVQQQ4CDcN6, tCH~, \ c3h , $sCx. , $sCx. , d38n ,R$sCx.Rh%S#X%&JJJCDcN2S#X4c3h4c3hr7r)(r+jsonpickletypingrrrrnumpynpfaiss django.confrdjango.db.modelsrr r r r r sentence_transformersrlangchain_openairlangchain.schemarrlangchain.text_splitterrlangchain_community.embeddingsr langchain_community.vectorstoresrlangchain.docstore.documentrmodelsrrrrrr7r4rsJ ,, 995'8B@20III I r7