o h@sddlZddlZddlZddlmZmZmZmZddlZ ddl Z ddl m Z ddl mZmZmZmZmZmZddlmZddlmZddlmZmZddlmZdd lmZdd lm Z dd l!m"Z"d d l#m$Z$m%Z%m&Z&GdddZ'dS)N)ListDictAnyOptional)settings)QSumCountAvgMaxMin)SentenceTransformer) ChatOpenAI) HumanMessage SystemMessage)RecursiveCharacterTextSplitter)HuggingFaceEmbeddings)FAISS)Document)DevelopmentApplication PDFDocumentExtractedPDFDatac@seZdZddZdeefddZddZdd Zd2d e d e deefd dZ d e deede fddZ de de e effddZde defddZde de e effddZde e effddZde e effddZde e effddZde e effdd Zde e effd!d"Zde e effd#d$Zde e effd%d&Zde defd'd(Zde de e effd)d*Zde e effd+d,Zde e effd-d.Zde e effd/d0Zd1S)3RAGChatbotServicecCsHtd|_tddtjd|_tjtj d|_ d|_ t ddt d|_dS) Nall-MiniLM-L6-v2zgpt-4o-2024-05-13g?)model temperatureapi_key faiss_index) chunk_size chunk_overlaplength_function)r embedding_modelrrOPENAI_API_KEYllmospathjoin MEDIA_ROOTvector_store_path vector_storerlen text_splitter)selfr0:/home/shobhit/Desktop/shao (1)/shao/shaoApp/rag_service.py__init__s  zRAGChatbotService.__init__returnc Csg}td|tjdd}|D]H}|jdur%d|jdnd}d|jp,d d |j p2d d |j p8d d |j p>d d |j pDd d|d|j pMd d|jpSd d|jpYd d|jp_d d|jped d|jpkd d|jpqdd}|t|d|j|j ddd|jD]}z|j}|rJdgd|jpd d|jpd d|jpdd |jpdd!|jpdd"|jpdd#|jpdd$|j pdd%|j!pdd&|j"pdd'|j#pdd(|j$pdd)|j%pdd*|j&pdd+|j'pdd,|j(pdd-|j)p dd.|j*p)dd/|j+p2dd0}|t|d1|j|j|jd2d3dWqt,y^} zWYd} ~ qd} ~ wwq|S)4z4Create documents from the database for vectorizationz================pdf_documents__extracted_dataapplication_idN$,.2f Not specifiedu **Development Application Summary** **Application Details:** • **Application ID:** Unknownu • **Council:** u • **Decision:** u% • **Lodgement Date:** u& • **Determined Date:** u# • **Project Cost:** u` **Development Specifications:** • **Development Type:** u* • **Number of Dwellings:** u( • **Number of Storeys:** u' • **Number of Places:** u+ • **Number of Apartments:** u- • **Number of Subdivisions:** u[ **Additional Information:** • **Application URL:** z Not availablez application_metadatametadata)sourcer5 council_nametype) page_contentr;u **PDF Document Information** **Document Details:** • **File Name:** u0 • **Document Type:** u~ **Property Information:** • **Land Description:** u8 • **Registered Proprietor:** u/ • **Encumbrances:** uy **Development Details:** • **Proposed Use:** u. • **Description:** uv **Contact Information:** • **Applicant:** u1 • **Contact Person:** u2 • **Contact Address:** u0 • **Contact Email:** u0 • **Contact Phone:** uu **Site Specifications:** • **Lot Size:** u0 • **Site Coverage:** u- • **Total Area:** u4 • **Ground Floor Area:** u3 • **First Floor Area:** u5 • **Private Open Space:** u> • **Secluded Private Open Space:** z pdf_extracted_dataextracted_data)r<r5pdf_file document_typer>)-printrobjectsselect_relatedprefetch_relatedallorder_bycostr5r=decisionlodgement_datedetermined_datedevelopment_typenumber_of_dwellingsnumber_of_storeysnumber_of_placesnumber_of_apartmentsnumber_of_subdivisionsapplication_urlappendr pdf_documentsrBr) file_namerDland_descriptionregistered_proprietor encumbrances proposed_use descriptionapplicant_name contact_namecontact_address contact_email contact_phonelot_size site_coverage total_areaground_floor_areafirst_floor_areaposspos Exception) r/ documents applicationsappcost_str app_metadatapdf_doc extracted pdf_contenter0r0r1create_documents_from_database#s                   ! 4z0RAGChatbotService.create_documents_from_databasecCstd|}|stddStdt|d|j|}tdt|dtdd d id }t|||_t j |j d d |j |j td|j dS)z%Build and save the FAISS vector storez&Building vector store from database...zNo documents found in databaseNz Splitting z documents into chunks...zCreated z chunksrdevicecpu model_name model_kwargsT)exist_okzVector store saved to ) rErtr-r.split_documentsrrfrom_documentsr,r'makedirsr+ save_local)r/rk split_docs embeddingsr0r0r1build_vector_stores  z$RAGChatbotService.build_vector_storecCsBtj|jrtdddid}tj|j|dd|_tddSdS) z$Load the existing FAISS vector storerrurvrwT)allow_dangerous_deserializationz Vector store loaded successfullyF) r'r(existsr+rr load_localr,rE)r/rr0r0r1load_vector_storesz#RAGChatbotService.load_vector_storequerykcsN|js |s ||jsgStfdddDrd}|jj|d}|S)z0Search for similar documents in the vector storec3s|] }|vVqdSN)lower).0wordrr0r1 sz=RAGChatbotService.search_similar_documents..) how manycounttotal statisticssummaryoverviewlist allshow allall applicationapplication idsrr)r,rranysimilarity_search)r/rr similar_docsr0rr1search_similar_documentssz*RAGChatbotService.search_similar_documents context_docscCsFddd|D}td|dd}t|d}|j||g}|jS)z2Generate response using LLM with retrieved context cSsg|]}|jqSr0)r?)rdocr0r0r1 sz7RAGChatbotService.generate_response..ad You are a knowledgeable and friendly urban planning assistant specializing in development applications and council planning data. Your role is to provide clear, engaging, and well-structured responses about development projects, planning decisions, and council information. **Response Guidelines:** 1. **Be conversational and engaging** - Use a friendly, professional tone 2. **Structure your responses clearly** - Use bullet points, numbered lists, and sections when appropriate 3. **Highlight key information** - Use **bold** for important details like costs, dates, and decisions 4. **Provide context** - Explain what the information means in practical terms 6. **Format numbers nicely** - Use commas for large numbers and proper currency formatting 7. **Be comprehensive** - Include relevant details from the context 8. **Suggest follow-up questions** - End with a helpful suggestion for what they might want to know next **Response Structure:** - Start with a direct answer to their question - Provide supporting details with clear formatting - Use bullet points for lists of applications, types, or statistics - Highlight key metrics and important dates - End with a helpful suggestion or related information **Context Information:** z Remember: You're helping people understand complex planning information in an accessible way. Make the information engaging and easy to understand! )content)r)rrr&invoker)r/rrcontextsystem_message user_messageresponser0r0r1generate_responses z#RAGChatbotService.generate_responsemessagec Csze||r ||WS||r||WS|j|dd}|s'dgddWS|||}g}|D]"}|jdd|jdd|jd d|jd dd }||q1||t |d krad dWSddWSt y}zdt |dgddWYd}~Sd}~ww)z7Main chat method that combines retrieval and generationrru)I apologize, but I couldn't find any relevant information in my database to answer your question. 🤔 **Here are some suggestions to help you get better results:** • **Try rephrasing your question** - Use different keywords or be more specific • **Ask about development applications** - I have data on planning applications, council decisions, and development types • **Query specific councils** - Try asking about "City of Darebin" or other councils in the database • **Ask about development types** - I can help with residential, commercial, industrial, or subdivision projects • **Request statistics** - Ask about application counts, costs, or decision trends **💡 Example questions you can try:** - "How many development applications are in the database?" - "What types of developments are most common?" - "Tell me about applications in City of Darebin" - "What are the recent planning decisions?" - "Show me applications with high costs" Feel free to try again with a different approach! I'm here to help you explore the planning data. 🏗️lowrsources confidencer5r9r=rDr>r5r=rD source_typehighmediumugI'm sorry, but I encountered a technical issue while processing your request. 😔 **What happened:** u **What you can do:** • **Try again** - Sometimes temporary issues resolve themselves • **Rephrase your question** - Use simpler or different wording • **Ask something else** - Try a different type of question about development applications • **Contact support** - If the issue persists, there might be a system problem I'm here to help you explore the planning data, so please don't hesitate to try again! 🏗️errorN) _is_statistical_question_handle_statistical_question_is_bulk_listing_request_handle_bulk_listing_requestrrr;getrVr-rjstr)r/rrrrr source_inforsr0r0r1chatsF           zRAGChatbotService.chatc&gd}|tfdd|DS)z8Check if the question is asking for statistics or counts)rrrz number ofrrrrlprojects developmentscouncilsrrrzall applicationsrc3|]}|vVqdSrr0rkeyword message_lowerr0r1rEz=RAGChatbotService._is_statistical_question..rr)r/rstatistical_keywordsr0rr1r=sz*RAGChatbotService._is_statistical_questionc CszR|}d|vsd|vsd|vr|WSd|vsd|vr#|WSd|vs+d|vr0|WSd|vs8d |vr=|WSd |vsId |vsId |vrN|WS|WStyo}zd t|gddWYd}~Sd}~ww)z9Handle statistical questions with direct database querieszapplication idrrcouncilrdevelopment typeztypes of developmentrLapprovalrKvaluebudgetz4I encountered an error while retrieving statistics: rrN) r_get_all_application_ids_get_council_statistics _get_development_type_statistics_get_decision_statistics_get_cost_statistics_get_comprehensive_statisticsrjrr/rrrsr0r0r1rGs(       z.RAGChatbotService._handle_statistical_questionc Cstj}tj}tj}tjjddd}tdd|D}tjjddd}tdd|D}tjjddd}td d|D} tjjd d } tjjd d j t d d dpad} d|dd|d|dd|dd|d| d| dd| dd} | dddddgd d!S)"z%Get comprehensive database statisticsr=TflatcSg|]}|r|qSr0r0)rcr0r0r1rizCRAGChatbotService._get_comprehensive_statistics..rOcSrr0r0)rdtr0r0r1rlrrLcSrr0r0)rdr0r0r1rorF cost__isnullrKrrru\📊 **Development Applications Database Overview** 🏗️ **📋 Total Applications:** **,u=** development applications **🏛️ Council Coverage:** **u<** councils in the database **📄 Document Processing:** **u7** PDF documents processed **📊 Data Extraction:** **ub** documents with extracted data **🏗️ Development Diversity:** • **Development Types:** **uB** different types of developments • **Decision Categories:** **u4** different decision outcomes • **Cost Data:** **uF** applications with cost information • **Total Project Value:** **$r7u** across all projects **💡 What you can explore:** • Ask about specific councils (e.g., "Tell me about City of Darebin") • Query development types (e.g., "Show me residential projects") • Check decision trends (e.g., "What are the recent approvals?") • Explore high-value projects (e.g., "Show me projects over $1 million") This database contains comprehensive planning information to help you understand development patterns and trends! 🚀zDatabase Statistics All CouncilsStatistical Summarydatabase_queryrrr) rrFrrr values_listdistinctr-filter aggregater) r/total_applications total_pdfstotal_extractedr council_countdevelopment_typesdevelopment_type_count decisionsdecision_countapplications_with_cost total_costrr0r0r1rbs@   "    z/RAGChatbotService._get_comprehensive_statisticscCtjdjtddjddd}d}|d|d 7}|d 7}|d d D]}|d |dd|ddd7}q)|d krN|d|d d7}|dddddgddS)zGet council-specific statisticsr=idrFcouncil_name__isnull-countu%🏛️ **Council Statistics** 📊 **Total Councils:** rz**Applications by Council:** N • **:** rr applications ... and z more councilszCouncil StatisticsrrrrrrrrFvaluesannotater rrJr)r/ council_statsrrr0r0r1r$ " z)RAGChatbotService._get_council_statisticscCr)zGet development type statisticsrOrrFdevelopment_type__isnullru.🏗️ **Development Type Statistics** 📊 **Total Development Types:** rz#**Most Common Development Types:** Nrrrrrrrz more typeszDevelopment Type Statisticsrrrrrrr)r/dev_type_statsrdev_typer0r0r1rrz2RAGChatbotService._get_development_type_statisticscCr)zGet decision statisticsrLrrFdecision__isnullru#📋 **Decision Statistics** 📊 z**Total Decision Categories:** rz**Decision Outcomes:** Nrrrrrrrz more decision typeszDecision Statisticsrrrrrrr)r/decision_statsrrLr0r0r1rrz*RAGChatbotService._get_decision_statisticscCstjjdd}|jtdddpd}|jtdddpd}|jtdd d p*d}|jtdd d p6d}d }|d|dd7}|d7}|d|dd7}|d|dd7}|d|dd7}|d|dd7}|dddddgddS)zGet cost statisticsFrrKrrr)avgr )maxr )minr u💰 **Cost Statistics** 📊 z!**Applications with Cost Data:** rrz**Financial Overview:** u• **Total Project Value:** $r7 u• **Average Project Cost:** $u• **Highest Project Cost:** $u• **Lowest Project Cost:** $zCost Statisticsrrrrrr) rrFrrrr r r r)r/ cost_statsravg_costmax_costmin_costrr0r0r1rs z&RAGChatbotService._get_cost_statisticscCsz;tjjddjddd}|}|dkr!dgdd WSd }|d |d d 7}|d7}d}g}td||D]6}||||}|D])}|jrRd|jdnd} |j pXd} |j p]d} |j pbd} | |j | | | | dqFq:t|dD]/\}} ||dd| dd| dd| dd| dd| dd 7}|d dkr|d7}qvtd!d"|D}td#d"|D}td$d"|D}|d%7}|d&t|d d7}|d't|d7}|d(t|d7}|d)t|d7}|d*|r|ddnd+d7}|d,|r |d-dnd+d7}|d.7}|d/|r|ddnd0d17}|d27}|d37}|d47}|d5d6d7d8d9gd:d WStyZ}zd;t|gdzPGet all application IDs with comprehensive information using parallel processingF)application_id__isnullr@)r5r5ru❌ **No Application IDs Found** I couldn't find any application IDs in the database. This might indicate that the data hasn't been imported yet or there's an issue with the data.rru=📋 **Complete Development Applications Database** 🏗️ u**📊 Total Applications:** **rz** applications u"**📋 All Application Details:** dr6r7r8r9)rrrLrKr>r3d. **rz** | rz | rLrKr>r cs|]}|dVqdS)rNr0rrmr0r0r1rrz=RAGChatbotService._get_all_application_ids..csr)rLNr0rr0r0r1rrcsr)r>Nr0rr0r0r1rru! **📈 Comprehensive Summary:** u• **Total Applications:** u• **Unique Councils:** u• **Decision Types:** u• **Development Types:** u• **First Application:** zN/Au• **Last Application:** u **💡 Next Steps:** u4• Ask about specific applications: "Tell me about z D-101-2023z" uC• Get council statistics: "Show me City of Darebin applications" uD• Check development types: "What residential projects are there?" u1• Explore costs: "Show me high-value projects" zAll ApplicationsrzComplete Application Databaserrrz=I encountered an error while retrieving all application IDs: rN)rrFrGrexcluderJrrangerKr=rLrOrVr5 enumeratesetr-rjr)r/rl total_countr batch_size all_apps_dataibatchrmrnrrLrapp_datarr dev_typesrsr0r0r1rs     B  "" z*RAGChatbotService._get_all_application_idsc Csz)tj}tj}tj}tjjddd}|||t|tj |j dWSt yA}z dt |iWYd}~Sd}~ww)z'Get statistics about the knowledge baser=Tr)rtotal_pdf_documentstotal_extracted_datarvector_store_existsrN)rrFrrrrrlistr'r(rr+rjr)r/rrrrrsr0r0r1get_statistics<s    z RAGChatbotService.get_statisticscr)z8Check if the question is asking for bulk listing of data) rrzall ofeveryz complete listz full listzentire databasez all recordsz all entriesz all itemsc3rrr0rrr0r1rVrz=RAGChatbotService._is_bulk_listing_request..r)r/r bulk_keywordsr0rr1rOsz*RAGChatbotService._is_bulk_listing_requestc Csz5|}d|vs d|vr|WSd|vr|WSd|vs#d|vr(|WSd|vr1|WS|WStyR}zdt|gdd WYd }~Sd }~ww) zrLzBI encountered an error while processing the bulk listing request: rrN)rr_get_all_councils_list_get_all_development_types_list_get_all_decisions_listrrjrrr0r0r1rXs$      z.RAGChatbotService._handle_bulk_listing_requestc CzMtjdjtddjddd}d}|d|d 7}|d 7}t|d D]\}}||d d |dd|ddd7}q)|dddddgddWSt yj}zdt |gddWYd}~Sd}~ww)z(Get all councils with application countsr=rrFrru+🏛️ **All Councils in Database** 📊 rrz***Council List with Application Counts:** r2dr** - rrrz Council ListrzCouncil DatabaserrrrzError retrieving council list: rN rrFrrr rrJrrrjr)r/rrr rrsr0r0r1r,q0 ( z(RAGChatbotService._get_all_councils_listc Cr/)z%Get all development types with countsrOrrFrru4🏗️ **All Development Types in Database** 📊 rrz/**Development Types with Application Counts:** rr0rr1rrrzDevelopment TypesrzDevelopment Type Databaserrrrz$Error retrieving development types: rNr2)r/r#rr rrsr0r0r1r-r3z1RAGChatbotService._get_all_development_types_listc Cr/)z"Get all decision types with countsrLrrFrru.📋 **All Decision Types in Database** 📊 z**Total Decision Types:** rz,**Decision Types with Application Counts:** rr0rr1rrrzDecision TypesrzDecision Databaserrrrz!Error retrieving decision types: rNr2)r/rrr rLrsr0r0r1r.r3z)RAGChatbotService._get_all_decisions_listN)r)__name__ __module__ __qualname__r2rrrtrrrintrrrrrboolrrrrrrrrr(rrr,r-r.r0r0r0r1rs,h'K .T r)(r'jsonpickletypingrrrrnumpynpfaiss django.confrdjango.db.modelsrrr r r r sentence_transformersr langchain_openairlangchain.schemarrlangchain.text_splitterrlangchain_community.embeddingsr langchain_community.vectorstoresrlangchain.docstore.documentrmodelsrrrrr0r0r0r1s"