@h$`dZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl Z e je je jeZGddZGd d eZGd d eZdd efdZdeeeeffdZedk(rdZej2eyy)z{ Market price scraper for FarmGenius This module provides tools to scrape agricultural market prices from various sources. N) BeautifulSoup)ListDictAnyOptional)datetime)levelc&eZdZdZdedeefdZy) MarketScraperz"Base scraper class for market dataurlreturncnK tjd4d{}|j|d{}|j|jcdddd{S7J737 #1d{7swYyxYw#t $r%}t jd|d|Yd}~yd}~wwxYww)zFetch HTML content from a URLg>@)timeoutNzError fetching : )httpx AsyncClientgetraise_for_statustext Exceptionloggererror)selfr clientresponsees JC:\Users\aadil\OneDrive\Desktop\404\backend\app\services\market_scraper.py fetch_pagezMarketScraper.fetch_pages ((66&!'C0))+}}76607666  LL?3%r!5 6 sB5BA)BA/A+A/ B#A-$B(B5)B+A/-B/B5A8 6B=BB5B B2 B-(B5-B22B5N)__name__ __module__ __qualname____doc__strrrrr r s, C HSM r%r c HeZdZdZdZedZd dedededeee e ffd Z y) AgMarknetScraperz:Scraper for Agmarknet portal (Indian agricultural markets)zhttps://agmarknet.gov.inz/SearchCmmMkt.aspxN commoditystatedistrictr c K tjjd}|jdvr,dddd|ddd d d|dd dd d |ddd dd|ddddd|ddddd|dgS|jdvr,dddd|ddddd|dd d!d"d#|dd$d%d&d'|dd(d)d*d+|dd,dd-d.|dgS|jd/vr%d0d1d2d3|dd4d5d6d |dd7d1d8d9|dd:d1d;d<|dd=d>d?d@|dgS|jdAvr,dBdCdDd|ddEdFdGdH|ddIdJdKdL|ddMdNdOdP|ddQdRdSdT|ddUdVdWdX|dgS|jdYvr%dZd[dDd\|d]d^d[d_d`|d]dad[dbdc|d]ddd[dedf|d]dgd[dhdi|d]gS|jdjvr%dkdldmdn|ddodpdqdr|ddsdtdudv|ddwdxdydz|dd{d|dWd}|dgSgS#t$r-}t j d~|d|d|d|gcYd}~Sd}~wwxYww)zFetch prices for a specific commodity across different states in India Note: This is a simplified implementation that would need to be expanded to handle the actual form submission and parsing of the AgMarknet portal. z%Y-%m-%d)wheatgehugahuez Wheat (Gehu)i zPune Mandi, Maharashtra)idcropprice_per_quintallocationdatefzWheat (Premium)i( gizIndore Mandi, Madhya Pradeshhi zLudhiana Mandi, PunjabizWheat (Standard)iL zKarnal Mandi, HaryanajizKanpur Mandi, Uttar Pradesh)onionkandapyazz Onion (Kanda)iFzNashik Mandi, Maharashtraz Onion (Red)izOnion (Medium)izAhmedabad Mandi, Gujaratz Onion (Small)izBengaluru Mandi, Karnatakaz Onion (Large)izAzadpur Mandi, DelhiizKurnool Mandi, Andhra Pradesh)soybeansoyabeani-SoybeanizUjjain Mandi, Madhya Pradeshi.zSoybean (Yellow)i*i/izAkola Mandi, Maharashtrai0izKota Mandi, Rajasthani1zSoybean (Black)i0zDavangere Mandi, Karnataka)ricepaddychawaliz Rice (Common)i izRice (Basmati)i4!zAmritsar Mandi, PunjabizRice (Sona Masuri)ihzNizamabad Mandi, Telanganaiz Rice (Ponni)izThanjavur Mandi, Tamil NaduizRice (Parboiled)i zBardhaman Mandi, West Bengaliz Rice (Joha)i zJorhat Mandi, Assam) sugarcanez sugar caneoosi SugarcanezMill Gate Rate, Maharashtra)r0r1price_per_tonner3r4iiN zMill Gate Rate, Uttar Pradeshii zMill Gate Rate, Karnatakaiil zMill Gate Rate, Tamil Naduii zMill Gate Rate, Gujarat)cottonkapasiYzCotton (Medium Staple)i8zRajkot Mandi, GujaratiZzCotton (Long Staple)izAdilabad Mandi, Telanganai[z Cotton (J-34)izSirsa Mandi, Haryanai\zCotton (Shankar-6)i,zAhmednagar Mandi, Maharashtrai]zCotton (DCH-32)zDharwad Mandi, KarnatakazError fetching prices for z in z, rN)rnowstrftimelowerrrr)rr(r)r*todayrs r get_priceszAgMarknetScraper.get_prices%s9 C LLN++J7E $==T_xCHI(9PTb{FKLT_}HMN(9PTbzEJK(:QUczEJKT_|GLM "&@@d`{FKL D^yDIJ(8ta{FKLc_{FKLd`vAFG D^}HMN "&== ZxCHI(:QUdBLQR ZtDE Zq|AB(9PTb~INO "&AAd`wBGH(8tayDIJ(UYhCMRSd`vAFG("F93F>4G9F>>GNN) rr r!r"BASE_URL SEARCH_URLr#rrrrSr$r%rr'r'sPD)H:/0JI#IcICI[_`dehjmem`n[oIr%r'c >eZdZdZdZd dedededeeeeffdZ y) KrishiMandiScraperz/Scraper for KrishiMandi (another market source)zhttps://krishimandi.example.comNr1r)r*r cKgSw)z)Get prices from Krishi Mandi across Indiar$)rr1r)r*s rrSzKrishiMandiScraper.get_pricesus  srT) rr r!r"rUr#rrrrSr$r%rrXrXps>90HSsVZ[_`ceh`h[iVjr%rXsourcecttd}|j|j}|st d||S)z/Return appropriate scraper based on source name) agmarknet krishimandizUnknown scraper source: )r'rXrrQ ValueError)rZscrapers scraper_classs r get_scraperra|sE&)H LL0M 3F8<== ?r%r cKtd}gd}|Dcgc]}|j|}}tj|d{}g}|D]}|j ||Scc}w7%w)zMGet current prices for common crops across all of India from multiple sourcesr\)r,r:rCrIrFrMN)rarSasynciogatherextend)scraper common_cropsr1tasksresults all_pricesresults rget_all_pricesrlsx+&GPL2> ?,$W   %,E ?NNE**GJ&!  @*sA-A&A-A+&A-__main__ctKtd{}ttj|dy7%w)N)indent)rlprintjsondumps)pricess rtestrus)%'' djj*+(s 86&8)r\)r"rrcbs4rtypingrrrrrrrlogging basicConfigINFO getLoggerrrr r'rXr#rarlrurunr$r%rr}s ,, ',,'   8 $  O}Ob    d4S>2" z,GKK r%