o zj#@sdZddlZddlZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl Z e j e jde eZGdddZGd d d eZGd d d eZddefddZdeeeeffddZedkrrddZeedSdS)zz Market price scraper for CropPilot This module provides tools to scrape agricultural market prices from various sources. N) BeautifulSoup)ListDictAnyOptional)datetime)levelc@s&eZdZdZdedeefddZdS) MarketScraperz"Base scraper class for market dataurlreturnc sz3tjdd4IdH}||IdH}||jWdIdHWS1IdHs-wYWdStyR}ztd|d|WYd}~dSd}~ww)zFetch HTML content from a URLg>@)timeoutNzError fetching : )httpx AsyncClientgetraise_for_statustext Exceptionloggererror)selfr clientresponseer>/var/www/html/CropPilot/backend/app/services/market_scraper.py fetch_pages4zMarketScraper.fetch_pageN)__name__ __module__ __qualname____doc__strrrrrrrr sr c @sFeZdZdZdZedZd dedededeee e ffd d Z dS) AgMarknetScraperz:Scraper for Agmarknet portal (Indian agricultural markets)zhttps://agmarknet.gov.inz/SearchCmmMkt.aspxN commoditystatedistrictr c sz(td}|dvr=dddd|ddd d d|dd dd d |ddd dd|ddddd|ddddd|dgWS|dvrpdddd|ddddd|dd d!d"d#|dd$d%d&d'|dd(d)d*d+|dd,dd-d.|dgWS|d/vrd0d1d2d3|dd4d5d6d |dd7d1d8d9|dd:d1d;d<|dd=d>d?d@|dgWS|dAvrdBdCdDd|ddEdFdGdH|ddIdJdKdL|ddMdNdOdP|ddQdRdSdT|ddUdVdWdX|dgWS|dYvrdZd[dDd\|d]d^d[d_d`|d]dad[dbdc|d]ddd[dedf|d]dgd[dhdi|d]gWS|djvr(dkdldmdn|ddodpdqdr|ddsdtdudv|ddwdxdydz|dd{d|dWd}|dgWSgWStyP}ztd~|d|d|d|gWYd}~Sd}~ww)zFetch prices for a specific commodity across different states in India Note: This is a simplified implementation that would need to be expanded to handle the actual form submission and parsing of the AgMarknet portal. z%Y-%m-%d)wheatgehugahuez Wheat (Gehu)i zPune Mandi, Maharashtra)idcropprice_per_quintallocationdatefzWheat (Premium)i( gizIndore Mandi, Madhya Pradeshhi zLudhiana Mandi, PunjabizWheat (Standard)iL zKarnal Mandi, HaryanajizKanpur Mandi, Uttar Pradesh)onionkandapyazz Onion (Kanda)iFzNashik Mandi, Maharashtraz Onion (Red)izOnion (Medium)izAhmedabad Mandi, Gujaratz Onion (Small)izBengaluru Mandi, Karnatakaz Onion (Large)izAzadpur Mandi, DelhiizKurnool Mandi, Andhra Pradesh)soybeansoyabeani-SoybeanizUjjain Mandi, Madhya Pradeshi.zSoybean (Yellow)i*i/izAkola Mandi, Maharashtrai0izKota Mandi, Rajasthani1zSoybean (Black)i0zDavangere Mandi, Karnataka)ricepaddychawaliz Rice (Common)i izRice (Basmati)i4!zAmritsar Mandi, PunjabizRice (Sona Masuri)ihzNizamabad Mandi, Telanganaiz Rice (Ponni)izThanjavur Mandi, Tamil NaduizRice (Parboiled)i zBardhaman Mandi, West Bengaliz Rice (Joha)i zJorhat Mandi, Assam) sugarcanez sugar caneoosi SugarcanezMill Gate Rate, Maharashtra)r*r+price_per_tonner-r.iiN zMill Gate Rate, Uttar Pradeshii zMill Gate Rate, Karnatakaiil zMill Gate Rate, Tamil Naduii zMill Gate Rate, Gujarat)cottonkapasiYzCotton (Medium Staple)i8zRajkot Mandi, GujaratiZzCotton (Long Staple)izAdilabad Mandi, Telanganai[z Cotton (J-34)izSirsa Mandi, Haryanai\zCotton (Shankar-6)i,zAhmednagar Mandi, Maharashtrai]zCotton (DCH-32)zDharwad Mandi, KarnatakazError fetching prices for z in z, r N)rnowstrftimelowerrrr)rr#r$r%todayrrrr get_prices%sl     "zAgMarknetScraper.get_pricesNN) rrrr BASE_URL SEARCH_URLr!rrrrMrrrrr"s  ,r"c @s<eZdZdZdZd dedededeeeeffdd Z dS) KrishiMandiScraperz/Scraper for KrishiMandi (another market source)zhttps://krishimandi.example.comNr+r$r%r csgS)z)Get prices from Krishi Mandi across Indiar)rr+r$r%rrrrMuszKrishiMandiScraper.get_pricesrN) rrrr rOr!rrrrMrrrrrQps,rQ agmarknetsourcecCs0ttd}||}|std||S)z/Return appropriate scraper based on source name)rR krishimandizUnknown scraper source: )r"rQrrK ValueError)rSscrapers scraper_classrrr get_scraper|srXr csPtdgd}fdd|D}tj|IdH}g}|D]}||q|S)zMGet current prices for common crops across all of India from multiple sourcesrR)r&r4r=rCr@rGcsg|]}|qSr)rM).0r+scraperrr sz"get_all_prices..N)rXasynciogatherextend) common_cropstasksresults all_pricesresultrrZrget_all_pricess re__main__cs$tIdH}ttj|dddS)N)indent)reprintjsondumps)pricesrrrtests rm)rR)r rr]bs4rtypingrrrrrjrlogging basicConfigINFO getLoggerrrr r"rQr!rXrermrunrrrrs&   Q