tj\UdZddlZddlZddlmZddlmZmZddlm Z ddl m Z m Z ddl Z ddlmZeegefdzed<e jrdd lmZndZiZeee fed <iZeeefed <Gd d ZGddZGddeZeegefdzed<e jrddlmZ ndZ GddeZ!eegefdzed<e j"rddlm#Z$ndZ$GddeZ%e GddZ&GddeZ'GddeZ(Gd d!eZ)iZ*eee+efed"<d#a,d$ee j-zd%e+ed&dfd'Z.d$ee j-zd&e+efd(Z/d&ee0ee+effd)Z1d+d*Z2dS),a Device abstraction layer for TorchDynamo and Inductor backends. This module provides a unified interface for different hardware backends (CUDA, XPU, CPU, MPS, MTIA) through a common device interface. Key components include: - DeviceInterface: Base class defining the common API for all device types - Device-specific implementations: CudaInterface, XpuInterface, CpuInterface, MpsInterface, MtiaInterface - Device registration system for managing available backends - Worker APIs for multi-processing scenarios - Stream and event management across different devices - Device property caching for worker processes The abstraction layer enables device-agnostic code in TorchDynamo while allowing specialized implementations for each hardware backend's unique features. N namedtuple)CallableIterable) dataclass)AnyLiteral has_torch_tpuget_cuda_stream)_cuda_getCurrentRawStream caching_worker_device_propertiescaching_worker_current_devicesc eZdZdZGddZGddZGddZGdd Zed e fd Z ede j j d d fd Zede d e fdZede d e fdZed e fdZed efdZede jd efdZed e jfdZede jd d fdZede de de d d fdZede d e fdZed'de j j d d fdZed'de j j d efdZed'de j j d efdZed(d ed efd!Ze d(d"e jd ed efd#Z ed'de j j d e fd$Z!ed'de j j d efd%Z"ed'de j j d d fd&Z#d S))DeviceInterfacez This is a simple device runtime interface for Inductor. It enables custom backends to be integrated with Inductor in a device-agnostic semantic. c2eZdZdejjdefdZdS)DeviceInterface.devicedevicereturnctNNotImplementedErrorclsrs \/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/torch/_dynamo/device_interface.py__new__zDeviceInterface.device.__new__/s% %N)__name__ __module__ __qualname__torchtypesDevicerrrrrr.s; &!3 & & & & & & &rrc"eZdZdededefdZdS)DeviceInterface.Eventargskwargsrc td)NzYEvent should be inherited from torch.Event, otherwise, it couldn't be captured by dynamo.rrr(r)s rrzDeviceInterface.Event.__new__3s%k rNrr r!rrr%rrEventr'2=  s s      rr-c"eZdZdededefdZdS)DeviceInterface.Streamr(r)rc td)Nz[Stream should be inherited from torch.Stream, otherwise, it couldn't be captured by dynamo.rr+s rrzDeviceInterface.Stream.__new__9s%m rNr,r%rrStreamr08r.rr2ceZdZdZededdfdZedefdZeddej j de fdZ dS) DeviceInterface.Workera Worker API to query device properties that will work in multi processing workers that cannot use the GPU APIs (due to processing fork() and initialization time issues). Properties are recorded in the main process before we fork the workers. rrNctrrrs r set_devicez!DeviceInterface.Worker.set_deviceF% %rctrrr%rrcurrent_devicez%DeviceInterface.Worker.current_deviceJr8rctrrr6s rget_device_propertiesz,DeviceInterface.Worker.get_device_propertiesNr8rr) rr r!__doc__ staticmethodintr7r:r"r#r$rr<r%rrWorkerr4>s    &s &t & & &  &  & & & &  &  & &%+*< & & & &  & & &rr@rctrrr%rrr:zDeviceInterface.current_deviceR!!rNctrrr6s rr7zDeviceInterface.set_deviceVrBrctrrr6s rmaybe_exchange_devicez%DeviceInterface.maybe_exchange_deviceZrBrctrrr6s rexchange_devicezDeviceInterface.exchange_device^rBrctrrr%rr device_countzDeviceInterface.device_countbrBrctrrr%rr is_availablezDeviceInterface.is_availablefrBrstreamctrrrLs rrLzDeviceInterface.streamjrBrctrrr%rrcurrent_streamzDeviceInterface.current_streamnrBrctrrrNs r set_streamzDeviceInterface.set_streamrrBr stream_id device_index device_typectrr)rSrTrUs r_set_stream_by_idz!DeviceInterface._set_stream_by_idvrBr device_idxctrrrXs rget_raw_streamzDeviceInterface.get_raw_streamzrBrctrrr6s r synchronizezDeviceInterface.synchronize~rBrc6|j|Sr)r@r<rs rr<z%DeviceInterface.get_device_propertiessz//777rctrrr6s rget_compute_capabilityz&DeviceInterface.get_compute_capabilityrBrFincluding_emulationctrrras ris_bf16_supportedz!DeviceInterface.is_bf16_supportedrBrdtypecL|tjkp||Sr)r"bfloat16rdrreras ris_dtype_supportedz"DeviceInterface.is_dtype_supporteds%&T#*?*?@S*T*TTrctrrr6s rmemory_allocatedz DeviceInterface.memory_allocatedrBrcdS)z Returns True if the device has Triton support, False otherwise, even if the appropriate Triton backend is not available. Fr%r6s ris_triton_capablez!DeviceInterface.is_triton_capables urcL|stddS)aH Raises a `RuntimeError` with the appropriate human-readable instructions to resolve the issue if Triton is not available for the given device, or the default device if `device` is `None`. The caller should ensure the presence of the 'triton' package before calling this method. z/This device is not capable of supporting TritonN)rm RuntimeErrorrs rraise_if_triton_unavailablez+DeviceInterface.raise_if_triton_unavailables4$$&& RPQQ Q R RrrF)$rr r!r=rr-r2r@r>r?r:r"r#r$r7rErGrIboolrKrrLrPrRrWr[r] classmethodr<r`rdrerirkrmrpr%rrrr(s &&&&&&&&  &&&&&&&&("C"""\""5;-"$"""\""c"c"""\""""""\""#"""\""$"""\""u|""""\""EL"""\""5<"D"""\""S""#"RV"""\""3"3"""\"""EK."$"""\"885;+=8888[8""u{'9"S"""\"""t""""\"=BUUKU6:U UUU[U ""!3"s"""\"%+"4\ R R1C Rt R R R[ R R Rrrc deZdZdZdeededzddfdZd dZde d e d e de d fd Z dS) DeviceGuarda_ This class provides a context manager for device switching. This is a stripped down version of torch.{device_name}.device. The context manager changes the current device to the given device index on entering the context and restores the original device on exiting. The device is switched using the provided device interface. device_interfaceindexNrc0||_||_d|_dS)N)rvidxprev_idx)selfrvrws r__init__zDeviceGuard.__init__s!1 rc`|j&|j|j|_dSdSr)rzrvrGr{)r|s r __enter__zDeviceGuard.__enter__s0 8  1AA$(KKDMMM rtypevalue tracebackFc\|j$|j|j|_dS)NF)rzrvrEr{)r|rrrs r__exit__zDeviceGuard.__exit__s) 8 ,BB4=QQDHurrN) rr r!r=rrr?r}rrr rr%rrrurus $_ 5>ADj LLLLSrrucleZdZejjZejjZejjZGddZe ejj Z e ejj Z e ejj Z e ejj Z e ejjZe ejjZe ejjZe ejjZe ejjZe eZe ejjZe ejjZe ejjZe ejjZe defdZe d dejjdee zfdZ!e d dejjdefdZ"e d dejjddfd Z#dS) CudaInterfaceceZdZededdfdZedefdZeddejj de fdZ dS)CudaInterface.WorkerrrNc|td<dSNcudarr6s rr7zCudaInterface.Worker.set_device5; *6 2 2 2rcjdtvr tdStjSr)rr"rr:r%rrr:z#CudaInterface.Worker.current_device-7775f==:,,.. .rc|mt|tr7tj|}|jdkrt d|jdt|tjr|j}|tj }dtvr?dttj D}|td<td|S)Nrz"Expected device type 'cuda', got ''cLg|]!}tj|"Sr%)r"rr<.0is r z>CudaInterface.Worker.get_device_properties..8J44Q77r) isinstancestrr"rrAssertionErrorrwrr@r:rrangerrIr device_props rr<z*CudaInterface.Worker.get_device_properties!fc**"\&11F{f,,,OOOOfel33*#\F~&-<<>>==="5:#:#:#<#<== r?r7r:r"r#r$rr<r%rrr@r  tjSr)r"rrKr%rrrKzCudaInterface.is_availablesz&&(((rNrctjj*tj|\}}|dz|zStj|jdddS)N :r)r"versionhiprget_device_capabilityr< gcnArchNamesplit)rmajormins rr`z$CudaInterface.get_compute_capabilityse =  $99&AAJE32:# #:33F;;GMMcSTUUVWX Xrcxtjjdup'tj|jdkS)N)r"rrrr<rr6s rrmzCudaInterface.is_triton_capables7 M T ) Cz//77=B rc`ddlm}t|sr:r7rIrLrPrRrWr]r<r r[_exchange_devicerG_maybe_exchange_devicerErkrdrrrKr#r$r?rr`rmrpr%rrrrsb Z F J E Z F D D D D D D D DD"\%*";<??O(L)JKK#|EJ$?@@$ UZ%ABB)$)))\)YYu{'9YS3YYYY\Y  %+"4    \  M MEK,> M$ M M M\ M M Mrrget_mtia_stream)_mtia_getCurrentRawStreamcfeZdZejjZejjZejjZGddZe ejj Z e ejj Z e ejj Z e ejj Z e ejjZe ejjZe ejjZe ejjZe ejjZe eZe ejjZe ejjZe ejjZe ejjZe defdZe d dejjdefdZ e d dejjdefdZ!e d dejjddfd Z"dS) MtiaInterfaceceZdZededdfdZedefdZeddejj de fdZ dS)MtiaInterface.WorkerrrNc|td<dSNmtiarr6s rr7zMtiaInterface.Worker.set_device6rrcjdtvr tdStjSr)rr"rr:r%rrr:z#MtiaInterface.Worker.current_device:rrc|mt|tr7tj|}|jdkrt d|jdt|tjr|j}|tj }dtvr?dttj D}|td<td|S)Nrz"Expected device type 'mtia', got 'rcLg|]!}tj|"Sr%)r"rr<rs rrz>MtiaInterface.Worker.get_device_properties..Orr)rrr"rrrrwrr@r:rrrrIrs rr<z*MtiaInterface.Worker.get_device_properties@rrrrr%rrr@r5rrr@rcBtj}|Sr)r"rrK)rets rrKzMtiaInterface.is_availablegsj%%'' rNrcDtj|}|Sr)r"rrrccs rr`z$MtiaInterface.get_compute_capabilityls Z - -f 5 5 rcdSNTr%r6s rrmzMtiaInterface.is_triton_capableqtrcHddl}d|jjvrtddS)Nrrz(triton not built with the 'mtia' backendrrrorrs rrpz)MtiaInterface.raise_if_triton_unavailableus5 1 1 1IJJ J 2 1rr)#rr r!r"rrr-r2r@r>r:r7rIrLrPrRrWr]r<rr[rrGrrErkrdrrrKr#r$rr`rmrpr%rrrr/sU Z F J E Z F D D D D D D D DD"\%*";<??O(L)JKK#|EJ$?@@$ UZ%ABB$\u{'9S\%+"4\KKEK,>K$KKK\KKKrrget_xpu_stream)_xpu_getCurrentRawStreamc^eZdZejjZejjZejjZGddZe ejj Z e ejj Z e ejj Z e ejj Z e ejjZe ejjZe ejjZe ejjZe ejjZe eZe ejjZe ejjZe ejjZe defdZe d dejjdefdZe dd edefd Z e d dejjdefd Z!e d dejjddfd Z"dS) XpuInterfaceceZdZededdfdZedefdZeddejj de fdZ dS)XpuInterface.WorkerrrNc|td<dSNxpurr6s rr7zXpuInterface.Worker.set_devices4: *5 1 1 1rcjdtvr tdStjSr)rr"rr:r%rrr:z"XpuInterface.Worker.current_devices-6665e<<9++-- -rc|mt|tr7tj|}|jdkrt d|jdt|tjr|j}|tj }dtvr?dttj D}|td<td|S)Nrz!Expected device type 'xpu', got 'rcLg|]!}tj|"Sr%)r"rr<rs rrz=XpuInterface.Worker.get_device_properties..s8I33A66r)rrr"rrrrwrr@r:rrrrIrs rr<z)XpuInterface.Worker.get_device_propertiess!fc**"\&11F{e++,N NNNfel33*#\F~%,;;==<<<"59#9#9#;#;<< ;F073E:6B Brrrr%rrr@rs  ;s ;t ; ; ;  ;  . . . .  .  C C%+*< C C C C  C C Crr@rc>tjSr)r"rrKr%rrrKzXpuInterface.is_availablesy%%'''rNrcDtj|}|Sr)r"rrrs rr`z#XpuInterface.get_compute_capabilitys Y , ,V 4 4 rFrac>tjSr)r"rrdrcs rrdzXpuInterface.is_bf16_supportedsy**,,,rcdSrr%r6s rrmzXpuInterface.is_triton_capablerrcHddl}d|jjvrtddS)Nrintelz)triton not built with the 'intel' backendrrs rrpz(XpuInterface.raise_if_triton_unavailables5 &/2 2 2JKK K 3 2rrrq)#rr r!r"rrr-r2r@r>r:r7rIrLrPrRrWr]r<rr[rrGrrErkrrrKr#r$rr`rdrmrpr%rrrrsr Y F IOE Y F C C C C C C C CD"\%)":;;Nei233J< 677L \%)* + +F!\%)":;;Nei233J$ UY%@AA,uy455K(L)HII!\.11N"l59#=>>O(L)IJJ#|EI$>??($(((\(u{'9S\--t----\-%+"4\LLEK,>L$LLL\LLLrrceZdZUeed<dS)CpuDevicePropertiesmulti_processor_countN)rr r!r?__annotations__r%rrrrsrrceZdZGddejZGddZedefdZeddedefd Z edd ej j de fd Z ed edefdZedefdZedd ej j dd fdZedd ej j defdZedd ej j dd fdZd S) CpuInterfacecBeZdZd deddfdZdedefdZd deddfd ZdS) CpuInterface.EventT enable_timingrNcd|_dS)Ngtime)r|rs rr}zCpuInterface.Event.__init__s DIIIrotherc&|j|jz dzS)Nir)r|rs r elapsed_timezCpuInterface.Event.elapsed_timesJ*d2 2rrLc6tj|_dSr)r perf_counter)r|rLs rrecordzCpuInterface.Event.records)++DIIIr)Tr) rr r!rrr}rfloatrrr%rrr-rs  $ $     3c 3e 3 3 3 3 , , , , , , , , ,rr-cFeZdZe ddejjdefdZdS)CpuInterface.WorkerNrrcPddl}|}t|SNr)multiprocessing cpu_countr)rrrs rr<z)CpuInterface.Worker.get_device_propertiess/ # " " "'1133I&y11 1rr) rr r!r>r"r#r$rr<r%rrr@rsO )- 2 2K& 2  2 2 2  2 2 2rr@rcdSrr%r%rrrKzCpuInterface.is_availablerrFracdSrr%rcs rrdzCpuInterface.is_bf16_supportedrrNrcdSNr%r6s rr`z#CpuInterface.get_compute_capabilityrrrXcdSrr%rZs rr[zCpuInterface.get_raw_streamqrcdSrr%r%rrr:zCpuInterface.current_devicerrcdSrr%r6s rr]zCpuInterface.synchronizes rcdSrr%r6s rrmzCpuInterface.is_triton_capablerrcHddl}d|jjvrtddS)Nrcpuz'triton not built with the 'cpu' backendrrs rrpz(CpuInterface.raise_if_triton_unavailable s5 0 0 0HII I 1 0rrqr)rr r!r"r-r@r>rrrKrdr#r$rr`rr?r[r:r]rmrpr%rrrrs,,,,, ,,,22222222$\t\u{'9S\33\C\  EK. $   \ %+"4\JJEK,>J$JJJ\JJJrrc eZdZeddedefdZe ddejdedefdZ edefdZ ede fdZ edd ej jdefd Zedd ej jdd fd ZGd dZd S) MpsInterfaceFrarcdSrr%rcs rrdzMpsInterface.is_bf16_supportedrrrec>|tjtjfvrdSdS)NFT)r"float64 complex128rhs rrizMpsInterface.is_dtype_supporteds# U]E$45 5 55trcHtjjSr)r"rmpsrKr%rrrKzMpsInterface.is_available!s~!..000rcdSrr%r%rrr:zMpsInterface.current_device%rrNrcdSrr%r6s rr`z#MpsInterface.get_compute_capability)rrcBtjdSr)r"rr]r6s rr]zMpsInterface.synchronize-s rc`eZdZeddejjdefdZede fdZ dS)MpsInterface.WorkerNrrcxtddgtjjS)N MPSPropertiesr)rr"rrget_core_countr6s rr<z)MpsInterface.Worker.get_device_properties3s8I:o0G/HII"1133 rcdSrr%r%rrr:z"MpsInterface.Worker.current_device91rr rr r!r>r"r#r$rr<r?r:r%rrr@r2q   %+*<               rr@rqr)rr r!r>rrrdrsr"rerirKr?r:r#r$rr`r]r@r%rrr r smt\=BK6: [1$111\1C\u{'9S\  EK. $   \           rr ceZdZeddedefdZe ddejdedefdZ edefdZ ede fdZ edd ej jdefd ZGd d Zd S) TpuInterfaceFrarcdSrr%rcs rrdzTpuInterface.is_bf16_supported?rrrecx|tjtjtjtjtjfvSr)r"r complex32 complex64rhalfrhs rrizTpuInterface.is_dtype_supportedCs0 M O O   J    rctSrr r%rrrKzTpuInterface.is_availableOs rcdSrr%r%rrr:zTpuInterface.current_deviceSrrNrcdSrr%r6s rr`z#TpuInterface.get_compute_capabilityWrrc`eZdZeddejjdefdZede fdZ dS)TpuInterface.WorkerNrrc6tddgdS)N TPUPropertiesrrrr6s rr<z)TpuInterface.Worker.get_device_properties]s'I:o0G/HII rcdSrr%r%rrr:z"TpuInterface.Worker.current_devicecrrrrr%rrr@r+\rrr@rqr)rr r!r>rrrdrsr"rerirKr?r:r#r$rr`r@r%rrr!r!>s5t\=B   K  6:      [  $\C\u{'9S\          rr!device_interfacesFrrvrc\t|tjr|j}|t|<dSr)rr"rrr/)rrvs rregister_interface_for_devicer1ls0&%,'' 0frct|tjr|j}tst |t vr t |Std|)NzNo interface for device )rr"rr_device_initializedinit_device_regr/rr6s rget_interface_for_devicer5ts_&%,''  """ (( AAA B BBrc^tsttSr)r3r4r/itemsr%rr get_registered_device_interfacesr8~s(   " " $ $$rctdtttjD]}td|ttdt ttjD]}td|t tdtttj D]}td|ttdttdttd td a dS) Nrzcuda:rzxpu:rzmtia:r rtpuT)r1rrr"rrIrrrrrr r!r3)rs rr4r4s.!&-888 5:**,, - -BB%kakk=AAAA!%666 59))++ , ,@@%jQjj,????!&-888 5:**,, - -BB%kakk=AAAA!%666!%666!%666rr)3r=rr collectionsrcollections.abcrr dataclassesrtypingrr r"torch.utils._pallasr r?rr _is_compiledtorch._Cr r rdictrrrrurrrrrrrrrrrr r!r/rr3rr1r5tupler8r4r%rrrDse" """"""........!!!!!! ------3%*%,,,,:EEEEEEEO46 $sCx.55513S#X333CRCRCRCRCRCRCRCRL6\M\M\M\M\MO\M\M\M~3%*%,,,,:EEEEEEEOKKKKKKKKKKOKKKKKK\#$t++++9CCCCCCCNMLMLMLMLML?MLMLML`  8J8J8J8J8J?8J8J8Jv'''''?'''T'''''?'''T794T/2238881 %, 1262G1 1111CS5<%7CDS9S3T*U%%%% r