tjVddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl mZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZmZmZmZddlmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5ddl6m7Z7m8Z8m9Z9m:Z:m;Z;mm?Z?m@Z@mAZAmBZBe/jCeDZEdZFe-deGdde5ZGe-deGddeZHGdd e=ZIGd!d"ejJZKGd#d$ejJZLGd%d&ejJZMGd'd(e%d)*ZNe-Gd+d,eZOGd-d.e%d)*ZPGd/d0e'd)*ZQe-Gd1d2e(ZRGd3d4e:ZSGd5d6ejJZTGd7d8ejJZUGd9d:ejJZVGd;de8ZXe-Gd?d@e#ZYGdAdBe<ZZGdCdDe;Z[GdEdFeYe9Z\GdGdHe?Z]GdIdJeBZ^GdKdLeAZ_e-dMNGdOdPe@Z`gdQZadS)RN)strict)nn) functional)initialization)ACT2FN)Cache)PreTrainedConfig)TorchvisionBackend) BatchFeatureget_patch_output_sizeselect_best_resolution)divide_to_patches)ChannelDimensionPILImageResamplingSizeDictget_image_size)FlashAttentionKwargs)BaseModelOutputWithPooling)PreTrainedModel) ImagesKwargsMultiModalDataProcessingKwargsProcessorMixinUnpack) TensorTypeTransformersKwargsauto_docstringcan_return_tuplelogging)CONFIG_MAPPING AutoConfig AutoTokenizer) LlamaConfig)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModelLlamaPreTrainedModel LlamaRMSNorm)LlavaCausalLMOutputWithPastLlavaForConditionalGeneration LlavaModelLlavaModelOutputWithPastc|jd}|jd}tj|||j|j}tj|d}tjdtj|j}tj||f}t|jdD]A}||} ||dz} || | } tj | ||} | || | <B|S)a* Compute the matrix multiplication (GEMM) for each expert sequentially. This approach is computationally inefficient, especially when dealing with a large number of experts. Args: token_states (torch.Tensor): Input tensor of shape (num_tokens, in_features). expert_weights (torch.Tensor): Weight tensor of shape (num_experts, in_features, out_features). tokens_per_expert (torch.Tensor): Number of tokens assigned to each expert. Returns: torch.Tensor: Output tensor of shape (num_tokens, out_features). rdtypedevicedim) shapetorchzerosr4r5cumsumlongcatrangematmul) token_statesexpert_weightstokens_per_expert num_tokens out_featuresoutputcumsum_num_tokens zero_tensor expert_numstartendtokensouts c/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/aria/modular_aria.pysequential_experts_gemmrOCs#A&J!'+L [\9KT`Tg h h hF %6A>>>+auz:K:RSSSK ;0A"BCCN0344  !*- Q/eCi(l6>*#=>>uSy Mzrhymes-ai/Aria) checkpointc|eZdZUdZdZdZddddddddZdZee d<d Z ee d <d Z ee d <d Z ee d <d Z edze d<dS)AriaTextConfigaA moe_num_experts (`int`, *optional*, defaults to 8): The number of experts in the MoE layer. moe_topk (`int`, *optional*, defaults to 2): The number of top experts to route to for each token. moe_num_shared_experts (`int`, *optional*, defaults to 2): The number of shared experts. aria_text text_configcolwiserowwise)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projz%layers.*.mlp.shared_experts.gate_projz#layers.*.mlp.shared_experts.up_projz%layers.*.mlp.shared_experts.down_projiintermediate_sizemoe_num_expertsr!moe_topkmoe_num_shared_expertsN pad_token_id)__name__ __module__ __qualname____doc__ model_typebase_config_keybase_model_tp_planrXint__annotations__rZr[r\r]rPrNrSrSbsJ#O%.%.%.%.1:/81:"s!!!OSHc"#C### L#*     rPrSceZdZUdZdZddiZeedZdZ e e zdze d<dZ e ezdze d<d Zeeeze d <dZe dze d <d Zee d<d Zee d<dZee d<fdZxZS) AriaConfigzq projector_patch_to_query_dict (`dict`, *optional*): Mapping of patch sizes to query dimensions. ariaimage_token_idimage_token_index)rU vision_configNrmrUr2vision_feature_layerprojector_patch_to_query_dict g{Gz?initializer_rangeFtie_word_embeddingsc |j ddd|_d|jD|_t|j|_t |jtr2d|jd<t|jddi|j|_n |jtd|_t |j tr d|j vrtdi|j |_ n|j t|_ tj di|dS)N)ii$cNi|]"\}}t|t|#Srgre.0kvs rN z,AriaConfig.__post_init__..s*-t-t-tAc!ffc!ff-t-t-trPidefics3_visionrbrg) roitemsmaxvalues'max_value_projector_patch_to_query_dict isinstancermdictr"rUrSsuper __post_init__)selfkwargs __class__s rNrzAriaConfig.__post_init__sL  - 522D ..u-tIkIqIqIsIs-t-t-t*7:4;];d;d;f;f7g7g4 d($ / / E/@D | ,!/0B<0P!Q!g!gTXTf!g!gD     '!/0A!B!D!DD  d& - - 0,$BR2R2R-AA0@AAD     %-//D '''''''rP)r^r_r`rarb attribute_maprSr# sub_configsrmrr rfrUrnrelistrorlrqfloatrrboolr __classcell__rs@rNriris J-M#1:NNK48M4**T188804K&-444,.#S /...15!4$;555s#u### %%%%(((((((((rPriceZdZdS)AriaTextRMSNormNr^r_r`rgrPrNrrDrPrc(eZdZdZfdZdZxZS)AriaProjectorMLPa! Feed-Forward Network module for the Aria Projector. Args: in_features (`int`): Input embedding dimension. hidden_features (`int`): Hidden dimension of the feed-forward network. output_dim (`int`): Output dimension. cttj||d|_tj||d|_t d|_dS)NFbiasgelu_new)r__init__rLinear linear_in linear_outract)r in_featureshidden_features output_dimrs rNrzAriaProjectorMLP.__init__sY ;eLLL)OZeLLL*%rPc|||}||}|SN)rrr)r hidden_statess rNforwardzAriaProjectorMLP.forwards6 !>!>??  66 rPr^r_r`rarrrrs@rNrrsQ  &&&&& rPrc6eZdZdZddedeffd Zd dZxZS) AriaCrossAttentionzv Aria Cross-Attention module. Args: config (`AriaConfig`): The configuration to use. rconfig dropout_ratec0t|jj}|jj}||_t j||d|_t j||d|_ t j||d|_ t j ||d|_ t j|||_ t j||_t j||_t j||_dS)NFrT) batch_first)rrrm hidden_sizenum_attention_heads num_headsrrq_projk_projv_projMultiheadAttentionmultihead_attnlinearDropoutdropout LayerNorm layer_norm layer_norm_kv)rrrrrrs rNrzAriaCrossAttention.__init__s *6 (< "i [uEEE i [uEEE i [uEEE !3KX\]]]i [99 z,// ,{33\+66rPNc\|||}||}||}||}|||||\}}|||}|S)a Forward pass of the AriaCrossAttention module. Args: key_value_states (`torch.Tensor`): Input tensor for key and value. hidden_states (`torch.Tensor`): Input tensor for query. attn_mask (`torch.Tensor`, *optional*, defaults to None): Attention mask. Returns: torch.Tensor: Output tensor after cross-attention.  attn_mask)rrrrrrrr) rkey_value_statesrrquerykeyvalue attn_output_s rNrzAriaCrossAttention.forwards  DOOM::;;--.>??kk*++ ,--,,UC),TT Qll4;;{#;#;<< rP)rr) r^r_r`rarirrrrrs@rNrrsn77z7777777"rPrcTeZdZdZdeffd ZddejdejdzfdZxZ S) AriaProjectora  Aria Projector module. This module projects vision features into the language model's embedding space, enabling interaction between vision and language components. Args: config (`AriaConfig`): Configuration object for the model. rc$t|j|_|jj|_|jj|_|jj|_ |j j|_ |j j|_ tjtj|j|j|_t'||_tj|j|_t/|j|j |j |_dSr)rrropatch_to_query_dictrmrrrrkv_dimrUrrr Parameterr:r;rrr cross_attnrrr feed_forwardrrrs rNrzAriaProjector.__init__s #)#G !/;-A*6 %1= ,8\%+f.\^b^n"o"opp ,V44,t'788,T-=t?SUYUdeerPNrrcb|jd|jd}}||jvr-td|d|jd|j|}|jd|d|dd}|X||jd}|d d| dd}| |||}| | |}|S) a Forward pass of the Projector module. Args: key_value_states (`torch.Tensor`): Input tensor of shape (batch_size, num_patches, kv_dim). attn_mask (`torch.Tensor`, *optional*, default is None): Attention mask. Returns: `torch.Tensor`: Output tensor of shape (batch_size, query_number, output_dim). rr8zNumber of patches z: not found in patch_to_query_dict amongst possible values .Nr2r)r9rKeyErrorkeysr unsqueezerepeatrepeat_interleaverexpandsizerrr) rrr batch_size num_patches query_numqueries attention_outrMs rNrzAriaProjector.forward#sR#3"8";=M=STU=VK d6 6 6O[OOlpmEmJmJmLmLOOO ,[9 *ZiZ(22155< >?? rPr) r^r_r`rarirr:Tensorrrrs@rNrrsfffffff( PTATrPrcTeZdZUdZeed<eed<eeeed<eed<dS)AriaImageProcessorKwargsa max_image_size (`int`, *optional*, defaults to `self.max_image_size`): Maximum image size. Must be either 490 or 980. min_image_size (`int`, *optional*, defaults to `self.min_image_size`): Minimum image size. Images smaller than this in any dimension will be scaled up. split_resolutions (`list[list[int]]`, *optional*, defaults to `self.split_resolutions`): A list of possible resolutions as (height, width) pairs for splitting high-resolution images into patches. split_image (`bool`, *optional*, defaults to `self.split_image`): Whether to split the image into patches using the best matching resolution from `split_resolutions`. max_image_sizemin_image_sizesplit_resolutions split_imageN)r^r_r`rarerfrrrgrPrNrrEsV  DI&&&rPrF)totalceZdZgdZeZejZgdZ gdZ dZ dZ dZ dZdZdZdZdeeffd Zd ed ed eefd Zddd eddd dfdZddd ed dfdZdddeeededdd edf dZ d'deddedededeeezdzdeeezdzdedzdeezdzded ed!eeedzd"eddd efd#Z d(d$ed%efd&Z!xZ"S))AriaImageProcessor pixel_values pixel_mask num_crops)?rrPFNTrc |dgd}d|D|d<tjdi|dS)Nr))r8r!)r8r)r8)r8)r8)r8)r8rY)r!r)r!r)r!r!)r!r8)rr8)rr!)rr8)rr!)rr8)rr8)rr8)rYr8c:g|]}|ddz|ddzgS)rr8rg)ryels rN z/AriaImageProcessor.__init__..js-*d*d*d"BqECKA+E*d*d*drPrg)getrr)rrdefault_resolutionsrs rNrzAriaImageProcessor.__init__gsh ::) * * 2#{#{#{ *d*dPc*d*d*dF& '""6"""""rPoriginal_resolutiontarget_resolutionreturnc|\}}|\}}t||z d\}}t||z d\} } || ||z| | zgS)zNGet padding size for patching, returns [left, top, right, bottom] for tvF.pad.r!)divmod) rrroriginal_heightoriginal_width target_height target_widthpaste_xr_xpaste_yr_ys rN_get_padding_sizez$AriaImageProcessor._get_padding_sizems]*='&7# |l^;Q?? mo=qAA 'C-3??rPimagez torch.Tensorresamplez7PILImageResampling | tvF.InterpolationMode | int | Nonect||tj\}}||t |||S)zFResize an image to a target resolution while maintaining aspect ratio.input_data_formatheightwidth)r rFIRSTresizer)rrrr new_height new_widths rN_resize_for_patchingz'AriaImageProcessor._resize_for_patchingusO!6 $8H8N! ! !  I{{5(*I"N"N"NPXYYYrPct||tj}|||}t j||S)zCPad an image to a target resolution while maintaining aspect ratio.r)padding)r rr rtvFpad)rrrnew_resolutionrs rN_pad_for_patchingz$AriaImageProcessor._pad_for_patchingsH /u6G[k[qrrr((9JKKwug....rPgrid_pinpoints patch_sizec t|tstdt|tj}t ||}||||}|||}t||} | S)a Process an image with variable resolutions by dividing it into patches. Args: image (`torch.Tensor`): The input image to be processed (channels-first format). grid_pinpoints (`list[list[int]]`): A list of possible resolutions as (height, width) pairs. patch_size (`int`): Size of each square patch to divide the image into. resample (`PILImageResampling | tvF.InterpolationMode | int | None`): Resampling filter to use when resizing. Returns: `list[torch.Tensor]`: A list of image patches in channels-first format. z6grid_pinpoints must be a list of possible resolutions.) channel_dim)r) rr TypeErrorrrr rrrr) rrrrr image_sizebest_resolution resized_image padded_imagepatchess rNget_image_patchesz$AriaImageProcessor.get_image_patchess..$// VTUU U#E7G7MNNN 0^LL11%(SS --m_MM #LZHHHrPimages do_rescalerescale_factor do_normalize image_mean image_stddisable_groupingreturn_tensorsrrrrc | dvrtdg}g}d}|D]d}| r||| | | }n|g}|t||krt|}|D]}|jd|jd}}| t ||z }||kr#t t ||z| }| }n"| }t t ||z| }||t||| }| |z }| |z }tj |dd||g}tj | | ftj }d|d|d|f<| || |ftj|d }|||||||}tj|d }t!|||d | S) Nrrz(max_image_size must be either 490 or 980r2r r)r4Tr6r)data tensor_type) ValueErrorr"lenr9rrerrrrr:r;rappendstackrescale_and_normalizer )rr#r$r%r&r'r(r)r*rrrrrr pixel_masksprocessed_cropsrr crop_images crop_imagehwscalenew_hnew_wpadding_bottom padding_rightrstacked_images stacked_maskss rN _preprocesszAriaImageProcessor._preprocesssA"  + +GHH H   3 3E &"44U*"":...&&z2222' 3*_!<<<33 J jR[   KQ777  .+& '     rPr r c8|d|j}|d|j}|d|j}t ||f|\}}|sdn.t j||z t j||z z} | S)a A utility that returns number of image patches for a given image size. Args: height (`int`): Height of the input image. width (`int`): Width of the input image. images_kwargs (`dict`, *optional*): Any kwargs to override defaults of the image processor. Returns: `int`: Number of patches per image. rrrr8)rrrrrmathceil) rr r  images_kwargsrrrresized_height resized_widthrs rNget_number_of_image_patchesz.AriaImageProcessor.get_number_of_image_patchess$'' t7GHH &**+;T=PQQ)--.A4CYZZ(>Pa(b(b%  hAA>N:;;di XfHf>g>gg  rP)rrNFNr)#r^r_r`model_input_namesr valid_kwargsrBICUBICrr'r(rrrrdo_convert_rgbr$r&rrtuplerrerrrr"rrstrrr rBrIrrs@rNrrWsCCC+L!)H JINNKNJL#(@!A###### @U@u@Y]^aYb@@@@ Z Z! ZL Z  Z Z Z Z//!/  ////T#Y  L  n  V"!48!NRC C ^$C C  C  C DK'$. C 4;&-C +C j(4/C C C  S ?T1C C LC !C C C C J#crPrc2eZdZUdZeed<eed<eed<dS)AriaImagesKwargsa split_image (`bool`, *optional*, defaults to `False`): Whether to split large images into multiple crops. When enabled, images exceeding the maximum size are divided into overlapping crops that are processed separately and then combined. This allows processing of very high-resolution images that exceed the model's input size limits. max_image_size (`int`, *optional*, defaults to `980`): Maximum image size (in pixels) for a single image crop. Images larger than this will be split into multiple crops when `split_image=True`, or resized if splitting is disabled. This parameter controls the maximum resolution of individual image patches processed by the model. min_image_size (`int`, *optional*): Minimum image size (in pixels) for a single image crop. Images smaller than this will be upscaled to meet the minimum requirement. If not specified, images are processed at their original size (subject to the maximum size constraint). rrrN)r^r_r`rarrfrergrPrNrQrQ sB  rPrQc<eZdZUeed<ddddddejdZdS)AriaProcessorKwargsrFF)rreturn_mm_token_type_idsr)rr) text_kwargsrFr*N)r^r_r`rQrfrPYTORCH _defaultsrgrPrNrSrS"sT####(-  "   %,  IIIrPrSc eZdZeZ d deezdedzdeee ze fdzffd Z dede defd Z d d Z e deefd ZxZS) AriaProcessorN tokenizer chat_templatesize_conversionc|ddd}d|D|_|j|_|j|_||j |j|_t |||dS)zx size_conversion (`Dict`, *optional*): A dictionary indicating size conversions for images. Nrtrur,c4i|]\}}t||Srgrwrxs rNr|z*AriaProcessor.__init__..Cs$NNNdaANNNrP)r[)r~r\ image_tokenrk pad_token unk_tokenrr)rimage_processorrZr[r\rs rNrzAriaProcessor.__init__6s  "$'c22ONNo6K6K6M6MNNN$0'6  Y%8%@"+"5I  )=QQQQQrP image_inputs image_idxrch|j|djd}|d|z}|j|zS)Nrr!r)r\r9r_)rrcrdtokens_per_imagenum_image_tokenss rNreplace_image_tokenz!AriaProcessor.replace_image_tokenLs>/ ^0L0RST0UV' 47GG"222rPc Bi}|tjdi|ddp jjfd|D}fd|D}|||dt di|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. NrFrc8g|]}jjg|RSrg)rbrI)ryrrFrs rNrz.bsE!!!A$@\*\m\\\!!!rPc0g|]}j|zSrg)r\)ryrmax_sizers rNrz.fs'rrrQ\ 4X > LrrrrP)rgnum_image_patchesrg)rSrWrupdaterbrr)r image_sizesr vision_datarmrgrFrls` @@rN_get_num_multimodal_tokensz(AriaProcessor._get_num_multimodal_tokensQs  "/9==orRRM   ( ( ($(()94@@gDDXDgH!!!!!"-!!!  srrrr`qrrr    4D[lmm n n n,, ,,,rPcdgS)Nrrg)rs rNunused_input_namesz AriaProcessor.unused_input_namesks }rP)NNNNr)r^r_r`rSvalid_processor_kwargsr$rOrrrerrhrqpropertyrrsrrs@rNrYrY2s0)-$(9= RR!3&RTz R eck3./$6 RRRRRR,3333333 ----4DIXrPrYc(eZdZdZdeffd ZxZS)AriaSharedExpertsMLPa/ Shared Expert MLP for shared experts. Unlike routed experts, shared experts process all tokens without routing. This class reconfigures the intermediate size in comparison to the LlamaMLP. Args: config (`AriaTextConfig`): Configuration object for the Aria language model. rcrt||j|jz|_dSr)rrrXr\rs rNrzAriaSharedExpertsMLP.__init__{s4    !'!9FVYZ>Z\b\rss)&*BFDVX^XnoorPctj||jjd\}}tj|d}||fS)Nr8)rzr7r2r6)r:topkrr[rrsoftmax)r router_logits top_logits top_indicesscoress rNroute_tokens_to_expertsz#AriaExperts.route_tokens_to_expertssH"'*]dk>RXY"Z"Z"Z K&&zr&::F""rPc||\}}|j}tj|tj|jjd|jjdz |}|}| d}tj |} | d| |jj z} | | |} tj| dd\} } tj| | z} || |}tj|jd|jj z|df|j|j}|d| || d|jj |d}||dzd}|S)Nrr8)binsminrr2r!r6r3)rr4r:histcflattentofloat32rrZviewargsort index_selectr[rchunkrrsilurr;r9rr5 index_copy_rsum)rrr top_k_index top_k_weightsoriginal_dtyperCindicesflatten_indicessorted_indicespermuted_tokens fc1_output projectiongate expert_outputunpermuted_tokensrFs rNrzAriaExperts.forwards%)%A%A-%P%P" ]$*!K    ! ! $ $U] 3 3, +a/    "^    !,,r**77'44Q$+J^8^__XXo/@AA  ;z1"=== D]'' 33d: ->?? !K   #dk&: :M.(s$ f f f !&) 4 4 f f frPF) rrr ModuleListr?num_hidden_layerslayersgradient_checkpointing post_initrs `rNrzAriaTextModel.__init__%sq    m f f f feFLdFeFe f f f   ',# rP)r^r_r`rSrrrs@rNrr$sD~rPrcFeZdZddiZdeffd ZefdZxZS)AriaTextForCausalLMlm_head.weightzmodel.embed_tokens.weightrct|t||_|j|_t j|j|jd|_| dSr) rrrr vocab_sizerrrlm_headrrs rNrzAriaTextForCausalLM.__init__1sj    "6**  +y!3V5FUSSS  rPc <tj|fi|dSr)rr)r super_kwargsrs rNrzAriaTextForCausalLM.forward:s'-- -----rP) r^r_r`_tied_weights_keysrSrrrrrs@rNrr.su*,GH~....^.....rPrceZdZdS)AriaCausalLMOutputWithPastNrrgrPrNrr?rrPrceZdZdS)AriaModelOutputWithPastNrrgrPrNrrCrrPrcVeZdZdeffd ZdZ ddejdejdzdee ezd e dzd e e d e ezf d Z dd ejdzdejdzdejdzdejdzdejdzdedzdejdzde dzd e ed e ezfdZxZS) AriaModelrcrt|t||_dSr)rrrmulti_modal_projectorrs rNrzAriaModel.__init__Hs1    %26%:%:"""rPc:|dS|d|jjj|jjj}|d|jjj|jjj}|ddkS)Nr8) dimensionrstepr!)r2r-r6r)unfold vision_towerrrrr)rrpatches_subgrids rN_create_patch_attention_maskz&AriaModel._create_patch_attention_maskLs  4$++")4")4,   *00")4")41    ###11A5;;===rPNr2rrrnoutput_hidden_statesrrc ||}|j|f|ddd|}d}|)|d} tj| }|j|} || ||_|S)NT)patch_attention_maskr return_dictr8r)rrrr: logical_notrr pooler_output) rrrrnrrr image_outputsimage_attn_maskflattened_maskselected_image_features rNget_image_featureszAriaModel.get_image_features\s $@@LL))  !5!%        +199!<!(C(C!44)%%)[%E 5   ,..}/C]EXYYN!%!:!:~";"" *889K^\\M%$% )%+'      '%77@JG33d!/)2>2JPT     rP)Nr2N)NNNNNNNN)r^r_r`rirrr: FloatTensorrerrrrrNrr LongTensorrr rrrrrs@rNrrGs;z;;;;;;>>>&0402,0 '%,"DIo  #Tk  +,  + +8.215.2.204(,26!%, , #d*, '$., $t+ ,  t+ , &- , , (4/, $;, -.,  ( (, , , , , , , , rPrz Aria model for conditional generation tasks. This model combines a vision tower, a multi-modal projector, and a language model to perform tasks that involve both image and text inputs. ) custom_introceZdZddiZe ddejdejdzdeeezde e d e e zf d Z ee dd ejdzdejdzdejdzd ejdzdejdzdedzdejdzdejdzdedzdeejzde e d e ezfdZ dfd ZxZS)AriaForConditionalGenerationrz(model.language_model.embed_tokens.weightNr2rrrnrrc .|jjd|||d|S)N)rrrnrg)rr)rrrrnrs rNrz/AriaForConditionalGeneration.get_image_featuress:-tz, %!!5      rPrrrrrrlabelsrlogits_to_keepc `|jd|||||||| d| } | d} t| trt| dn| }|| dd|ddf}d}| |jd|||jjjd| }t||| j | j | j S)a{ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or `model.image_token_id` (where `model` is your instance of `AriaForConditionalGeneration`). Tokens with indices set to `model.image_token_id` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> import httpx >>> from io import BytesIO >>> import torch >>> from PIL import Image >>> from io import BytesIO >>> from transformers import AutoProcessor, AutoModel >>> from transformers.image_utils import load_image >>> # Note that passing the image urls (instead of the actual pil images) to the processor is also possible >>> image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg") >>> image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg") >>> image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg") >>> processor = AutoProcessor.from_pretrained("Rhymes-AI/Aria") >>> model = AutoModel.from_pretrained("Rhymes-AI/Aria", dtype=torch.bfloat16, device_map="auto") >>> # Create inputs >>> messages = [ ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "In this image, we can see the city of New York, and more specifically the Statue of Liberty."}, ... {"type": "image"}, ... {"type": "text", "text": "What can we see in this image?"}, ... ] ... }, ... { ... "role": "user", ... "content": [ ... {"type": "image"}, ... {"type": "text", "text": "In which city is that bridge located?"}, ... ] ... } ... ] >>> prompts = [processor.apply_chat_template([message], add_generation_prompt=True) for message in messages] >>> images = [[image1, image2], [image3]] >>> inputs = processor(text=prompts, images=images, padding=True, return_tensors="pt").to(model.device) >>> # Generate >>> generated_ids = model.generate(**inputs, max_new_tokens=256) >>> generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True) >>> print(generated_texts[0]) Assistant: There are buildings, trees, lights, and water visible in this image. >>> print(generated_texts[1]) Assistant: The bridge is in San Francisco. ```)rrrrrrrrrN)logitsrr)lossrrrrrg) rrreslicer loss_functionrrUrrrrr)rrrrrrrrrrrrrr slice_indicesrrs rNrz$AriaForConditionalGeneration.forwards Z$*  %!)%+'        8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD*#3!/)     rPFc tj|f|||||d| } |s| dds || d<|| d<| S)N)rrrris_first_iterationrTrr)rprepare_inputs_for_generationr) rrrrrrrrr!r model_inputsrs rNr"z:AriaForConditionalGeneration.prepare_inputs_for_generation+s~=uww<  +'))1       4VZZ T%B%B 4 ,8L ()3L &rPr) NNNNNNNNNr)NNNNNNF)r^r_r`rrr:rrerrrrNrrrrrr rrrr"rrs@rNrrs+,VW0402   '  %,  "DIo  +,  + +    ^  .215.2.204(,26*.!%-.h h #d*h '$.h $t+ h  t+ h &- h h (4/h  4'h $;h el*h +,h  + +h h h ^h Z rPr) rirSrrYrrrrrr)brDr:huggingface_hub.dataclassesrrtorchvision.transforms.v2rrrr activationsr cache_utilsr configuration_utilsr image_processing_backendsr image_processing_utilsr r rimage_transformsr image_utilsrrrrmodeling_flash_attention_utilsrmodeling_outputsrmodeling_utilsrprocessing_utilsrrrrrutilsrrrrr autor"r#r$llama.configuration_llamar%llama.modeling_llamar&r'r(r)r*r+r,llava.modeling_llavar-r.r/r0 get_loggerr^loggerrOrSrirModulerrrrrrQrSrYrwrzrrrrrrrrrrrr__all__rgrPrNr;s ......777777&&&&&&!!!!!! 333333;;;;;;aaaaaaaaaa111111 CBBBBB::::::------ffffffffffffff=<<<<<<<<<333333  H % %>+,,,!!!!![!!-,!:+,,,*(*(*(*(*(!*(*(-,*(Z     l   ry244444444n>>>>>BI>>>B|5$rrrrr+rrrj|5*     *%     :::::N:::z Z Z Z Z Z8 Z Z Z ) ) ) ) ) RY) ) ) X)))))"))))X44444ry444"HHHHHHHH,,,,,,,,,$UUUUUoUUU. P P P P P. P P PJ.....13C..."     !<        6   [ [ [ [ [ [ [ [ |\\\\\#@\\\~   rP