tj1zdZddlZddlmZddlZddlmZddlmZddlm Z ddl m Z dd l mZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZddlmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'e"j(e)Z*Gddej+Z,Gddej+Z-Gddej+Z.GddeZ/e!GddeZ0e!Gdde0Z1e!d !Gd"d#e0eZ2e!d$!Gd%d&e0Z3gd'Z4dS)(zPyTorch OpenAI ImageGPT model.N)Any)nn)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPast)PreTrainedModel)Conv1D)auto_docstringlogging torch_float)maybe_autocast)ImageGPTConfigcZeZdZddeedeffd ZdejdejfdZ xZ S) ImageGPTLayerNormh㈵> hidden_sizeepsct||_tjt j||_dSN)super__init__rr ParametertorchTensorweight)selfrr __class__s l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/imagegpt/modeling_imagegpt.pyr#zImageGPTLayerNorm.__init__1s? l5< #<#<== tensorreturnc|tjtjtj|dd|jzz }||jz}|S)NT)axiskeepdim)r%sqrtmeansquarerr')r(r,s r*forwardzImageGPTLayerNorm.forward6sL%*UZ V0D0D2W[%\%\%\_c_g%ghhh$+% r+)r) __name__ __module__ __qualname__tupleintfloatr#r%r&r5 __classcell__r)s@r*rr0sz>>E#J>U>>>>>> elu|r+rceZdZddedzdedzffd ZddZddZdZd Z dd e j d e dzd e j dzd e j dzde j dzdedzdedzde fdZxZS)ImageGPTAttentionFNis_cross_attention layer_idxc t||_|j}|dt jt j||ft j dd||d|j |_ |j |_ |j |j z|_|j |_|j|j z|j kr t!d|j d|j d|j|_||_|j|_||_|j|_|jrBt-d |j z|j |_t-|j |j |_n"t-d |j z|j |_t-|j |j |_t5j|j|_t5j|j|_dS) NbiasdtyperF) persistentz=`embed_dim` must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r) r"r#configmax_position_embeddingsregister_bufferr%trilonesboolviewr embed_dimnum_attention_heads num_headshead_dim split_size ValueErrorscale_attn_weightsr@scale_attn_by_inverse_layer_idxrAreorder_and_upcast_attnrc_attnq_attnc_projrDropout attn_pdrop attn_dropout resid_pdrop resid_dropout)r(rHr@rA max_positionsr)s r*r#zImageGPTAttention.__init__>s  6    Juz=-"@ SSS T T Y Y1m]       +3$.8 . =4> )T^ ; ;'PTP^''N'''  #)";"406/U,"'-'E$  " E T^!3T^DDDK @@DKK T^!3T^DDDKT^T^<< Jv'899Z(:;;r+ctj||dd}|jr(|t |ddzz }|jr|t|jdzz }|j s|d|d}}|j dddd||z |d|f}tj |j j } tj| |j |j} tj||| }|||z}t#jd|}||j }||}tj||} | |fS)Nr/?rrEdevicedim)r%matmul transposerUrsizerVr;rAr@rCfinforEminr,rewhererSoftmaxtyper]) r(querykeyvalueattention_mask attn_weights query_length key_length causal_mask mask_value attn_outputs r*_attnzImageGPTAttention._attnfs|E3==R+@+@AA  " M'+ejjnn6K*L*LLL  / D'%0B*C*CCL& N',zz"~~sxx||*L)AAAqqq*|*Cj*PR]S]R]$]^K\%788'-Wa  !*&9:: 2'266t~FF %K+5+K((+5+J(('1$2DW..-  W4**  t %*% M22*1$.AF,3DNCJ M22![[88>>tTU>VV UhhsB FFPPQRTUVV 3DNDMJJTTUVXYZZ $ N ; ; A A$/WX A Y Y E3((3DNDMBBLLQPQRRCJJsB FFPPQRTUVVE  !-44S%PPJC! =8< %dn5 3GGQQRSUVWW  ' V(,(G(GsTY[i(j(j %K(, 5#un(U(U %K'' T^T]SS kk+.. ((55 L((r+)FNr!NNNNFF)r6r7r8rMr:r#rzrrrr%r&r r9r5r<r=s@r*r?r?=sS&<&<4$;&z*ImageGPTModel.__init__..s&lllq f B B Blllr+rF)r"r#rrOr Embedding vocab_sizewterIwper[ embd_pdropdrop ModuleListrangenum_hidden_layershrrln_fgradient_checkpointing post_initr(rHr)s `r*r#zImageGPTModel.__init__s    +< 14>BB< >OOJv011 llllERXRjLkLklllmm%dn&:STTT &+# r+c|jSr!r)r(s r*get_input_embeddingsz"ImageGPTModel.get_input_embeddingss xr+c||_dSr!r)r(new_embeddingss r*set_input_embeddingsz"ImageGPTModel.set_input_embeddingss !r+Nrpast_key_valuesrstoken_type_ids position_ids inputs_embedsrrrroutput_hidden_states return_dictrr-c | | n |jj} | | n |jj} | | n |jj} | | n |jj} ||t d|T||||}|d|d}|j d}n;|*|dd}|j d}nt d||j n|j }|j r%|j r| rtdd} ||d|d}| r|t|j}|L||nd}t#j|d| |z}|d}|||d}|||}||}|||j z}t/|jd drt1|j||| }nt3|j|| }|t3|j||| }|||}||z}||}||dfz}| rdnd}| r|jjrdnd}| rdnd}t9|jD]M\}}| r||fz}||||||| | }|d}| r$||dfz}|jjr ||dfz}N||}|j|}| r||fz}| st?d|||||fDStA|||||S)a; input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTModel >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTModel.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer/rz5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rH)re is_decoder)rHrrsr)rHrrs)rHrrsr)rrrrrGc3K|]}||V dSr!r )rvs r* z(ImageGPTModel.forward..=s0= ===r+)last_hidden_staterr attentionscross_attentions)!rHrrrrrT%warn_if_padding_and_no_attention_maskrjrNrrertraininglogger warning_oncer get_seq_lengthr%arange unsqueezerrtogetattrrr rr enumeraterrr9r)r(rrrsrrrrrrrrrr input_shape batch_sizerepast_seen_tokensposition_embedsrtoken_type_embeds output_shapeall_self_attentionsall_cross_attentionsall_hidden_statesrblockrs r*r5zImageGPTModel.forwards`2C1N--TXT_Tq$8$D $+Jj "+!6IIDKcdd d  "  6 6y. Q Q Q#..**K!r;r?;;I"+JJ  &',,..ss3K&,Q/JJTUU U%.%:!! @T  & "4= " "##p"  %+00[_EEN  ?0*$+>>>O  CRC^==???de  < BGGGJZZL'11!44L  %+00R@@N   HHY//M((<00%(:(:=;O(P(PP 4; e 4 4 /{+- / NN7{+-N " -%>{+5&; &&& "  % $ 8 8 ),==M -00 "m&8&8&<&<%>> $5?bb4%6d4;;Zdrr`d"6@BBD!$&)) P PHAu# I$58H$H!e%'=#"3G$AJM  P&9WQZM&I#;2P+?71:-+O( -00 * *L9   E 1]4D D  ':KM`bvw  9+++*1     r+) NNNNNNNNNNNN)r6r7r8rr#rrrr%r&r rMrr9rr5r<r=s@r*rrs~ """*.(,.2.2,0-1596:!%)-,0#'g g <$&g g  t+ g  t+ g lT) g |d*g  %|d2g !& t 3g $;g  $;g #Tkg D[g g  : :g g g ^g g g g g r+rz The ImageGPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc!@eZdZddiZdeffd Ze ddejdzde dzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzdejdzde dzde dzde dzde dzde de e zfdZxZS)ImageGPTForCausalImageModelingzlm_head.weightztransformer.wte.weightrHct|t||_t j|j|jdz d|_| dS)NrFrC) r"r#rrrLinearn_embdrlm_headrrs r*r#z'ImageGPTForCausalImageModeling.__init__Use    (00y0AA0EERRR  r+Nrrrsrrrrrlabelsrrrrrr-c R| | n |jj} |||||||||| | | |  }|d}||}d}| |dddddf}| dddf}t }||d|d|d}| s|f|ddz}||f|zn|St|||j |j |j |j S)a& input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForCausalImageModeling >>> import torch >>> import matplotlib.pyplot as plt >>> import numpy as np >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small") >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> model.to(device) # doctest: +IGNORE_RESULT >>> # unconditional generation of 8 images >>> batch_size = 4 >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1) # initialize with SOS token >>> context = context.to(device) >>> output = model.generate( ... input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40 ... ) >>> clusters = image_processor.clusters >>> height = image_processor.size["height"] >>> width = image_processor.size["width"] >>> samples = output[:, 1:].detach().cpu().numpy() >>> samples_img = [ ... np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [height, width, 3]).astype(np.uint8) for s in samples ... ] # convert color cluster tokens back to pixels >>> f, axes = plt.subplots(1, batch_size, dpi=300) >>> for img, ax in zip(samples_img, axes): # doctest: +IGNORE_RESULT ... ax.axis("off") ... ax.imshow(img) ```N) rrsrrrrrrrrrr.r/r)losslogitsrrrr) rHrrr+rrrNrjrrrrr)r(rrrsrrrrrr,rrrrrtransformer_outputsr lm_logitsr. shift_logits shift_labelsloss_fctoutputs r*r5z&ImageGPTForCausalImageModeling.forward]stJ&1%>L!#qrr'?5577L'))H8L--b,2C2CB2G2GHH,J[J[\^J_J_``D F\$7$;;F)-)9TGf$$v E0/?-;*50A     r+) NNNNNNNNNNNNN)r6r7r8_tied_weights_keysrr#rr%r&r rMrr9rr5r<r=s@r*r&r&Ls+,DE~*.(,.2.2,0-1596:&*!%)-,0#'l l <$&l l  t+ l  t+ l lT) l |d*l  %|d2l !& t 3l  t#l $;l  $;l #Tkl D[l l 2 2!l l l ^l l l l l r+r&z The ImageGPT Model transformer with an image classification head on top (linear layer). [`ImageGPTForImageClassification`] average-pools the hidden states in order to do the classification. c eZdZdeffd Ze ddejdzdedzdejdzdejdzdejdzd ejdzd ejdzd e dzd e dzd e dzde dzde de e zfdZ xZS)ImageGPTForImageClassificationrHct||j|_t||_t j|j|jd|_| dS)NFr() r"r# num_labelsrrrr)r*scorerrs r*r#z'ImageGPTForImageClassification.__init__si     +(00Yv}doEJJJ  r+Nrrrsrrrr,rrrrrr-c | | n |jj} ||||||||| | |  } | d}|d}||}d}|||||j}| s|f| ddz}||f|zn|St ||| j| j| j S)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForImageClassification >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForImageClassification.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits ```N) rrsrrrrrrrrrrf)r.r/rrr) rHrrr3r; loss_functionrrrr)r(rrrsrrrr,rrrrrr0rpooled_hidden_statesr/r.r5s r*r5z&ImageGPTForImageClassification.forwardsf&1%rQs%$  %%%%%%&&&&&&!!!!!!CCCCCCCCCC))))))JJJJJJJJ999999 .-----###### ,+++++222222  H % %        M)M)M)M)M) M)M)M)`")"E*E*E*E*E*.E*E*E*PoD     +   D x x x x x %