h$<dZddlZddlZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZmZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZmZmZddlm Z m!Z!m"Z"ddl#m$Z$e!jJe&Z'dZ(GddejRZ*GddejRZ+GddejRZ,GddeZ-e GddeZ.e Gdde.Z/e dGd d!e.eZ0e d"Gd#d$e.Z1gd%Z2y)&zPyTorch OpenAI ImageGPT model.N)AnyOptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions SequenceClassifierOutputWithPast)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_conv1d_layer)auto_docstringlogging torch_float)ImageGPTConfigc ddl}ddl}t j j|}tjd||jj|}g}g}|D]l\} } tjd| d| |jj|| } |j| |j| jnt||D]\} } | dd} | jd} t!d | Ds| d d vr4tjd j#dj%| j|} | d d vr t'| d} | D]W} |j)d| r|jd| }n| g}|ddk(s|ddk(r t'| d} n|ddk(r t'| d} n|ddk(s|ddk(rt'| |d} t'| d} n|ddvrt'| d} t'| d} nt+| dk(r,| ddk(r$|ddk(rt'| |d} t'| d} nQ|ddk(rt'| d} t'| d} n0|dd k(rt'| d} t'| d} nt'| |d} t+|d!k\sEt-|d}| |} Zt+| dkDr| ddk(s| d dk(s| d d k(s| d dk(rn | j.| j.k(sJ tjd"| | d d#k(rbt5j6| j9|j:|j:j<| j>ddd|j:f<| d d$k(rot5j6| j9|j:|j:j<| j>dd|j:d!|j:zf<!| d d%k(ret5j6| j9|j:|j:j<| j>ddd!|j:zdf<t+| dk(rP| ddk(rH| d!dk(r@t5j6| j9|j:|j:| _| d dk(rt5j6| | _| d dk(r7t5j6| | j>d|j@dz ddf<O| d d k(r$t5j6| | j>d <{t5j6| | _|S#t$rtj dwxYw#t0$r1}|xj2| j.| j.fz c_d}~wwxYw)&z0 Load tf checkpoints in a pytorch model rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape /c3$K|]}|dv yw))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns m/var/www/html/aiagenthome/venv/lib/python3.12/site-packages/transformers/models/imagegpt/modeling_imagegpt.py z.load_tf_weights_in_imagegpt..Ps  n n)_stepz Skipping {})wtet transformerz [A-Za-z]+\d+z(\d+)wgweightbbiaswpewte)q_projk_projv_projc_attnrrattnc_projr,lm_headsoszInitialize PyTorch weight r5r6r7)!re tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendsqueezezipsplitanyformatjoingetattr fullmatchlenintshapeAssertionErrorargstorch from_numpyreshapen_embdTdata vocab_size)modelconfigimagegpt_checkpoint_pathr>tftf_path init_varsnamesarraysnamerUarraypointerm_name scope_namesnumes r'load_tf_weights_in_imagegptrn/sh ggoo67G KK8 BC''0I E F e (l5'BC&&w5 T emmo& ! 5&) eABxzz#    "X " KK ,,SXXd^< =  88 #g}5GF||OV4 hhx8 %h 1~$ A#(=!'84Q3&!'62Q5(KNe,C!';q>:!'84Q#AA!'84!'84TaDGv$5+a.H:T!';q>:!'84Q6)!'95!'84Q5(!'51!'84!';q>:;1$+a.)!#,;> t9q=T!W.$r(f2DRTYHY]abd]ein]n  }} 333  078 8x /4/?/? fmm]c]j]j@k/l/n/nGLLOfmmO+ , "X !AFAQAQ fmmV]];Ba LLFMMA ,=== >"X !383C3CEMMRXR_R_agananDo3p3r3rGLLA -// 0 Y!^Q6 1d1g6I ++EMM&--,WXGL "X  ++E2GL "X 7<7G7G7NGLL06,,q00!3 4 "X $//6GLL  ++E2GLY*\ LC   Q   P" 7==%++66 s#U7)V7 V W#,WWcheZdZddeedeffd ZdejdejfdZ xZ S)ImageGPTLayerNorm hidden_sizeepsct|||_tjt j ||_yN)super__init__rrr ParameterrXTensorr0)selfrqrr __class__s r'rvzImageGPTLayerNorm.__init__s. ll5<< #<= tensorreturnc|tjtjtj|dd|jzz }||j z}|S)Nr*T)axiskeepdim)rXsqrtmeansquarerrr0)ryr|s r'forwardzImageGPTLayerNorm.forwardsK%**UZZ V0D2W[%\_c_g_g%ghh$++% r{)gh㈵>) __name__ __module__ __qualname__tuplerTfloatrvrXrxr __classcell__rzs@r'rprps5>E#J>U> ellu||r{rpcBeZdZddeedeeffd ZdZddZddZ dZ dZ dd e jd eed ee jd ee jd ee jdee jdeedeedee jdefdZxZS)ImageGPTAttentionis_cross_attention layer_idxc t||j}|jdt j t j ||ftjjdd||d|jdt jdd|j|_ |j|_ |j|jz|_|j|_|j|jz|jk7r&t!d|jd |jd |j"|_||_|j&|_||_|j*|_|j$rNt-d |jz|j|_t-|j|j|_n(t-d |jz|j|_t-|j|j|_t5j6|j8|_t5j6|j<|_tA|_!y) Nr2dtyperF) persistent masked_biasgz=`embed_dim` must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r=r)"rurvmax_position_embeddingsregister_bufferrXtrilonesboolviewr|rq embed_dimnum_attention_heads num_headshead_dim split_size ValueErrorscale_attn_weightsrscale_attn_by_inverse_layer_idxrreorder_and_upcast_attnrr8q_attnr:rDropout attn_pdrop attn_dropout resid_pdrop resid_dropoutset pruned_heads)ryr`rr max_positionsrzs r'rvzImageGPTAttention.__init__s 66    JJuzz=-"@ S T Y Y1m]    ]ELL,>5Q++33$..8 .. ==4>> )T^^ ;OPTP^P^O_`NN#2'  #)";";"406/U/U,"'-'E'E$  " " T^^!3T^^DDK @DK T^^!3T^^DDKT^^T^^< JJv'8'89ZZ(:(:;Er{cFt|dk(ryt||j|j|j\}}t j |||jz|d|jzzg}t|j|d|_ t|j|d|_ |j|jz|jt|z z|_|jt|z |_|jj||_y)Nrr=rdim) rSrrrrrXcatrrr8r:union)ryheadsindex index_attns r' prune_headszImageGPTAttention.prune_headss u:? 7t~~t}}^b^o^op uYYut'>T__I\@]^_ )jaH (eC  ??dnn<RUV[R\A\]#e*4 --33E:r{cDtj||jdd}|jr |t |j ddzz }|j r|t|jdzz }|js|j d|j d}}|jdddd||z |d|f} tj|jj} tj| |j|j} tj | || }|||z}t#j$d|}|j'|j}|j)|}|||z}tj||} | |fS)Nr*?rrdevicer)rXmatmul transposerrsizerrrrr2finforminr|rwhererSoftmaxtyper) ryquerykeyvalueattention_mask head_mask attn_weights query_length key_length causal_mask mask_value attn_outputs r'_attnzImageGPTAttention._attnst||E3==R+@A  " "'+ejjn6K*LLL  / /'%0B*CCL&&',zz"~sxx|*L))Aq*|*Cj*PR]S]R]$]^K\%7%78< &&',zz"~sxx|*L))Aq*|*Cj*PR]S]R]$]^K\%7%78< =s BJ99Kcx|jdd||fz}|j|}|jddddS)zJ Splits hidden_size dim into attn_head_size and num_heads Nr*rr=rr)rrpermuteryr|rattn_head_size new_shapes r' _split_headszImageGPTAttention._split_heads=sEKKM#2&)^)DD i(~~aAq))r{c|jddddj}|jdd||zfz}|j|S)zS Merges attn_head_size dim and num_attn_heads dim into hidden_size rr=rrNr)r contiguousrrrs r' _merge_headszImageGPTAttention._merge_headsEsO1a+668KKM#2&)n*D)FF {{9%%r{ hidden_states layer_pastrrencoder_hidden_statesencoder_attention_mask use_cacheoutput_attentionscache_positionr}c |du} |j\} } } |St|trA|jj |j }| r |j }n|j}n|}| r|n|}| r%t|ds td|[rY|j|}j|j j}|j|j j}nQ|j|}|j|j|j d\}}|j#| d|j$|j&j)dd}|j#| d|j$|j&j)dd}n|j|j|j d\}}}|j#| d|j$|j&j)dd}|j#| d|j$|j&j)dd}|D| s| nd} j+|||j d| i\}}| rd|j|j <|j#| | |j$|j&j)dd}|j,r|j/|||||\}}n|j1|||||\}}|j3||j$|j&}|j5|}|j7|}||fS) NrzIf class is used as cross attention, the weights `q_attn` have to be defined. Please make sure to instantiate class with `ImageGPTAttention(..., is_cross_attention=True)`.r=rr*rrT)rU isinstancer is_updatedgetrcross_attention_cacheself_attention_cachehasattrrrlayerskeysvaluesr8rMrrrrrupdaterrrrr:r)ryrrrrrrrrrrrseq_lenrrcurr_past_key_valuecurrent_statesrrrrrs r'rzImageGPTAttention.forwardMs3$>'--Wa  !*&9:'2266t~~F %*4*J*J'*4*I*I'&0#2D.- 4* t %* M2)00@EE+224>>BII M2![[8>>tTU>V UhhsB FPPQRTUV 3DNNDMMJTTUVXYZ $ N ; A A$//WX A Y E3((3DNNDMMBLLQPQRCJJsB FPPQRTUVE  !3E^4N,33CQacqPrsJC!8< %%dnn5 3GQQRSUVW  ' '(,(G(GsTY[ikt(u %K(, 5#unV_(` %K'' T^^T]]S kk+. ((5 L((r{)FN)NNNNNNNFFN)rrrrrrTrvrrrrrrXrxr rrrrs@r'rrs)"8D>)"V^_bVc)"V ;$)L2)h*&'+15,08<9=$),115D)||D)UOD)!. D) ELL) D) ( 5 D)!) 6D)D>D)$D>D)!.D) D)r{rcVeZdZfdZdej dej fdZxZS) ImageGPTMLPct||j}t|||_t|||_t |j|_tj|j|_ yrt) rurvrqrc_fcr:r activation_functionactrrrdropout)ryintermediate_sizer`rrzs r'rvzImageGPTMLP.__init__s_ && ,i8 Y(9: &445zz&"4"45 r{rr}c|j|}|j|}|j|}|j|}|Srt)rrr:r)ryrs r'rzImageGPTMLP.forwards@ -0 /  M2  ]3 r{)rrrrvrXrxrrrs@r'rrs#6U\\ellr{rc eZdZd fd Z ddej deedeej deej deej deej deed eed eej d e fd Z xZ S) ImageGPTBlockct||j}|j |jnd|z}t ||j |_t|||_t ||j |_ |jr/t|d||_ t ||j |_ t|||_y)NrrrT)rr)rurvrqn_innerrplayer_norm_epsilonln_1rr9ln_2add_cross_attentioncrossattention ln_cross_attnrmlp)ryr`rrq inner_dimrzs r'rvzImageGPTBlock.__init__s (( &,nn&@FNNa+o %kv7P7PQ %f B %kv7P7PQ  % %"3Ft_h"iD !2;FD]D]!^D y&1r{rrrrrrrrrr}c |} |j|}|j||||||| } | d} | dd} | | z}|Yt|dstd|d|} |j |}|j |||||||| }|d} | | z}| |ddz} |} |j |}|j|}| |z}|f| zS)N)rrrrrrrrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`)rrrrrrr)rr9rrrrrr)ryrrrrrrrrrresidual attn_outputsroutputscross_attn_outputsfeed_forward_hidden_statess r'rzImageGPTBlock.forwards:! -0 yy !)/)! #1o qr"#h. ,4!12 =dVDZZ%H ..}=M!%!4!4%-#&;'="3-"5 " -Q/K${2M 212 66G  -0 %)XXm%<" #== '))r{rtr) rrrrvrXrxrr rrrrrs@r'r r s 2$'+15,08<9=$),115:*||:*UO:*!. :* ELL) :* ( 5 :*!) 6:*D>:*$D>:*!.:* :*r{r cFeZdZUeed<eZdZdZdZ dgZ fdZ dZ xZ S)ImageGPTPreTrainedModelr`r- input_idsTr c$t||i|yrt)rurv)ryinputskwargsrzs r'rvz ImageGPTPreTrainedModel.__init__s &+F+r{c t|tjtfrl|jj j d|jj|j|jj jnt|tjry|jj j d|jj|jg|jj |jjn5t|tr%|jj jd|jD]m\}}d|vs d|vs|j j d|jjt!j"d|jj$zz oy)zInitialize the weights.g)rstdNrr:r0r=)rrLinearrr0r]normal_r`initializer_ranger2zero_ Embedding padding_idxrpfill_named_parametersmathrn_layer)rymodulergps r' _init_weightsz%ImageGPTPreTrainedModel._init_weightssJ fryy&1 2 MM   & &CT[[5R5R & S{{&   &&(  - MM   & &CT[[5R5R & S!!- ""6#5#56<<>  1 2 MM   $ $S )..0GD!4H$4Cdkk.K.KdiiXY\`\g\g\o\oXoNp.pr1r{)rrrr__annotations__rnload_tf_weightsbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrvr3rrs@r'r r s3 1O%!O&*#(),sr{r c$eZdZdeffd ZdZdZdZe dde e jde e de e jd e e jd e e jd e e jd e e jd e e jde e jde e de e de e de e de e jdedeeeff dZxZS) ImageGPTModelr`c vt|||j|_t j |j |j|_t j |j|j|_ t j|j|_ t jt|jDcgc]}t!||c}|_t%|j|j&|_d|_d|_d|_|j1ycc}w)NrrF)rurvrqrrr+r^r4rr3r embd_pdropdrop ModuleListrangenum_hidden_layersr hrprln_fmodel_parallel device_mapgradient_checkpointing post_init)ryr`irzs r'rvzImageGPTModel.__init__s  ++<< 1 14>>B<< > >OJJv001 ERXRjRjLklLkq f BLklm%dnn&:S:ST $&+#  ms D6c|jSrtr4)rys r'get_input_embeddingsz"ImageGPTModel.get_input_embeddings.s xxr{c||_yrtrJ)rynew_embeddingss r'set_input_embeddingsz"ImageGPTModel.set_input_embeddings1s !r{c|jD]-\}}|j|jj|/y)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrBr9r)ryheads_to_prunelayerrs r' _prune_headszImageGPTModel._prune_heads4s7+002LE5 FF5M   * *5 13r{r!past_key_valuesrtoken_type_ids position_idsr inputs_embedsrrrroutput_hidden_states return_dictrr$r}c | | n|jj} | | n|jj} | | n|jj} | | n|jj} | | t d|G|j |||j}|jd|d}|jd}n0|#|jdd}|jd}n t d| |jn |j}|jr%|jr| rtjdd} | r6|4tt!|jt!|j}| r:t#|t$r*tjd tj&|}||j)n|}||jd|d}|>t+j,||d|zt*j.| }|j1d}|z|dkr t d |j|d}|ddddddf}|j3|j4 }d |z t+j6|j4j8z}|jj:rE|C|j\}}}||f}| t+j<||} |j?| } nd} |jA||jjB}||jE|}|jG|}||j3|jz}||jE|}||z}|jI|}||jdfz}| rdnd}| r|jj:rdnd}| rdnd}tK|jLD]L\}} |jNr{t*jPjS|j||j3|j}t#|t*jTr|j3|j}| r||fz}| ||||||| | | | }!|!d}| r(||!dfz}|jj:r ||!dfz}|jNs|jVjYD]J\}"}#||#dk(sdt[|"z|j\k7s+|j3dt[|"dzz}LO|j_|}|j|}| r||fz}| st%d|||||fDSta|||||S)a input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTModel >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTModel.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer*rz5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r`zPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.rz$batch_size has to be defined and > 0rr)rr$)rrrrrr=zcuda:c3$K|]}|| ywrtr$)r%vs r'r(z(ImageGPTModel.forward..swA=wr))last_hidden_staterTr attentionscross_attentions)1r`rrXruse_return_dictr%warn_if_padding_and_no_attention_maskrrrUrrFtrainingrA warning_oncer r rrfrom_legacy_cacheget_seq_lengthrXarangelong unsqueezetorrrrrinvert_attention_mask get_head_maskr0r4r3r> enumeraterBrDcuda set_devicerxrErPstr last_devicerCr)$ryr!rTrrUrVrrWrrrrrXrYrr$ input_shape batch_sizer past_lengthencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeposition_embedsrtoken_type_embeds output_shapeall_self_attentionsall_cross_attentionsall_hidden_statesrHblockrrr\s$ r'rzImageGPTModel.forward;s`2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKcd d  "  6 6y. Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U%.%:!! @T@T  & &4==##p" 01,dkk2RT`hlhshsTtuO OU;   \  2CCOTO:I:Uo446[j  %+00[_EN   << [_{5RZ_ZdZdmstL'11!4L  %Q !GHH+00R@N ,AtT1,<=N,..TZZ.@N!N2ekk$**6M6Q6QQN ;; * */D/P=R=W=W=Y :  7$68O#P %-).4HQW)X&%)%?%?@V%W "%) " &&y$++2E2EF   HHY/M((<0%(:(:=;O;O(PP  % $ 8 ),==M -0 "m&8&8&<%>> $5b4%64;;;Z;Zr`d"6BD!$&&)HAu"" %%m&:&:;!-%3%6%6}7K7K%LNi6 ) ]-A-A BI#$58H$H!! %'=#"3- G$AJM &9WQZM&I#;;22+?71:-+O("" OO113DAqAbEzgA&6$:J:J&J(5(8(83q1u:9M(N 4A*H -0 * **L9  1]4D D ':KM`bvw  9+++*1   r{)NNNNNNNNNNNNNN)rrrrrvrKrNrSrrrXrxr rrrrrrrrs@r'r;r;s~~&"2-1+/1515/3,0048<9=$(,0/3&*15K ELL)K "%K !. K !. K u||, K ELL)K   -K  ( 5K !) 6K D>K $D>K 'tnK d^K !.K !K " u?? @#K K r{r;z The ImageGPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc&eZdZdgZdeffd Ze ddeejdee deejdeejdeejd eejd eejd eejd eejd eejdee dee dee dee deejde de eeff"dZxZS)ImageGPTForCausalImageModelingzlm_head.weightr`ct||t||_t j |j |jdz d|_d|_ d|_ |jy)NrFr2) rurvr;r-rr'r[r^r;rDrErGryr`rzs r'rvz'ImageGPTForCausalImageModeling.__init__s[  (0yy0A0AA0EER $ r{r!rTrrUrVrrWrrlabelsrrrXrYrr$r}c (||n|jj}|j||||||||| | | | ||}|d}|j|}d}| r|dddddfj }| dddfj }t }||j d|jd|j d}|s|f|ddz}||f|zS|St|||j|j|j|jS)a& input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForCausalImageModeling >>> import torch >>> import matplotlib.pyplot as plt >>> import numpy as np >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForCausalImageModeling.from_pretrained("openai/imagegpt-small") >>> device = torch.device("cuda" if torch.cuda.is_available() else "cpu") >>> model.to(device) # doctest: +IGNORE_RESULT >>> # unconditional generation of 8 images >>> batch_size = 4 >>> context = torch.full((batch_size, 1), model.config.vocab_size - 1) # initialize with SOS token >>> context = context.to(device) >>> output = model.generate( ... input_ids=context, max_length=model.config.n_positions + 1, temperature=1.0, do_sample=True, top_k=40 ... ) >>> clusters = image_processor.clusters >>> height = image_processor.size["height"] >>> width = image_processor.size["width"] >>> samples = output[:, 1:].detach().cpu().numpy() >>> samples_img = [ ... np.reshape(np.rint(127.5 * (clusters[s] + 1.0)), [height, width, 3]).astype(np.uint8) for s in samples ... ] # convert color cluster tokens back to pixels >>> f, axes = plt.subplots(1, batch_size, dpi=300) >>> for img, ax in zip(samples_img, axes): # doctest: +IGNORE_RESULT ... ax.axis("off") ... ax.imshow(img) ```N) rTrrUrVrrWrrrrrXrYrr.r*r)losslogitsrTrr^r_) r`r`r-r;rrrrrrTrr^r_)ryr!rTrrUrVrrWrrrrrrXrYrr$transformer_outputsr lm_logitsr shift_logits shift_labelsloss_fctoutputs r'rz&ImageGPTForCausalImageModeling.forwardsRN&1%L!#qr'?557L')HL--b,2C2CB2GH,J[J[\^J_`D\$7$;;F)-)9TGf$ Ev E0/??-;;*550AA   r{)NNNNNNNNNNNNNNN)rrr_tied_weights_keysrrvrrrXrxr rrrrrrrrs@r'rr s++ ~ -1+/1515/3,0048<9=)-$(,0/3&*15!p ELL)p "%p !. p !. p u||, p ELL)p   -p  ( 5p !) 6p &p D>p $D>p 'tnp d^p !.!p "#p $ u77 8%p p r{rz The ImageGPT Model transformer with an image classification head on top (linear layer). [`ImageGPTForImageClassification`] average-pools the hidden states in order to do the classification. c feZdZdeffd Ze ddeejdee deejdeejdeejdeejd eejd eejd ee d ee d ee dee de de e effdZxZS)ImageGPTForImageClassificationr`ct|||j|_t||_t j |j|jd|_|jy)NFr) rurv num_labelsr;r-rr'r[scorerGrs r'rvz'ImageGPTForImageClassification.__init__sR   ++(0YYv}}dooEJ  r{r!rTrrUrVrrWrrrrXrYr$r}c | | n|jj} |j|||||||| | | |  }|d}|jd}|j |}d}||j |||j}| s|f|ddz}||f|zS|St |||j|j|jS)ay input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoImageProcessor`]. See [`ImageGPTImageProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoImageProcessor, ImageGPTForImageClassification >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> image_processor = AutoImageProcessor.from_pretrained("openai/imagegpt-small") >>> model = ImageGPTForImageClassification.from_pretrained("openai/imagegpt-small") >>> inputs = image_processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits ```N) rTrrUrVrrWrrrXrYrrr)rrrTrr^) r`r`r-rr loss_functionrrTrr^)ryr!rTrrUrVrrWrrrrXrYr$rrpooled_hidden_statesrrrs r'rz&ImageGPTForImageClassification.forwardsd&1%T $D>T 'tnT d^T T  u66 7T T r{r)rrr;r rn)3__doc__r/rCtypingrrrrXrtorch.nnr activationsr cache_utilsr r r generationr modeling_layersrmodeling_outputsrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_imagegptr get_loggerrrArnModulerprrr r r;rr__all__r$r{r'rsL% '' %!CC)9 .YY 3   H %iX  i) i)X"))"J*.J*Z#so#s#sLm +m m `  %