tj|dZddlmZddlmZddlZddlmZddlmZm Z m Z ddl m Z dd lmZdd lmZdd lmZmZmZmZmZmZmZdd lmZmZdd lmZddlm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e%j-e.Z/Gddej0Z1 dCdej0dej2dej2dej2dej2dzde3dzde3dee#fdZ4Gd d!ej0Z5Gd"d#ej0Z6Gd$d%ej0Z7Gd&d'ej0Z8e$Gd(d)eZ9e$d*+eGd,d-e"Z:e$Gd.d/e9Z;e$d0+Gd1d2e9Z<Gd3d4ej0Z=Gd5d6ej0Z>e$Gd7d8e9Z?e$d9+Gd:d;e9Z@e$Gd<d=e9ZAe$Gd>d?e9ZBe$Gd@dAe9ZCgdBZDdS)DzPyTorch ALBERT model.)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) AlbertConfigc eZdZdZdeffd Z d dejdzdejdzdejdzdejdzd ej f d Z xZ S) AlbertEmbeddingszQ Construct the embeddings from word, position and token_type embeddings. configcttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_|dt%j|jdd|dt%j|jt$jddS) N) padding_idxeps position_idsrF) persistenttoken_type_ids)dtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr(sizelongselfr# __class__s h/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/albert/modeling_albert.pyr/zAlbertEmbeddings.__init__6s5 !|F,=v?Tbhbuvvv#%<0NPVPe#f#f %'\&2H&J_%`%`"f&;AVWWWz&"<==   EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg      N input_idsr,r( inputs_embedsreturnc||}n|dd}|\}}||jddd|f}|t|drT|j|jdd}t j|d|}|||}n+t j|t j |jj }|| |}| |} || z} | |} | | z} || } || } | S)Nr*r,rr)dimindex)r-device)rCr(hasattrr,rAshaper?gatherrBrDrPr4r8r6r9r=) rFrJr,r(rK input_shape batch_size seq_lengthbuffered_token_type_idsr8 embeddingsr6s rHforwardzAlbertEmbeddings.forwardGsl  #..**KK',,..ss3K!, J  ,QQQ  ^ J J"%:: "66|DD"55 ^^J// \\*-- rI)NNNN) __name__ __module__ __qualname____doc__r r/r? LongTensor FloatTensorTensorrY __classcell__rGs@rHr"r"1s |      &.2260426 ''#d*'(4/'&- ' (4/ '  ''''''''rIr"modulequerykeyvalueattention_maskscalingr=kwargsc ||ddz}tj||dd|z}|||z}tj|d}tj|||j}tj||} | dd } | |fS)Nr*r rN)ptrainingr) rCr?matmul transposer functionalsoftmaxr=rp contiguous) rdrerfrgrhrir=rj attn_weights attn_outputs rHeager_attention_forwardrxrs**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$rIc eZdZdeffd Z d dejdejdzdee de ejejffdZ xZ S) AlbertAttentionr#cht|j|jzdkr/t |dst d|jd|j||_|j|_|j|_|j|jz|_|j|jz|_|jdz|_ tj |j |_ tj |j|_tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_d|_dS)Nrr2zThe hidden size (z6) is not a multiple of the number of attention heads (rlr&F)r.r/ hidden_sizenum_attention_headsrQ ValueErrorr#attention_head_size all_head_sizerirr;attention_probs_dropout_probattention_dropoutr<output_dropoutLinearrerfrgdenser9r: is_causalrEs rHr/zAlbertAttention.__init__s    : :a ? ?PVXhHiHi ?7F$677 477  #)#= !-#)#59S#S !58PP/5 !#F,O!P!P j)CDDYv143EFF 9V/1CDDYv143EFF Yv163EFF f&8f>STTTrIN hidden_statesrhrjrLc |jdd}g|d|jR}||j|dd}||j|dd}||j|dd}tj|j j t} | |||||f|j sdn |j j|jd|\} } | jg|dR} || } || } ||| z} | | fS)Nr*rrmrc)r=ri)rRrreviewrrrfrgr get_interfacer#_attn_implementationrxrprrorireshaperurrr9) rFrrhrjrT hidden_shape query_layer key_layer value_layerattention_interfacerwrvs rHrYzAlbertAttention.forwards $)#2#. CCbC$*BCC 5djj//4lCMMaQRSS 0DHH]++0,?II!QOO 4djj//4lCMMaQRSS (?(M K ,.E) ) %8$7      % $}JCC$2H2JL % %  % % ! \*k);;;;;;FFHH jj-- ))+66 nn][%@AA L((rIN rZr[r\r r/r?r`r_rrtuplerYrarbs@rHrzrzs|<48")")|"))D0")+, ") u|U\) * ")")")")")")")")rIrzc eZdZdeffd Z d dejdejdzdee de ejejffdZ d ejdejfd Z xZ S) AlbertLayerr#ct||_|j|_d|_t j|j|j|_ t||_ t j |j|j |_t j |j |j|_t |j|_t j|j|_dS)Nrr&)r.r/r#chunk_size_feed_forward seq_len_dimrr9r|r:full_layer_layer_normrz attentionrintermediate_sizeffn ffn_outputr hidden_act activationr;r<r=rEs rHr/zAlbertLayer.__init__s  '-'E$%'\&2D&J_%`%`%`"(009V/1IJJ)F$PQQ !23z&"<== rINrrhrjrLc |j||fi|\}}t|j|j|j|}|||z}|Sr)rrff_chunkrrr)rFrrhrjattention_output_rs rHrYzAlbertLayer.forwardsi -dn]NUUfUU!. M  (      22:@P3PQQ rIrc||}||}||}|Sr)rrr)rFrrs rHrzAlbertLayer.ff_chunks<XX.// __Z00 __Z00 rIr)rZr[r\r r/r?r`r_rrrrYrrarbs@rHrrs >| > > > > > > 48|)D0+,  u|U\) * "%,rIrc eZdZdeffd Z d dejdejdzdee de eje ejzdffd Z xZ S) AlbertLayerGroupr#cttjfdt jD|_dS)Nc.g|]}tS)r.0rr#s rH z-AlbertLayerGroup.__init__..s!+g+g+gAK,?,?+g+g+grI)r.r/r ModuleListrangeinner_group_num albert_layersrEs `rHr/zAlbertLayerGroup.__init__sR ]+g+g+g+gvOeIfIf+g+g+ghhrINrrhrjrL.c Nt|jD]\}}|||fi|}|Sr) enumerater)rFrrhrj layer_index albert_layers rHrYzAlbertLayerGroup.forwardsF *343E)F)F R R %K(LQQ&QQMMrIrrrbs@rHrrsi|iiiiii48|)D0+,  u|eEL1136 7 rIrc leZdZdeffd Z d dejdejdzdee de e zfdZ xZ S) AlbertTransformerr#ct|_tjjj|_tjfdtj D|_ dS)Nc.g|]}tSr)rrs rHrz.AlbertTransformer.__init__.. s"1t1t1tq2B62J2J1t1t1trI) r.r/r#rrr2r|embedding_hidden_mapping_inrrnum_hidden_groupsalbert_layer_groupsrEs `rHr/zAlbertTransformer.__init__su  +-9V5JFL^+_+_(#%=1t1t1t1tTYZ`ZrTsTs1t1t1t#u#u   rINrrhrjrLc ||}t|jjD]@}t ||jj|jjz z }|j|||fi|}At|S)N)last_hidden_state)rrr#num_hidden_layersintrrr )rFrrhrji group_idxs rHrYzAlbertTransformer.forwards 88GG t{455  AA!>A^!^_``I?D4Y?MM ????rIr)rZr[r\r r/r?r`r_rrr rrYrarbs@rHrrsv|vvvvvv48@@|@)D0@+, @ 5 @@@@@@@@rIrcjeZdZeZdZdZdZdZdZ e e dZ e jfdZxZS)AlbertPreTrainedModelalbertT)r attentionsct|t|trt j|jdSt|trjt j|j tj |j j d dt j|jdSdS)zInitialize the weights.r*r)N)r. _init_weights isinstance AlbertMLMHeadinitzeros_biasr"copy_r(r?r@rRrAr,)rFrdrGs rHrz#AlbertPreTrainedModel._init_weights0s f%%% fm , , / K $ $ $ $ $  0 1 1 / Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h K- . . . . . / /rI)rZr[r\r config_classbase_model_prefix_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrz_can_record_outputsr?no_gradrrarbs@rHrr#sL N"&$% U]__////_/////rIrz2 Output type of [`AlbertForPreTraining`]. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dZ e ejdzed<dS)AlbertForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). sop_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logits sop_logitsrr) rZr[r\r]rr?r___annotations__rrrrrrrIrHrr;s  &*D% d ")))26u(4/666+/J!D(///59M5*+d299926Je'(4/66666rIrc$eZdZeZdZddedeffd Zdej fdZ dej dd fd Z e e e dd ejd zd ejd zd ejd zdejd zdejd zdeedeezfdZxZS) AlbertModelrTr#add_pooling_layerct|||_t||_t ||_|r=tj|j |j |_ tj |_ nd|_ d|_ |j |_|dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N)r.r/r#r"rXrencoderrrr|poolerTanhpooler_activationrattn_implementation post_init)rFr#rrGs rHr/zAlbertModel.__init__Ys     *622(00  *)F$68JKKDK%'WYYD " "DK%)D "#)#>  rIrLc|jjSrrXr4rFs rHget_input_embeddingsz AlbertModel.get_input_embeddingsos ..rIrgNc||j_dSrr)rFrgs rHset_input_embeddingsz AlbertModel.set_input_embeddingsrs*/'''rIrJrhr,r(rKrjc \|du|duz rtd|||||}t|j||}|j||fd|i|}|d} |j2||| dddfnd} t| | S)Nz:You must specify exactly one of input_ids or inputs_embeds)r(r,rK)r#rKrhr(r)r pooler_output)r~rXr r#rrrr) rFrJrhr,r(rKrjembedding_outputencoder_outputssequence_output pooled_outputs rHrYzAlbertModel.forwardus  -t"; < [YZZ Z?? L_l+  3;*)    '$,    &    *!,VZVaVm..t{{?111a4;P/Q/QRRRsw )-'    rI)T)NNNNN)rZr[r\r rrboolr/rr0rrrrrr?r^r_rrrrrYrarbs@rHrrTs^L |,/bl////0",040000 .237260426 $ $ #d*$ )D0$ (4/ $ &- $ (4/ $ +,$  $e +$ $ $ ^_ $ $ $ $ $ rIrz Albert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `sentence order prediction (classification)` head. cReZdZdddZdeffd ZdejfdZdejdd fd Z dej fd Z e e dd ejd zd ejd zdejd zdejd zdejd zdejd zdejd zdeedeezfdZxZS)AlbertForPreTraining(albert.embeddings.word_embeddings.weightpredictions.biaszpredictions.decoder.weightzpredictions.decoder.biasr#ct|t||_t ||_t ||_|dSr) r.r/rrr predictions AlbertSOPHeadsop_classifierrrEs rHr/zAlbertForPreTraining.__init__sb    !&)) (00+F33 rIrLc|jjSrrdecoderrs rHget_output_embeddingsz*AlbertForPreTraining.get_output_embeddings''rInew_embeddingsNc||j_dSrrrFrs rHset_output_embeddingsz*AlbertForPreTraining.set_output_embeddingss#1   rIc$|jjjSrrrXr4rs rHrz)AlbertForPreTraining.get_input_embeddings{%55rIrJrhr,r(rKlabelssentence_order_labelrjc |j|f||||dd|} | dd\} } || } || } d}||t}|| d|jj|d}|| dd|d}||z}t|| | | j| j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` sentence_order_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. `0` indicates original order (sequence A, then sequence B), `1` indicates switched order (sequence B, then sequence A). Example: ```python >>> from transformers import AutoTokenizer, AlbertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForPreTraining.from_pretrained("albert/albert-base-v2") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> sop_logits = outputs.sop_logits ```Trhr,r(rK return_dictNrmr*)rrrrr) rrrrrr#r1rrr)rFrJrhr,r(rKr r rjoutputsrrprediction_scores sop_scores total_lossloss_fctmasked_lm_losssentence_order_losss rHrYzAlbertForPreTraining.forwards3N$+  ))%'     *1!& ,,_==((77   "6"B'))H%X&7&<&!?!?@@ Yv163HII y!68IJJ  !23rIrrLc||}||}||}||}|}|Sr)rrr9r)rFrrs rHrYzAlbertMLMHead.forwardsR =11  66 }55  ]33 )  rI rZr[r\r r/r?r`rYrarbs@rHrrsj4|444444!U\!el!!!!!!!!rIrcHeZdZdeffd ZdejdejfdZxZS)rr#cttj|j|_tj|j|j|_ dSr) r.r/rr;classifier_dropout_probr=rr| num_labels classifierrEs rHr/zAlbertSOPHead.__init__sJ z&"@AA )F$68IJJrIrrLcZ||}||}|Sr)r=r!)rFrdropout_pooled_outputlogitss rHrYzAlbertSOPHead.forward s+ $ ] ; ;!677 rIrrbs@rHrrsqK|KKKKKK U\elrIrc6eZdZdddZfdZdejfdZdejddfd Zdej fd Z e e dd e jdzd e jdzd e jdzde jdzde jdzde jdzdeedeezfdZxZS)AlbertForMaskedLMrrrct|t|d|_t ||_|dSNF)r)r.r/rrrrrrEs rHr/zAlbertForMaskedLM.__init__-sW    !&EBBB (00 rIrLc|jjSrrrs rHrz'AlbertForMaskedLM.get_output_embeddings6rrIrNc@||j_|j|j_dSr)rrrrs rHrz'AlbertForMaskedLM.set_output_embeddings9s!#1 . 3rIc$|jjjSrrrs rHrz&AlbertForMaskedLM.get_input_embeddings=r rIrJrhr,r(rKr rjc <|jd|||||dd|}|d} || } d} |Kt} | | d|jj|d} t | | |j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, AlbertForMaskedLM >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForMaskedLM.from_pretrained("albert/albert-base-v2") >>> # add mask_token >>> inputs = tokenizer("The capital of [MASK] is Paris.", return_tensors="pt") >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> # retrieve index of [MASK] >>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0] >>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1) >>> tokenizer.decode(predicted_token_id) 'france' ``` ```python >>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"] >>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100) >>> outputs = model(**inputs, labels=labels) >>> round(outputs.loss.item(), 2) 0.81 ``` TrJrhr,r(rKrrNr*rr$rrr) rrrrr#r1rrr) rFrJrhr,r(rKr rjrsequence_outputsrrrs rHrYzAlbertForMaskedLM.forward@s^$+ ))%'     #1: ,,-=>>  '))H%X&7&<& 1` a classification loss is computed (Cross-Entropy). Tr-rN regressionsingle_label_classificationmulti_label_classificationr*r.r)rr=r!r# problem_typer r-r?rDrrsqueezerrrrrr) rFrJrhr,r(rKr rjrrr$rrs rHrYz'AlbertForSequenceClassification.forwards$$+ ))%'        ]33 //  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     rIr0)rZr[r\r r/rrr?r^r_rrrrrYrarbs@rHr2r2s |      .237260426*.; ; #d*; )D0; (4/ ; &- ; (4/ ;  4'; +,;  "E ); ; ; ^; ; ; ; ; rIr2ceZdZdeffd Zee d dejdzdej dzdejdzdejdzdej dzd ejdzd e e d e e zfd ZxZS)AlbertForTokenClassificationr#cdt||j|_t|d|_|j|jn|j}tj||_ tj |j |j j|_ |dSr()r.r/r rrrr<rr;r=rr|r#r!r)rFr#rrGs rHr/z%AlbertForTokenClassification.__init__s     +!&EBBB -9  * *+ z"9:: )F$6 8NOO rINrJrhr,r(rKr rjrLc \|j|f||||dd|}|d} || } || } d} |Ft} | | d|j|d} t | | |j|jS)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Tr rNr*r.) rr=r!rrr rrr) rFrJrhr,r(rKr rjrrr$rrs rHrYz$AlbertForTokenClassification.forwards $+  ))%'     "!*,,7711  '))H8FKKDO<G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  $+  ))%'        ]33 #}== ++b+66  '))H8OV44D("!/)     rIr0rNrbs@rHrPrP^s"|.237260426*.M M #d*M )D0M (4/ M &- M (4/ M  4'M +,M  $e +M M M ^M M M M M rIrP)rrrr&r2r;r?rP)Nrc)Er]collections.abcr dataclassesrr?rtorch.nnrrrr r activationsr masking_utilsr modeling_outputsr rrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_albertr get_loggerrZloggerModuler"r`floatrxrzrrrrrrrrrr&r2r;r?rP__all__rrIrHris$$$$$$!!!!!! AAAAAAAAAA&&&&&&!!!!!!666666GFFFFFFF&&&&&&NMMMMMMMMMMMIIIIIIII555555......  H % %=====ry===N!%% I% <% % < % L4' % T\ %%'(%%%%8>)>)>)>)>)bi>)>)>)B#####")###Lry"@@@@@ @@@://///O///.  7777777   7&G G G G G 'G G G T \ \ \ \ \ 0\ \  \ ~!!!!!BI!!!*     BI   _ _ _ _ _ -_ _ _ D J J J J J &;J J  J Z: : : : : #8: : : z@ @ @ @ @ !6@ @ @ FZ Z Z Z Z 3Z Z Z z   rI