tj|dZddlmZddlmZddlZddlmZddlmZddl m Z dd l m Z dd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddlmZddlmZmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(e!j)e*Z+Gddej,Z- d@dej,dej.dej.dej.dej.dzde/de/fdZ0Gd d!ej,Z1Gd"d#ej,Z2Gd$d%ej,Z3Gd&d'ej,Z4Gd(d)ej,Z5Gd*d+eZ6Gd,d-ej,Z7eGd.d/eZ8eGd0d1e8Z9Gd2d3ej,Z:Gd4d5ej,Z;eGd6d7e8Z<ed89eGd:d;eZ=ed<9Gd=d>e8Z>gd?Z?dS)AzPyTorch Splinter model.)Callable) dataclassN)nn)CrossEntropyLoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutput ModelOutputQuestionAnsweringModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)merge_with_config_defaults)capture_outputs)SplinterConfigc eZdZdZfdZ d dejdzdejdzdejdzdejdzdef d Z xZ S) SplinterEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c8ttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_|dt%j|jdddS)N) padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandselfconfig __class__s l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/splinter/modeling_splinter.pyr'zSplinterEmbeddings.__init__+s !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej      N input_idstoken_type_idsr" inputs_embedsreturnc||}n|dd}|d}||jddd|f}|+tj|tj|jj}|||}||}||z}||} || z }| |}| |}|S)Nr$rdtypedevice) sizer"r7zeroslongrGr,r0r.r1r5) r;r@rAr"rB input_shape seq_lengthr0 embeddingsr.s r>forwardzSplinterEmbeddings.forward9s  #..**KK',,..ss3K ^  ,QQQ  ^ J J"%:: "66|DD)) ^^J// \\*-- r?)NNNN) __name__ __module__ __qualname____doc__r'r7 LongTensor FloatTensortuplerN __classcell__r=s@r>rr(sQQ      .2260426 #d*(4/&-  (4/   r?rmodulequerykeyvalueattention_maskscalingr5c tj||dd|z}|||z}tj|dtj|j}tj |||j }tj||} | dd } | |fS)Nrr$)dimrF)ptrainingr) r7matmul transposer functionalsoftmaxfloat32torFr5rc contiguous) rYrZr[r\r]r^r5kwargs attn_weights attn_outputs r>eager_attention_forwardrn[s<s}}Q':':;;gEL!#n4 =((2U](SSVVW\WbccL=((6?([[L,|U33K''1--88::K  $$r?c eZdZfdZ ddejdejdzdeede ejejdzffdZ xZ S) SplinterSelfAttentionct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ tj |j|j |_ tj |j|j |_ tj |j|j |_tj|j|_|j|_|jdz|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)r&r'r*num_attention_headshasattr ValueErrorr<intattention_head_size all_head_sizerLinearrZr[r\r3attention_probs_dropout_probr5attention_dropoutr^r:s r>r'zSplinterSelfAttention.__init__ss:    : :a ? ?PVXhHiHi ?8F$688 4888   #)#= #&v'9FrNzSplinterSelfAttention.forwardsf $)#2#. CCbC$*BCC zz-0055lCCMMaQRSS XXm,,11,??II!QOO zz-0055lCCMMaQRSS (?(M K ,.E) ) %8$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH L((r?N) rOrPrQr'r7TensorrTrrrUrNrVrWs@r>rprprs66666048))|))D0)+, ) u|U\D00 1 ))))))))r?rpcPeZdZfdZdejdejdejfdZxZS)SplinterSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr ) r&r'rrzr*denser1r2r3r4r5r:s r>r'zSplinterSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== r?r} input_tensorrCc||}||}|||z}|Srrr5r1r;r}rs r>rNzSplinterSelfOutput.forward@ =11  ]33 }|'CDD r?rOrPrQr'r7rrNrVrWs@r>rri>>>>> U\RWR^r?rc jeZdZfdZ ddejdejdzdeedejfdZ xZ S) SplinterAttentionctt||_t ||_dSr)r&r'rpr;routputr:s r>r'zSplinterAttention.__init__s; )&11 (00 r?Nr}r]rkrCc \|}|j|fd|i|\}}|||}|SNr])r;r)r;r}r]rkresidual_s r>rNzSplinterAttention.forwardsV !$49   )    q  M8<< r?r) rOrPrQr'r7rrTrrrNrVrWs@r>rrs1111148  | )D0 +,          r?rcBeZdZfdZdejdejfdZxZS)SplinterIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r&r'rrzr*intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr:s r>r'zSplinterIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r?r}rCcZ||}||}|Sr)rr)r;r}s r>rNzSplinterIntermediate.forwards, =11 00?? r?rrWs@r>rrs^99999U\elr?rcPeZdZfdZdejdejdejfdZxZS)SplinterOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r&r'rrzrr*rr1r2r3r4r5r:s r>r'zSplinterOutput.__init__sf Yv79KLL f&8f>STTTz&"<== r?r}rrCc||}||}|||z}|Srrrs r>rNzSplinterOutput.forwardrr?rrWs@r>rrrr?rc peZdZfdZ d dejdejdzdeedejfdZ dZ xZ S) SplinterLayerct|j|_d|_t ||_t ||_t||_ dS)Nr) r&r'chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr:s r>r'zSplinterLayer.__init__s^ '-'E$*622088$V,, r?Nr}r]rkrCc h|j|fd|i|}t|j|j|j|}|Sr)rrfeed_forward_chunkrr)r;r}r]rks r>rNzSplinterLayer.forwards] '   )   2  #T%A4CSUb  r?c\||}|||}|Sr)rr)r;attention_outputintermediate_output layer_outputs r>rz SplinterLayer.feed_forward_chunk s2"//0@AA{{#68HII r?r) rOrPrQr'r7rrTrrrNrrVrWs@r>rrs-----48|)D0+,   $r?rc `eZdZfdZ ddejdejdzdeede fdZ xZ S) SplinterEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tS)r).0ir<s r> z,SplinterEncoder.__init__..s!#c#c#caM&$9$9#c#c#cr?F) r&r'r<r ModuleListrangenum_hidden_layerslayergradient_checkpointingr:s `r>r'zSplinterEncoder.__init__s`  ]#c#c#c#c5IaCbCb#c#c#cdd &+###r?Nr}r]rkrCc J|jD] }|||fi|} t|S)Nlast_hidden_state)rr )r;r}r]rk layer_modules r>rNzSplinterEncoder.forwardsY !J  L(LMM +    r?r) rOrPrQr'r7rrTrrr rNrVrWs@r>rrs,,,,,48  | )D0 +,           r?rc<eZdZUeed<dZdZeedZ fdZ xZ S)SplinterPreTrainedModelr<splinterT)r} attentionsct|t|trQt j|jtj|jj d ddSdS)Nr$r#) r& _init_weightsrrinitcopy_r"r7r8rr9)r;rYr=s r>rz%SplinterPreTrainedModel._init_weights3s{ f%%% f0 1 1 i Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h h h i ir?) rOrPrQr__annotations__base_model_prefixsupports_gradient_checkpointingrrp_can_record_outputsrrVrWs@r>rr)sn "&*#&+ iiiiiiiiir?rceZdZdZfdZdZdZeee dde j dzde j dzde j dzd e j dzd e j dzd e e d eezfd ZxZS) SplinterModela2 The model is an encoder (with only self-attention) following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. ct|||_t||_t ||_|dSr)r&r'r<rrMrencoder post_initr:s r>r'zSplinterModel.__init__AsX     ,V44&v..  r?c|jjSrrMr,)r;s r>get_input_embeddingsz"SplinterModel.get_input_embeddingsKs ..r?c||j_dSrr)r;r\s r>set_input_embeddingsz"SplinterModel.set_input_embeddingsNs*/'''r?Nr@r]rAr"rBrkrCc 2||td|+||||}n.||dd}ntd|\}} ||jn|j} |t j|| f| }|!t j|tj| }|||||} t|j | |}|j | fd |i|} | d } t| S) a token_type_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timer$z5You have to specify either input_ids or inputs_embedsrGrE)r@r"rArB)r<rBr]r]rr) rv%warn_if_padding_and_no_attention_maskrHrGr7onesrIrJrMr r<rr )r;r@r]rAr"rBrkrK batch_sizerLrGembedding_outputencoder_outputssequence_outputs r>rNzSplinterModel.forwardQss6  ]%>cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @T  !"Z*j)A6RRRN  !"[EJvVVVN??%)' +  3;*)    '$,   )    *!,-    r?)NNNNN)rOrPrQrRr'rrrrrr7rrrrUr rNrVrWs@r>rr9s- ///000 *..2.2,0-1 @ @ <$&@  t+@  t+ @ lT) @ |d* @ +,@   @ @ @ ^_ @ @ @ @ @ r?rcDeZdZdfd ZdejdejfdZxZS)SplinterFullyConnectedLayergeluc t||_||_t j|j|j|_t||_t j |j|_ dSr) r&r' input_dim output_dimrrzrr act_fnr1)r;rrrr=s r>r'z$SplinterFullyConnectedLayer.__init__sa "$Yt~t?? Z( do66r?inputsrCc||}||}||}|Sr)rrr1)r;rr}s r>rNz#SplinterFullyConnectedLayer.forwards; 6**  M22 }55 r?)rrrWs@r>rrsc777777elu|r?rc(eZdZdZfdZdZxZS)QuestionAwareSpanSelectionHeadzf Implementation of Question-Aware Span Selection (QASS) head, described in Splinter's paper: ctt|j|j|_t|j|j|_t|j|j|_t|j|j|_tj |j|jd|_ tj |j|jd|_ dS)NF)bias) r&r'rr*query_start_transformquery_end_transformstart_transform end_transformrrzstart_classifierend_classifierr:s r>r'z'QuestionAwareSpanSelectionHead.__init__s %@ASU[Ug%h%h"#>v?QSYSe#f#f :6;MvOabb89KVM_`` " &*PW\ ] ] ] i(:FrNz&QuestionAwareSpanSelectionHead.forwardsKKMM 1c##B''..q!S99 V%@@@ 55mDD11-@@))&11 %%f----.>?? ''1a00 |M:>> ++N;; ##Aq!,,\-:: Z''r?)rOrPrQrRr'rNrVrWs@r>rrsV \ \ \ \ \(((((((r?rceZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd e e d e e zfd Z xZS)SplinterForQuestionAnsweringct|t||_t ||_|j|_|dSrr&r'rrr splinter_qassquestion_token_idrr:s r>r'z%SplinterForQuestionAnswering.__init__]    %f-- ;FCC!'!9 r?Nr@r]rAr"rBstart_positions end_positionsquestion_positionsrkrCc  d} ||Atjtj||jd} n?tj|dtj|j|j } | d}d} |j |f||||d| } | d} | | |\}}| r*| d | d }}|N|d |z tj|jjzz}|d |z tj|jjzz}d}||t#|d kr| d}t#|d kr| d}|d }|d||d|t'| }|||}|||}||zd z }t)|||| j| j S) a token_type_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `batch_size, sequence_length`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. FNr$)rar)rFlayoutrGTr]rAr"rBr ignore_indexr`lossrrr}r)r7argmaxeqrrwrIrHrJrrGrrr squeezefinforFminlenclamp_rrr}r)r;r@r]rAr"rBrrrrkquestion_positions_were_none"question_position_for_each_exampleoutputsrrr total_loss ignored_indexloss_fct start_lossend_losss r>rNz$SplinterForQuestionAnswering.forwardsD(-$  %$5:\Xi)?@@EEGGR666226;[!&&q))MDXanau6662"D!M!Mb!Q!Q +/ ($-  ))%'      "!*#'#5#5oGY#Z#Z j ' V'3';';A'>'> @R@RST@U@U*L  %'1~+=\M_A`A`Ad*ddL#q>'9U[IY=Z=Z=^&^^J  &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M  " "1m 4 4 4  M 2 2 2']CCCH!,@@Jx M::H$x/14J+%!!/)     r?NNNNNNNN)rOrPrQr'rrr7rrSrrrUrrNrVrWs@r>r r s>*..2.2,0-137156:W W <$&W  t+W  t+ W lT) W |d* W )D0W '$.W ",t3W +,W  - -W W W ^W W W W W r?r zB Class for outputs of Splinter as a span selection model. ) custom_introceZdZUdZdZejdzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dZ e ejdzed<dS)SplinterForPreTrainingOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when start and end positions are provided): Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. start_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-start scores (before SoftMax). end_logits (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length)`): Span-end scores (before SoftMax). Nrrrr}r) rOrPrQrRrr7rTrrrr}rUrrr?r>r,r,7s&*D% d ")))-1L%#d*111+/J!D(///59M5*+d299926Je'(4/66666r?r,z Splinter Model for the recurring span selection task as done during the pretraining. The difference to the QA task is that we do not have a question, but multiple question tokens that replace the occurrences of recurring spans instead. c6eZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd e e d e e zfd Z dejd ejfdZxZS)SplinterForPreTrainingct|t||_t ||_|j|_|dSrr r:s r>r'zSplinterForPreTraining.__init__Vrr?Nr@r]rAr"rBrrrrkrCc  B|||td||td|||}|j|f||||d| } | d} | \} } }|| |\}}|d}|x|d| || }|d|z tj|j j zz}|d|z tj|j j zz}d}||| dtd| dz | dtd| dz t|jj}||| |z| || |z}||| |z| || |z}||zdz }t#|||| j| j S) a input_ids (`torch.LongTensor` of shape `(batch_size, num_questions, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `batch_size, num_questions, sequence_length`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `batch_size, num_questions, sequence_length`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_questions, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. start_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. end_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (`sequence_length`). Position outside of the sequence are not taken into account for computing the loss. question_positions (`torch.LongTensor` of shape `(batch_size, num_questions)`, *optional*): The positions of all question tokens. If given, start_logits and end_logits will be of shape `(batch_size, num_questions, sequence_length)`. If None, the first question token in each sequence in the batch will be the only one for which start_logits and end_logits are calculated and they will be of shape `(batch_size, sequence_length)`. NzCquestion_positions must be specified in order to calculate the lossz?question_positions must be specified when inputs_embeds is usedrrrrr`r) TypeError_prepare_question_positionsrrHr rr9r7rrFrr maxrr<r+rr,r}r)r;r@r]rAr"rBrrrrkr#rrsequence_lengthrarr num_questions attention_mask_for_each_questionr$r&r'r(s r>rNzSplinterForPreTraining.forward`sj  %/*E-Jcabb b  'I,=]^^ ^  '!%!A!A)!L!L $-  ))%'      "!*+:+?+?+A+A( OS#'#5#5oGY#Z#Z j*//22  %/=/G/G/J/J/Q/QM?00 ,(1/O+OSXS^_k_qSrSrSv*vvL#q+K'Ku{[e[kOlOlOp&ppJ  &=+D  " "1c!_q-@&A&A B B B  C?Q+>$?$? @ @ @ (T[5MNNNH!!!*}">H%x/14J+%!!/)     r?c:tj||jjk\}}tj|}t |d|dkdtj||d}tj|d|j |z }tj |d| f|jj tj |j }||||f<|S)Nrz?All samples in the batch must have at least one question token.left)siderrE)r7wherer<rbincountrrH searchsortedr8rGfullr3r+rJ)r;r@rowsflat_positionsr5 first_idxcolsrs r>r2z2SplinterForPreTraining._prepare_question_positionss${9 8U+UVVnt,,    q ! !Y^^A%6%6 6 M   &tT??? |DIIaLL=== IJ ^^A   1 1 3 3 4 K $*#    !/ $*r?r))rOrPrQr'rrr7rrSrrrUr,rNr2rVrWs@r>r.r.Nsa*..2.2,0-137156:n n <$&n  t+n  t+ n lT) n |d* n )D0n '$.n ",t3n +,n  - -n n n ^n `U\elr?r.)r r.rrr)rX)@rRcollections.abcr dataclassesrr7rtorch.nnrrr activationsr masking_utilsr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrrr utils.genericrutils.output_capturingrconfiguration_splinterr get_loggerrOloggerModulerrfloatrnrprrrrrrrrrrr r,r.__all__rr?r>rVs%$$$$$$!!!!!! %%%%%%&&&&&&!!!!!!666666999999ZZZZZZZZZZFFFFFFFF&&&&&&666666jjjjjjjjjjjjjj777777555555222222  H % %////////t%% I% <% % < % L4' %  %%%%%.3)3)3)3)3)BI3)3)3)n         .     29        RY   .D     bi   2 i i i i io i i iZ Z Z Z Z +Z Z Z z")$#(#(#(#(#(RY#(#(#(Ld d d d d #:d d d N  77777;77   7"NNNNN4NNNb   r?