tj-dZddlZddlmZddlmZddlZddlZddlm Z ddl m Z m Z m Z ddlmZdd lmZmZdd lmZmZdd lmZdd lmZmZmZmZmZmZdd l m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'ddl(m)Z)e'j*e+Z,dZ-d@dZ.Gdde j/Z0Gdde j/Z1e&dGdde j/Z2e&deGdde%Z3Gdd e j/Z4Gd!d"e j/Z5Gd#d$e j/Z6Gd%d&e j/Z7Gd'd(e j/Z8e&Gd)d*e!Z9e&Gd+d,e9Z:e&d-Gd.d/e9eZ;e&d0Gd1d2e9Ze&d8eGd9d:e%Z?e&Gd;de9ZAgd?ZBdS)Az%PyTorch Flaubert model, based on XLM.N)Callable) dataclass)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)geluget_activation) DynamicCacheEncoderDecoderCache)GenerationMixin)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward) ModelOutputauto_docstringlogging)FlaubertConfigc tjfdt|D}d|_t jtj|dddddf|dddddf<t jtj|dddddf|dddddf<||S)NcJg|]fdtDS)c Rg|]#}tjdd|dzzz z $S)i')nppower).0jdimposs l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/flaubert/modeling_flaubert.py z;create_sinusoidal_embeddings...0s7\\\AcBHUAaL34F$G$GG\\\)range)r#r&r%s @r'r(z0create_sinusoidal_embeddings..0s=uuuad\\\\\QVWZQ[Q[\\\uuur)Frr r) r!arrayr* requires_gradtorch FloatTensorsincosdetach_)n_posr%out position_encs ` r'create_sinusoidal_embeddingsr5/s8uuuuhmnshthtuuuvvLC$RVLADqD,A%B%BCCC14a4L$RVLADqD,A%B%BCCC14a4LKKMMM Jr)ctj|tj|j}||}n<||ksJ||dddfk}|d}|r2|ddddf||d|ddddfk}n|}|||fksJ|dus||||fksJ||fS)zH Generate hidden states mask, and optionally an attention mask. dtypedeviceNrrF)r-arangelongr9maxitemsizerepeat)slenlengthscausal padding_maskalenmaskbs attn_masks r' get_masksrH9s <EJw~ F F FD{{}}!!##t++++gaaag&& aB tQQQ'..r4;;tD!!!TM?RR   99;;2t* $ $ $ $ U??inn..2tT2BBBB B ?r)c4eZdZddeffd Z ddZxZS) MultiHeadAttentionr layer_idxct||_||_||_||z|_|j|_|j|jzdksJtj |||_ tj |||_ tj |||_ tj |||_ dS)Nr)super__init__layer_idr%n_headshead_dimattention_dropoutdropoutrLinearq_link_linv_linout_lin)selfrPr%configrK __class__s r'rNzMultiHeadAttention.__init__Ts !  w / x$,&!++++YsC(( YsC(( YsC(( yc** r)NFc |\}}} |du} |dkr|d|dfn|dddf} |||d|j|jdd} |Ht|tr1|j |j } | r|j }n |j }n|}| r|n|}| r)|'| r%|j|j }|j|j }n||}||}||d|j|jdd}||d|j|jdd}|0||||j \}}| rd|j |j <| t'j|jz } t+j| |dd}|dk| |}||t+j|jjt8j|d |}t8j!||j!|j" }t+j||}|dd#|d|j|jz}|$|f}|r||fz}|S) zd Self-attention (if kv is None) or attention over source sentence (provided by kv). Nr rr Trr%ptraining)%r>r%rUviewrPrQ transpose isinstancer is_updatedgetrOcross_attention_cacheself_attention_cache key_cache value_cacherVrWupdatemathsqrtr-matmul expand_as masked_fill_finfor8minr functionalsoftmaxfloattype_asrSra contiguousrX)rYinputrEkvcacheoutput_attentionskwargsrFqlenr%is_cross_attention mask_reshapeqrecurr_past_key_valuescurrent_stateskvscoresweightscontextoutputss r'forwardzMultiHeadAttention.forwardbs D#t^,0HHJJ!OOAtR(("aB JJu   " "2r4< G G Q QRSUV W W  %!455 -"-11$-@@ %F+0+F((+0+E((',$1<u  ;%"3 "3$.t}=A$0?AA >**A >**Ar2t|T];;EEaKKAr2t|T];;EEaKKA +221aGG1%;6:E$T]3  $-(( (aQ!2!233  --77??D%+fl";";"?@@@-'' B'??GGOO-''4<$-'XX,w**##Aq))4466;;BDLSWS`D`aa<<((*  + *Gr))r)NNF)__name__ __module__ __qualname__intrNr __classcell__r[s@r'rJrJSsh + + + + + + + +$  >>>>>>>>r)rJc*eZdZfdZdZdZxZS)TransformerFFNc6t|j|_tj|||_tj|||_|jrtntj j |_ |j |_ d|_ dSNr)rMrNrSrrTlin1lin2gelu_activationr rsreluactchunk_size_feed_forward seq_len_dim)rYin_dim dim_hiddenout_dimrZr[s r'rNzTransformerFFN.__init__sy ~ Ifj11 Ij'22 !1I44r}7I'-'E$r)cDt|j|j|j|SN)rff_chunkrr)rYrxs r'rzTransformerFFN.forwards (8TVZVfhmnnnr)c||}||}||}tj||j|j}|S)Nr_)rrrrrsrSra)rYrxxs r'rzTransformerFFN.ff_chunksV IIe   HHQKK IIaLL M ! !!t|dm ! L Lr))rrrrNrrrrs@r'rrsYooor)rzl The bare Flaubert Model transformer outputting raw hidden-states without any specific head on top. ) custom_introc*eZdZdZfdZddZxZS)FlaubertPredLayerz? Prediction layer (cross_entropy or adaptive_softmax). cPt|j|_|j|_|j|_|j}|jdur#t j||jd|_dSt j ||j|j |j d|_dS)NFTbias) in_features n_classescutoffs div_value head_bias) rMrNasmn_words pad_indexemb_dimrrTprojAdaptiveLogSoftmaxWithLoss asm_cutoffs asm_div_value)rYrZr%r[s r'rNzFlaubertPredLayer.__init__s :~ )n :   #v~DAAADIII5 .* . DIIIr)Ncd}|jdurr||}|f|z}|Ttj|d|j|dd}|f|z}nA|j|}|f|z}||||\}}|f|z}|S)z,Compute the loss, and optionally the scores.FNr]mean) reduction)rrrrs cross_entropyrbrlog_prob)rYryrrloss_s r'rzFlaubertPredLayer.forwards 8u  YYq\\Fi')G}}226;;r4<3P3PRSRXRXY[R\R\hn2oo'G+Y''**Fi')G}))Aq//4'G+r)r)rrr__doc__rNrrrs@r'rrsV$r)rzl Base class for outputs of question answering models using a [`~modeling_utils.FlaubertSQuADHead`]. ceZdZUdZdZejdzed<dZejdzed<dZ ej dzed<dZ ejdzed<dZ ej dzed<dZ ejdzed<dS) FlaubertSquadHeadOutput9 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned if both `start_positions` and `end_positions` are provided): Classification loss as the sum of start token, end token (and is_impossible if provided) classification losses. start_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top config.start_n_top start token possibilities (beam-search). start_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top config.start_n_top start token possibilities (beam-search). end_top_log_probs (`torch.FloatTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). end_top_index (`torch.LongTensor` of shape `(batch_size, config.start_n_top * config.end_n_top)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Indices for the top `config.start_n_top * config.end_n_top` end token possibilities (beam-search). cls_logits (`torch.FloatTensor` of shape `(batch_size,)`, *optional*, returned if `start_positions` or `end_positions` is not provided): Log probabilities for the `is_impossible` label of the answers. Nrstart_top_log_probsstart_top_indexend_top_log_probs end_top_index cls_logits)rrrrrr-r.__annotations__rr LongTensorrrrrr)r'rrs"&*D% d ")))48*T1888/3OU%,33326u(4/666-1M5#d*111+/J!D(/////r)rcbeZdZdZdeffd Zd dejdejdzdejfdZxZ S) FlaubertPoolerStartLogitsz Compute SQuAD start logits from sequence hidden states. Args: config ([`FlaubertConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model. rZcttj|jd|_dSr)rMrNrrT hidden_sizedenserYrZr[s r'rNz"FlaubertPoolerStartLogits.__init__s3 Yv1155 r)N hidden_statesp_maskreturnc||d}|2|jtjkr|d|z zd|zz }n|d|z zd|zz }|S)a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. Returns: `torch.FloatTensor`: The start logits for SQuAD. r]Nrꌠ9Y>)F)rsqueezer8r-float16)rYrrrs r'rz!FlaubertPoolerStartLogits.forwardsm JJ} % % - -b 1 1  |u},,V$uv~5V$tf}4r)r) rrrrrrNr-r.rrrs@r'rr s6~666666U%6@QTX@Xdidur)rc eZdZdZdeffd Z d dejdejdzdejdzdejdzd ejf d Z xZ S) FlaubertPoolerEndLogitsz Compute SQuAD end logits from sequence hidden states. Args: config ([`FlaubertConfig`]): The config used by the model, will be used to grab the `hidden_size` of the model and the `layer_norm_eps` to use. rZcNttj|jdz|j|_tj|_tj|j|j |_tj|jd|_ dS)Nr epsr) rMrNrrTrdense_0Tanh activation LayerNormlayer_norm_epsdense_1rs r'rNz FlaubertPoolerEndLogits.__init__:sz y!3a!79KLL '))f&8f>STTTy!3Q77 r)Nr start_statesstart_positionsrrcJ| | Jd|a|jdd\}}|ddddfdd|}|d|}|d|d}|t j||gd}||}||}|| d}|2|j tj kr|d|z zd|zz }n|d|z zd|zz }|S) a Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`): The final hidden states of the model. start_states (`torch.FloatTensor` of shape `(batch_size, seq_len, hidden_size)`, *optional*): The hidden states of the first tokens for the labeled span. start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*): The position of the first token for the labeled span. p_mask (`torch.FloatTensor` of shape `(batch_size, seq_len)`, *optional*): Mask for tokens at invalid position, such as query and special symbols (PAD, SEP, CLS). 1.0 means token should be masked. One of `start_states` or `start_positions` should be not `None`. If both are set, `start_positions` overrides `start_states`. Returns: `torch.FloatTensor`: The end logits for SQuAD. N7One of start_states, start_positions should be not Noner]r^rrr) shapeexpandgatherrr-catrrrrr8r)rYrrrrr@hszrs r'rzFlaubertPoolerEndLogits.forwardAs>:'?+F+F E,G+FF  &%+BCC0ID#-aaatm<CCBCPPO(//ODDL'..r4<