tjjdZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZdd lmZdd lmZmZmZmZmZdd lmZdd lmZdd lmZmZddlmZej e!Z"Gddej#Z$Gddej#Z%Gddej#Z&Gddej#Z'Gddej#Z(Gddej#Z)Gddej#Z*GddeZ+Gd d!ej#Z,Gd"d#ej#Z-eGd$d%eZ.eGd&d'e.Z/ed()Gd*d+e.Z0eGd,d-e.Z1Gd.d/ej#Z2eGd0d1e.Z3gd2Z4dS)3zPyTorch LiLT model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)auto_docstringlogging) LiltConfigc:eZdZfdZ ddZdZdZxZS)LiltTextEmbeddingscttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_|dt%j|jdd|j|_tj|j|j|j|_ dS)N padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandrselfconfig __class__s d/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/lilt/modeling_lilt.pyr#zLiltTextEmbeddings.__init__+s  !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej    ".#%<  *F,>DL\$ $ $    NcN|K|4|||j|j}n||}||}n|dd}|+t j|t j|j j}|| |}| |}||z}| |}||z }| |}||}||fS)Nr dtypedevice)"create_position_ids_from_input_idsrtor?&create_position_ids_from_inputs_embedssizer3zeroslongrr(r,r*r-r1) r7 input_idstoken_type_idsr inputs_embeds input_shaper, embeddingsr*s r:forwardzLiltTextEmbeddings.forward?s/  $#FFyRVRbccff$    $JJ=YY  #..**KK',,..ss3K  !"[EJtO`OghhhN   00;;M $ : :> J J"%:: "66|DD)) ^^J// \\*-- <''r;c||}tj|d||z}||zS)a Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr3cumsumtype_asrE)r7rFrmaskincremental_indicess r:r@z5LiltTextEmbeddings.create_position_ids_from_input_idscs`||K((,,..$|Da888@@FF$N"''))K77r;c|dd}|d}tj|jdz||jzdztj|j}|d|S)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr rr=r)rCr3r4rrEr? unsqueezer5)r7rHrIsequence_lengthrs r:rBz9LiltTextEmbeddings.create_position_ids_from_inputs_embedsps$((**3B3/ %a.|  q /D4D"Dq"HPUPZcpcw   %%a((// <<">|"L"L&AD[&[#&*nn5P&Q&Q#&*ll3N&O&O#**sBB B*B%%B*)NN)rXrYrZr#rKr[r\s@r:r^r^sL>>>>>*++++++++r;r^c4eZdZdfd ZddZ d dZxZS) LiltSelfAttentionNct|j|jzdkr0t |ds t d|jd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj |j|jz|j|jz|_tj |j|jz|j|jz|_tj |j|jz|j|jz|_tj|j|_|j|_||_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r"r#r&num_attention_headshasattr ValueErrorrPattention_head_size all_head_sizerrjquerykeyvaluerh layout_query layout_key layout_valuer/attention_probs_dropout_probr1 layer_idx)r7r8rr9s r:r#zLiltSelfAttention.__init__s    : :a ? ?PVXhHiHi ?8F$688 4888  $*#= #&v'9F>,,77 _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S* *,CD  "67  3 22Gr;N)rNF)rXrYrZr#rrKr[r\s@r:rxrxsu######>%%%% AAAAAAAAr;rxcPeZdZfdZdejdejdejfdZxZS)LiltSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr) r"r#rrjr&denser-r.r/r0r1r6s r:r#zLiltSelfOutput.__init__#sf Yv163EFF f&8f>STTTz&"<== r;r input_tensorreturnc||}||}|||z}|Srrr1r-r7rrs r:rKzLiltSelfOutput.forward)@ =11  ]33 }|'CDD r;rXrYrZr#r3TensorrKr[r\s@r:rr"i>>>>> U\RWR^r;rc eZdZd fd Z d dejdejdejdzdedzdeejf d Z xZ S) LiltAttentionNctt|||_t ||_|j}|j|jz|_t ||_||_dS)Nr) r"r#rxr7routputr&rh layout_output)r7r8rori_hidden_sizer9s r:r#zLiltAttention.__init__1su %f BBB $V,,  ,#/63NN+F33,r;Frrrrrc|||||}||d|}||d|}||f|ddz}|S)Nrrrm)r7rr) r7rrrr self_outputsattention_outputlayout_attention_outputrs r:rKzLiltAttention.forward;suyy        ;;|A FF"&"4"4\!_m"T"T#%<= QRR@PPr;rr) rXrYrZr#r3r FloatTensorbooltuplerKr[r\s@r:rr0s------48). ||)D0   $;  u|  r;rcBeZdZfdZdejdejfdZxZS)LiltIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r"r#rrjr&intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr6s r:r#zLiltIntermediate.__init__Psn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r;rrcZ||}||}|Sr)rr)r7rs r:rKzLiltIntermediate.forwardXs, =11 00?? r;rr\s@r:rrOs^99999U\elr;rcPeZdZfdZdejdejdejfdZxZS) LiltOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r"r#rrjrr&rr-r.r/r0r1r6s r:r#zLiltOutput.__init__`sf Yv79KLL f&8f>STTTz&"<== r;rrrc||}||}|||z}|Srrrs r:rKzLiltOutput.forwardfrr;rr\s@r:rr_rr;rc eZdZd fd Z d dejdejdejdzdedzdeejf d Z d Z d Z xZ S) LiltLayerNct|j|_d|_t |||_t ||_t||_ |j }|j }|j |j z|_ |j |j z|_ t ||_ t||_||_ ||_ dS)Nrr)r"r#chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr&rrhlayout_intermediater)r7r8rrori_intermediate_sizer9s r:r#zLiltLayer.__init__ns '-'E$&vCCC,V44 ((  , & 8#/63NN#)#;v?Z#Z #3F#;#; '//,#8   r;Frrrrrc|||||}|d}|d}|dd}t|j|j|j|} t|j|j|j|} | | f|z}|S)N)rrrrm)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r7rrrrself_attention_outputsrrr layer_outputlayout_layer_outputs r:rKzLiltLayer.forwards"&   / "0" "  2!4"8";(,0  #T%A4CSUe  8  *D,H$JZ\s   !45?r;c\||}|||}|Sr)rrr7rintermediate_outputrs r:rzLiltLayer.feed_forward_chunks2"//0@AA{{#68HII r;c\||}|||}|Sr)rrrs r:rz#LiltLayer.layout_feed_forward_chunks4"667GHH))*=?OPP r;rr) rXrYrZr#r3rrrrrKrrr[r\s@r:rrms999999*48). ||)D0   $;  u|  : r;rceZdZfdZ d dejdejdejdzdedzd edzd edzd eeje zfd Z xZ S) LiltEncoderct|_tjfdt jD|_dS)Nc.g|]}tS)r).0_r8s r: z(LiltEncoder.__init__..s!#_#_#_!If$5$5#_#_#_r;)r"r#r8r ModuleListrangenum_hidden_layerslayerr6s `r:r#zLiltEncoder.__init__sV  ]#_#_#_#_uVE]?^?^#_#_#_`` r;NFTrrrroutput_hidden_states return_dictrc&|rdnd}|rdnd}t|jD]9\} } |r||fz}| ||||} | d}| d}|r || dfz}:|r||fz}|std|||fDSt|||S)Nrrrrmc3K|]}||V dSrr)rvs r: z&LiltEncoder.forward..s4 =  !===r;)last_hidden_stater attentions) enumeraterrr ) r7rrrrrrall_hidden_statesall_self_attentionsi layer_module layer_outputss r:rKzLiltEncoder.forwards$#7@BBD$5?bb4(44 P POA|# I$58H$H!(L! M*!,M)!,M  P&9]1=M>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer z5You have to specify either input_ids or inputs_embeds)r=)r?rG)rFrrGrH)r8rHr)rpr)rrrrrr)r pooler_outputrr)r8rrrr~%warn_if_padding_and_no_attention_maskrCr?r3rDrEonesr}rJrGr5r rrr r rr)r7rFrprrGrrHrrrkwargsrI batch_size seq_lengthr?buffered_token_type_ids buffered_token_type_ids_expandedembedding_outputlayout_embedding_outputencoder_outputssequence_outputr s r:rKzLiltModel.forwardsP2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @T <;{T1FSSSD  !"Z*j)A6RRRN  !t(899 [*./*HKZK*X'3J3Q3QR\^h3i3i0!A!&[ SY!Z!Z!Z)-%)' *9* * &,3;*)    #'"8"8dQ]"8"^"^,,  #)/!5# '  *!,8< 8O O444UY  J#]3oabb6II I)-')7&1     r;)T) NNNNNNNNN)rXrYrZr#r%r'rr3rrrr rKr[r\s@r:rrsU"///000*.$(.2.2,0-1)-,0#'j j <$&j lT!j  t+ j  t+ j lT) j |d*j  $;j #Tkj D[j  u| 9 9j j j ^j j j j j r;rz LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introceZdZfdZe ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd e dzd e dzd e dzd e eje zfdZ xZ S)LiltForSequenceClassificationct||j|_||_t |d|_t ||_|dSNF)r") r"r# num_labelsr8rrLiltClassificationHead classifierr!r6s r:r#z&LiltForSequenceClassification.__init__sg     + f>>> 088 r;NrFrprrGrrHlabelsrrrrc | | n |jj} ||||||||| |  } | d} || }d}|t||j}|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|||}n|jjdkrGt}||d|j|d}n*|jjdkrt!}|||}| s|f| d dz}||f|zn|St#||| j| j S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```NrprrGrrHrrrrr regressionsingle_label_classificationmulti_label_classificationr rmlosslogitsrr)r8rrr=rAr? problem_typer;r>r3rErPrsqueezerrrrrrr7rFrprrGrrHr>rrrr-rr5rFrEloss_fctrs r:rKz%LiltForSequenceClassification.forwards\&1%>> )/)B)NF % %TZTn z"455 )F$68IJJ r;NrFrprrGrrHr>rrrrc | | n |jj} ||||||||| |  } | d} || } || }d}|`||j}t}||d|j |d}| s|f| ddz}||f|zn|St||| j | j S)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nr@rr rmrD) r8rrr1r=rAr?rrr;rrrrIs r:rKz"LiltForTokenClassification.forward s)V&1%>> )F$68IJJ r;NrFrprrGrrHstart_positions end_positionsrrrrc | | n |jj} |||||||| | |  } | d}||}|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | j S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```Nr@rrr rM) ignore_indexrm)rE start_logits end_logitsrr)r8rrr]splitrHrlenrCclamprrrr)r7rFrprrGrrHr^r_rrrr-rr5rFrbrc total_loss ignored_indexrJ start_lossend_lossrs r:rKz LiltForQuestionAnswering.forward~s^&1%rys AAAAAAAAAA&&&&&&!!!!!!666666999999.-----666666,,,,,,,,******  H % %S=S=S=S=S=S=S=S=l5+5+5+5+5+295+5+5+pfffff fffT     RY   BI>     ry           77777*777t4 4 4 4 4 ")4 4 4 p         i i i i i/ i i iC C C C C #C C C L m m m m m $7m m  m `^ ^ ^ ^ ^ !4^ ^ ^ DRY,k k k k k 2k k k \   r;