tjVdZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z dd lmZdd lmZmZmZmZmZmZdd lmZmZdd lmZdd lmZddlmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(e"j)e*Z+Gddej,Z-Gddej,Z.Gddej,Z/Gddej,Z0Gddej,Z1Gddej,Z2Gdd ej,Z3Gd!d"ej,Z4Gd#d$ej,Z5 dDd&ej,d'ej6d(ej6d)ej6d*ej6dzd+e7d,e7fd-Z8Gd.d/ej,Z9Gd0d1ej,Z:Gd2d3eZ;Gd4d5ej,Ze Gd:d;e=Z?e d<=Gd>d?e=Z@e d@=GdAdBe=ZAgdCZBdS)EzPyTorch MarkupLM model.)CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs)MarkupLMConfigc*eZdZdZfdZddZxZS)XPathEmbeddingszConstruct the embeddings from xpath tags and subscripts. We drop tree-id in this version, as its info can be covered by xpath. ctj|_tjj|jzj|_tjj |_ tj |_ tjj|jzdjz|_ tjdjzj|_tjfdt!|jD|_tjfdt!|jD|_dS)NcNg|]!}tjjj"S)r Embeddingmax_xpath_tag_unit_embeddingsxpath_unit_hidden_size.0_configs l/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/markuplm/modeling_markuplm.py z,XPathEmbeddings.__init__..@s;    VA6C`aa   cNg|]!}tjjj"Sr#)rr$max_xpath_subs_unit_embeddingsr&r's r+r,z,XPathEmbeddings.__init__..Gs;    VBFDabb   r-)super__init__ max_depthrLinearr& hidden_sizexpath_unitseq2_embeddingsDropouthidden_dropout_probdropoutReLU activationxpath_unitseq2_inner inner2emb ModuleListrangexpath_tag_sub_embeddingsxpath_subs_sub_embeddingsselfr* __class__s `r+r1zXPathEmbeddings.__init__3s? ))+63PSWSa3acicu)v)v&z&"<== '))$&If.Kdn.\^_bhbt^t$u$u!1v'9#96;MNN(*     t~..   ) ) %*,    t~..   * * &&&r-Nc g}g}t|jD]n}||j||dddd|f||j||dddd|fot j|d}t j|d}||z}||| | |}|S)Ndim) r>r2appendr?r@torchcatr<r8r:r;)rBxpath_tags_seqxpath_subs_seqxpath_tags_embeddingsxpath_subs_embeddingsixpath_embeddingss r+forwardzXPathEmbeddings.forwardMs' " "t~&& e eA ! ( ()I)Fq)I.YZYZYZ\]\]\]_`Y`Ja)b)b c c c ! ( ()J)G)J>Z[Z[Z[]^]^]^`aZaKb)c)c d d d d % *?R H H H % *?R H H H03HH>>$,,ttG`G`aqGrGr7s7s*t*tuur-)NN)__name__ __module__ __qualname____doc__r1rQ __classcell__rCs@r+rr-sV      4        r-rcdeZdZdZfdZedZeddZ d dZxZ S) MarkupLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct||_tj|j|j|j|_tj|j |j|_ |j |_ t||_ tj|j|j|_tj|j|j|_tj|j|_|dt-j|j dd|j|_tj|j |j|j|_ dS)N) padding_idxeps position_idsrrEF) persistent)r0r1r*rr$ vocab_sizer4 pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsr2rrPtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr6r7r8register_bufferrIarangeexpandr[rAs r+r1zMarkupLMEmbeddings.__init__bs?  !|F,=v?Q_e_rsss#%<0NPVPb#c#c ) / 7 7%'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej    ".#%<  *F,>DL\$ $ $    r-c|dd}|d}tj|dz||zdztj|j}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrErdtypedevicer)sizerIrklongrp unsqueezerl) inputs_embedsr[ input_shapesequence_lengthr^s r+&create_position_ids_from_inputs_embedsz9MarkupLMEmbeddings.create_position_ids_from_inputs_embedszs~$((**3B3/ %a.| !O_{:Q>ejYfYm   %%a((// <<*::;;5:V\777N  !![4uzd;''4>*::;;5:V\888N)"66|DD $ : :> J J00PP%(;;>SSVff ^^J// \\*-- r-)r)NNNNNN) rRrSrTrUr1 staticmethodrwrrQrVrWs@r+rYrY_sQQ     0==\=" 8 8 8\ 8"11111111r-rYcPeZdZfdZdejdejdejfdZxZS)MarkupLMSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr\) r0r1rr3r4denserhrir6r7r8rAs r+r1zMarkupLMSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== r- hidden_states input_tensorreturnc||}||}|||z}|SNrr8rhrBrrs r+rQzMarkupLMSelfOutput.forward@ =11  ]33 }|'CDD r-rRrSrTr1rITensorrQrVrWs@r+rri>>>>> U\RWR^r-rcBeZdZfdZdejdejfdZxZS)MarkupLMIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r0r1rr3r4intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrAs r+r1zMarkupLMIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r-rrcZ||}||}|Sr)rrrBrs r+rQzMarkupLMIntermediate.forwards, =11 00?? r-rrWs@r+rrs^99999U\elr-rcPeZdZfdZdejdejdejfdZxZS)MarkupLMOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r0r1rr3rr4rrhrir6r7r8rAs r+r1zMarkupLMOutput.__init__sf Yv79KLL f&8f>STTTz&"<== r-rrrc||}||}|||z}|Srrrs r+rQzMarkupLMOutput.forwardrr-rrWs@r+rrrr-rcBeZdZfdZdejdejfdZxZS)MarkupLMPoolercttj|j|j|_tj|_dSr)r0r1rr3r4rTanhr:rAs r+r1zMarkupLMPooler.__init__sC Yv163EFF '))r-rrcr|dddf}||}||}|S)Nr)rr:)rBrfirst_token_tensor pooled_outputs r+rQzMarkupLMPooler.forwards@+111a40 #566  66 r-rrWs@r+rrs^$$$$$ U\elr-rcBeZdZfdZdejdejfdZxZS)MarkupLMPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) r0r1rr3r4rrrrr transform_act_fnrhrirAs r+r1z(MarkupLMPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr-rrc||}||}||}|Sr)rrrhrs r+rQz'MarkupLMPredictionHeadTransform.forwards= =11 --m<< }55 r-rrWs@r+rrscUUUUUU\elr-rc$eZdZfdZdZxZS)MarkupLMLMPredictionHeadctt||_t j|j|jd|_t j tj |j|_ dS)NT)bias) r0r1r transformrr3r4radecoder ParameterrIrrrAs r+r1z!MarkupLMLMPredictionHead.__init__$sj 8@@y!3V5FTRRR LV->!?!?@@ r-cZ||}||}|Sr)rrrs r+rQz MarkupLMLMPredictionHead.forward-s*}55  ]33 r-)rRrSrTr1rQrVrWs@r+rr#sLAAAAAr-rcBeZdZfdZdejdejfdZxZS)MarkupLMOnlyMLMHeadcptt||_dSr)r0r1r predictionsrAs r+r1zMarkupLMOnlyMLMHead.__init__5s/ 3F;;r-sequence_outputrc0||}|Sr)r)rBrprediction_scoress r+rQzMarkupLMOnlyMLMHead.forward9s ,,_==  r-rrWs@r+rr4s^<<<<<!u|! !!!!!!!!r-rmodulequerykeyvalueattention_maskscalingr8c tj||dd|z}|||z}tj|dtj|j}tj |||j }tj||} | dd } | |fS)NrrE)rGro)ptrainingr) rImatmul transposer functionalsoftmaxfloat32toror8r contiguous) rrrrrrr8kwargs attn_weights attn_outputs r+eager_attention_forwardr?s<s}}Q':':;;gEL!#n4 =((2U](SSVVW\WbccL=((6?([[L,|U33K''1--88::K  $$r-c eZdZfdZ ddejdejdzdeede ejejdzffdZ xZ S) MarkupLMSelfAttentionct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ tj |j|j |_ tj |j|j |_ tj |j|j |_tj|j|_|j|_|jdz|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)r0r1r4num_attention_headshasattr ValueErrorr*rzattention_head_size all_head_sizerr3rrrr6attention_probs_dropout_probr8attention_dropoutrrAs r+r1zMarkupLMSelfAttention.__init__Ws:    : :a ? ?PVXhHiHi ?8F$688 4888   #)#= #&v'9F.s!#c#c#caM&$9$9#c#c#cr-F) r0r1r*rr=r>num_hidden_layerslayergradient_checkpointingrAs `r+r1zMarkupLMEncoder.__init__s`  ]#c#c#c#c5IaCbCb#c#c#cdd &+###r-Nrrrrc J|jD] }|||fi|} t|S)N)last_hidden_state)r r )rBrrr layer_modules r+rQzMarkupLMEncoder.forwardsY !J  L(LMM +    r-r) rRrSrTr1rIrrrrr rQrVrWs@r+r r s,,,,,48  | )D0 +,           r-r cbeZdZUeed<dZeedZe j fdZ xZ S)MarkupLMPreTrainedModelr*markuplm)r attentionscvt|t|trt j|jdSt|trQt j|j tj |j j d ddSdS)zInitialize the weightsrEr_N)r0 _init_weightsrrinitzeros_rrYcopy_r^rIrkrrl)rBrrCs r+rz%MarkupLMPreTrainedModel._init_weightss f%%% f6 7 7 i K $ $ $ $ $  2 3 3 i Jv*EL9L9RSU9V,W,W,^,^_f,g,g h h h h h i ir-) rRrSrTr__annotations__base_model_prefixrr_can_record_outputsrIno_gradrrVrWs@r+rrs| "&+ U]__iiii_iiiiir-rceZdZdfd ZdZdZeee dde j dzde j dzde j dzd e j dzd e j dzd e j dzd e j dzd e e deezfdZxZS) MarkupLMModelTct|||_t||_t ||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r0r1r*rYrr encoderrpooler post_init)rBr*add_pooling_layerrCs r+r1zMarkupLMModel.__init__ss     ,V44&v.. 0AKnV,,,t  r-c|jjSrrrc)rBs r+get_input_embeddingsz"MarkupLMModel.get_input_embeddingss ..r-c||j_dSrr')rBrs r+set_input_embeddingsz"MarkupLMModel.set_input_embeddingss*/'''r-Nr}rKrLrrr^rtrrc ||td|+||||} n.||dd} ntd||jn|j} |t j| | }|!t j| tj| }|dd} | |j } d | z d z} | |||||| } |j | | fi|} | d }|j | |nd}t||S)a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMModel >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base") >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 768] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerEz5You have to specify either input_ids or inputs_embeds)rprnrr)rog?g)r}rKrLr^rrtr)r pooler_output)r%warn_if_padding_and_no_attention_maskrqrprIrrrrrsrrorr"r#r )rBr}rKrLrrr^rtrrurpextended_attention_maskembedding_outputencoder_outputsrrs r+rQzMarkupLMModel.forward sJ  ]%>cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !"Z FCCCN  !"[EJvVVVN"0":":1"="="G"G"J"J"9"<"<4:"<"N"N#&)@#@H"L??))%)' +  '$,  #     *!,8< 8O O444UY )-'    r-)T)NNNNNNN)rRrSrTr1r(r*rrrrI LongTensorrrrrr rQrVrWs@r+r r sZ ///000 .2262637260426K K #d*K (4/K (4/ K )D0 K (4/ K &-K (4/K +,K  + +K K K ^_ K K K K K r-r c>eZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd ee d e eje zfdZ xZ S)MarkupLMForQuestionAnsweringct||j|_t|d|_t j|j|j|_| dSNF)r%) r0r1 num_labelsr rrr3r4 qa_outputsr$rAs r+r1z%MarkupLMForQuestionAnswering.__init___sj     +%fFFF )F$68IJJ r-Nr}rKrLrrr^rtstart_positions end_positionsrrc H|j|f||||||d| } | d} || } | dd\}}|d}|d}d}|| t |dkr|d}t | dkr| d} |d}|d|| d|t|}|||}||| }||zdz }t|||| j | j S) ae xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> html_string = " My name is Niels " >>> question = "What's his name?" >>> encoding = processor(html_string, questions=question, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> processor.decode(predict_answer_tokens).strip() 'Niels' ```rKrLrrr^rtrrrErFN) ignore_indexr)loss start_logits end_logitsrr) rr7splitsqueezerlenrqclamp_rrrr)rBr}rKrLrrr^rtr8r9routputsrlogitsr>r? total_loss ignored_indexloss_fct start_lossend_losss r+rQz$MarkupLMForQuestionAnswering.forwardisZ $-   ))))%'      "!*11#)<<r<#:#: j#++B//::<< ''++6688   &=+D?''))**Q.."1"9"9""="==%%''((1,, - 5 5b 9 9 (--a00M  " "1m 4 4 4  M 2 2 2']CCCH!,@@Jx M::H$x/14J+%!!/)     r-) NNNNNNNNN)rRrSrTr1rrrIrrrrrrQrVrWs@r+r3r3\sW*..2.2.2.2,0-1/3-1T T <$&T  t+T  t+ T  t+ T  t+ T lT)T |d*T ,T |d*T +,T  u| ; ;T T T ^T T T T T r-r3zC MarkupLM Model with a `token_classification` head on top. ) custom_introc(eZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd ee d e eje zfd Z xZ S)MarkupLMForTokenClassificationcZt||j|_t|d|_|j|jn|j}tj||_ tj |j |j|_ | dSr5)r0r1r6r rclassifier_dropoutr7rr6r8r3r4 classifierr$rBr*rOrCs r+r1z'MarkupLMForTokenClassification.__init__s     +%fFFF )/)B)NF % %TZTn z"455 )F$68IJJ r-Nr}rKrLrrr^rtlabelsrrc >|j|f||||||d| } | d} || } d} |Kt}|| d|jj|d} t | | | j| jS)a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> processor.parse_html = False >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> nodes = ["hello", "world"] >>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] >>> node_labels = [1, 2] >>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```r;rNrEr=rErr) rrPrrr*r6rrr)rBr}rKrLrrr^rtrRrrDrrr=rHs r+rQz&MarkupLMForTokenClassification.forwardsV $-   ))))%'      "!* OOO<<  '))H8!&&r4;+ABB BD %$!/)     r-NNNNNNNN)rRrSrTr1rrrIrrrrrrQrVrWs@r+rMrMsA     *..2.2.2.2,0-1&*D D <$&D  t+D  t+ D  t+ D  t+ D lT)D |d*D  t#D +,D  u| ~ -D D D ^D D D D D r-rMz MarkupLM Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c(eZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd ee d e eje zfd Z xZ S)!MarkupLMForSequenceClassificationcdt||j|_||_t ||_|j|jn|j}tj ||_ tj |j |j|_ |dSr)r0r1r6r*r rrOr7rr6r8r3r4rPr$rQs r+r1z*MarkupLMForSequenceClassification.__init__(s     + %f-- )/)B)NF % %TZTn z"455 )F$68IJJ r-Nr}rKrLrrr^rtrRrrc |j|f||||||d| } | d} || } || } d} |Z|jjf|jdkr d|j_nN|jdkr7|jtjks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | } n|| |} n|jjdkrGt}|| d|j| d} n*|jjdkrt}|| |} t| | | j| jS) a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```r;rN regressionsingle_label_classificationmulti_label_classificationrErT)rr8rPr* problem_typer6rorIrrrzrrArrrrrr)rBr}rKrLrrr^rtrRrrDrrEr=rHs r+rQz)MarkupLMForSequenceClassification.forward7sT $-   ))))%'         ]33 //  {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B @ @&++b//RR)-III,..x//'!/)     r-rU)rRrSrTr1rrrIrrrrrrQrVrWs@r+rWrW sB     *..2.2.2.2,0-1&*T T <$&T  t+T  t+ T  t+ T  t+ T lT)T |d*T  t#T +,T  u| 7 7T T T ^T T T T T r-rW)r3rWrMr r)r)CrUcollections.abcrrIrtorch.nnrrrr r activationsr modeling_layersr modeling_outputsr r rrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrutils.output_capturingrconfiguration_markuplmr get_loggerrRloggerModulerrYrrrrrrrrfloatrrrrr rr r3rMrW__all__r#r-r+rps$$$$$$ AAAAAAAAAA&&&&&&!!!!!!999999GFFFFFFF&&&&&&666666RRRRRRRRRRRR777777555555222222  H % %/ / / / / bi/ / / dpppppppph             29        RY        RY    bi$     ry   "!!!!!")!!!$%% I% <% % < % L4' %  %%%%%.3)3)3)3)3)BI3)3)3)n ..D     bi   2iiiiioiii$f f f f f +f f f Rb b b b b #:b b b J V V V V V %<V V  V r g g g g g (?g g  g T   r-