g)8zPyTorch I-BERT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)gelu))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) IBertConfig)IntGELU IntLayerNorm IntSoftmaxQuantActQuantEmbedding QuantLinearc>^\rSrSrSrU4SjrSSjrSrSrU=r $)IBertEmbeddings0zN Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. cr>[TU]5 URUlSUlSUlSUlSUlSUl[URURURURURS9Ul [URURURURS9UlURS[ R""UR$5R'S5S S 9 [)US S 5UlURUl[UR$URUR,URURS9Ul[1URURS 9Ul[1URURS 9Ul[7URUR8URURUR:S9Ul[1UR URS 9Ul[@RB"URD5Ul#g)N ) padding_idx weight_bit quant_mode)r(r) position_ids)rF) persistentposition_embedding_typeabsoluter)eps output_bitr) force_dequant)$super__init__r) embedding_bitembedding_act_bitact_bit ln_input_bit ln_output_bitr vocab_size hidden_size pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddingsregister_buffertorcharangemax_position_embeddingsexpandgetattrr-r'position_embeddingsrembeddings_act1embeddings_act2rlayer_norm_epsr3 LayerNormoutput_activationrDropouthidden_dropout_probdropoutselfconfig __class__s `/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/ibert/modeling_ibert.pyr5IBertEmbeddings.__init__5s  ++!# -      ++))  &4  " "F$6$64CUCUbfbqbq& "  ELL)G)GHOOPWXej  (/v7PR\']$"..#1  * *   (()) $   ((>(>4??['(>(>4??[&   %%)) ..  "*$,,4??!Szz&"<"<= cUcDUb0[XRU5RUR5nOUR U5nUbUR 5nOUR 5SSnUc8[ R"U[ RURRS9nUcURU5upGOSnURU5upURUUUU S9upURS:Xa'URU5upURU U U U S9upURX5upUR!U 5n UR#X5upX4$)Nr+dtypedeviceidentityidentity_scaling_factorr.)"create_position_ids_from_input_idsr'torZ&create_position_ids_from_inputs_embedssizerBzeroslongr*r>r@rHr-rGrKrOrL)rQ input_idstoken_type_idsr* inputs_embedspast_key_values_length input_shapeinputs_embeds_scaling_factorr@$token_type_embeddings_scaling_factor embeddingsembeddings_scaling_factorrG"position_embeddings_scaling_factors rTforwardIBertEmbeddings.forwardis}  $A//1G "Y%%& $JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN  :>:N:Ny:Y 7M7+/ (FJF`F`aoFpC040D0D  (*$H 1E1 -   ' ': 5FJF^F^_kFl C 484H4H),(J 5I5 1J15z0e- \\*- 040F0Fz0m- 44rVcUR5SSnUSn[R"URS-X0R-S-[RUR S9nUR S5RU5$)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr+rrXr)rarBrCr'rcrZ unsqueezerE)rQrfrhsequence_lengthr*s rTr`6IBertEmbeddings.create_position_ids_from_inputs_embedss~$((*3B/ %a.||   q /4D4D"Dq"HPUPZPZcpcwcw %%a(// <)NNNNr) __name__ __module__ __qualname____firstlineno____doc__r5rnr`__static_attributes__ __classcell__rSs@rTr r 0s%2>jrs-5^==rVr c8^\rSrSrU4SjrSSjrSrU=r$)IBertSelfAttentionc ~>[TU]5 URUR-S:wa7[ US5(d&[ SURSURS35eUR UlSUlSUlSUl URUl[URUR- 5Ul URUR-Ul [URURSURURUR SS 9Ul[URURSURURUR SS 9Ul[URURSURURUR SS 9Ul[#URUR S 9Ul[#URUR S 9Ul[#URUR S 9Ul[#URUR S 9Ul[,R."UR05Ul[5US S 5UlUR6S :wa [ S 5e[9URUR UR:S9Ulg)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r#r&Tbiasr(bias_bitr) per_channelr/r-r.zDI-BERT only supports 'absolute' for `config.position_embedding_type`r)r3)r4r5r<num_attention_headshasattr ValueErrorr)r(rr8intattention_head_size all_head_sizerquerykeyvaluerquery_activationkey_activationvalue_activationrLrrMattention_probs_dropout_probrOrFr-rr3softmaxrPs rTr5IBertSelfAttention.__init__sB     : : :a ?PVXhHiHi#F$6$6#78 445Q8 !++  #)#=#= #&v'9'9Fu;:>((=:o7>Bjj>u;372G2G 3 / /3.A.A/.r+ 262G2G 3 / / %2$7$7! J!&&z2t7O7OQUQiQijtt q NN:r43K3KTMeMefppqrtuv !&&z2t7O7OQUQiQijtt q !<< 5H5HR5PQ $223+e3 ??.H.cfk.k +.2 +  %/.@ ;?,, =; 77 ,,7  - 9O _kB ) 5+ILf+f (+/ (%--aAq9DDF "/"4"4"6s";t?Q?Q>S"S%**,CD 7;6L6L 77 3 37H=/2mM]!*+I J.0  ---rV)r8rrrrOrrrrLr-r)rrrrrr(NNFrtrurvrwr5rnryrzr{s@rTr}r}s!8p| P.P.rVr}c.^\rSrSrU4SjrSrSrU=r$)IBertSelfOutputi8c t>[TU]5 URUlSUlSUlSUlSUlSUl[URURSURUR URSS9Ul [UR URS9Ul [URURURURURS9Ul[URURS9Ul[$R&"UR(5UlgNr#r&r%Trr/r0)r4r5r)r8r(rr9r:rr<denser ln_input_actrrJr3rKrLrrMrNrOrPs rTr5IBertSelfOutput.__init__9s  ++        ]]  %T%6%64??S%   %%)) ..  "*$,,4??!Szz&"<"<= rVcURX5upURU5nURUUUUS9upURX5upUR X5upX4$Nr[rrOrrKrLrQrr input_tensorinput_tensor_scaling_factors rTrnIBertSelfOutput.forwardVz6:jj6m3  ]3 6:6G6G  (!$? 7H7 3 7;nn]6q3 6:6L6L 7 3 ::rV rKr8rrrOrr9r:rLr)r(rr{s@rTrr8>:;;rVrc>^\rSrSrU4SjrSrSSjrSrU=r$)IBertAttentionigc>[TU]5 URUl[U5Ul[ U5Ul[5UlgN) r4r5r)r}rQroutputset pruned_headsrPs rTr5IBertAttention.__init__hs=  ++&v. %f- ErVc6[U5S:Xag[XRRURRUR 5up[ URRU5URl[ URRU5URl[ URRU5URl [ URRUSS9URl URR[U5- URlURRURR-URl UR RU5Ulg)Nrrdim)lenrrQrrrrrrrrrrunion)rQheadsindexs rT prune_headsIBertAttention.prune_headsos  u:? 7 9900$))2O2OQUQbQb  -TYY__eD *499==%@ ,TYY__eD .t{{/@/@%QO )- (E(EE (R %"&))"?"?$))B_B_"_  --33E:rVcURUUUUU5upgURUSUSX5upU4USS-n U 4USS-n X4$)Nrr)rQr) rQrrrrr self_outputsself_outputs_scaling_factorattention_outputattention_output_scaling_factorroutputs_scaling_factors rTrnIBertAttention.forwards59II  (    5 1 =AKK O8;]= 9$% QR(88"A!CFabcbdFe!e..rV)rrr)rQr) rtrurvrwr5rrnryrzr{s@rTrrgs"";, //rVrc.^\rSrSrU4SjrSrSrU=r$)IBertIntermediateic >[TU]5 URUlSUlSUlSUl[ URURSURUR URSS9Ul URS:wa [S5e[URURS9Ul[URURS9Ulg) Nr#r&Trr z3I-BERT only supports 'gelu' for `config.hidden_act`rr/)r4r5r)r8r(rrr<intermediate_sizer hidden_actrrr3intermediate_act_fnrrLrPs rTr5IBertIntermediate.__init__s  ++       $ $]]      &RS S#*dooU[UiUi#j !)$,,4??!SrVczURX5upURX5upURX5upX4$r)rrrL)rQrrs rTrnIBertIntermediate.forwardsL6:jj6m3 6:6N6N 7 3 7;6L6L 7 3 ::rV)r8rrrrLr)r(rr{s@rTrrsT( ; ;rVrc.^\rSrSrU4SjrSrSrU=r$) IBertOutputic t>[TU]5 URUlSUlSUlSUlSUlSUl[URURSURUR URSS9Ul [UR URS9Ul [URURURURUR S9Ul[URURS9Ul[&R("UR*5Ulgr)r4r5r)r8r(rr9r:rrr<rrrrrJr3rKrLrrMrNrOrPs rTr5IBertOutput.__init__s  ++    $ $   ]]  %T%6%64??S%   %%)) ..  "*$,,4??!Szz&"<"<= rVcURX5upURU5nURUUUUS9upURX5upUR X5upX4$rrrs rTrnIBertOutput.forwardrrVrrr{s@rTrrrrVrc>^\rSrSrU4SjrSSjrSrSrU=r$) IBertLayericL>[TU]5 URUlSUlSUl[ U5Ul[U5Ul[U5Ul [URURS9Ul [URURS9Ul g)Nr#rr/)r4r5r)r8 seq_len_dimr attentionr intermediaterrrpre_intermediate_actpre_output_actrPs rTr5IBertLayer.__init__s}  ++ '/-f5!&) $,T\\doo$V!&t||PrVcURUUUUUS9upgUSnUSn USSn URX5upU 4U -n U $)N)rrr)rfeed_forward_chunk) rQrrrrrself_attention_outputs%self_attention_outputs_scaling_factorrrr layer_outputlayer_output_scaling_factors rTrnIBertLayer.forwards}IM  (  / IWI E2!4*OPQ*R'(,484K4K 5 1  /G+rVcURX5upURX5up4URX45up4URX4X5upVXV4$r)rrrr)rQrrintermediate_output"intermediate_output_scaling_factorrrs rTrIBertLayer.feed_forward_chunksu<@[TU]5 XlURUl[R "[ UR5Vs/sHn[U5PM sn5Ul gs snfr) r4r5rRr)r ModuleListrangenum_hidden_layersrlayer)rQrRrrSs rTr5IBertEncoder.__init__$sT   ++]]fF^F^@_#`@_1Jv$6@_#`a #`sA/cFU(aSOSnU(aSOSn Sn [UR5H<upU(aX4-nUbXKOSn U "UUUU U5nUSnU(dM4XS4-n M> U(aX4-nU(d[SUUU U 455$[UUU U S9$)Nrrc3.# UH nUcMUv M g7frr).0vs rT 'IBertEncoder.forward..Ns" A s )last_hidden_stater attentionscross_attentions) enumeratertupler )rQrrrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_mask layer_outputss rTrnIBertEncoder.forward*s#7BD$5b4#(4OA#$58H$H!.7.CilO(,! M*!,M  &91=M[TU]5 URUl[R"UR UR 5Ul[R"5Ulgr) r4r5r)rLinearr<rTanh activationrPs rTr5IBertPooler.__init__asF  ++YYv1163E3EF '')rVc\USS2S4nURU5nURU5nU$Nr)rr1)rQrfirst_token_tensor pooled_outputs rTrnIBertPooler.forwardgs6+1a40 #56  6 rV)r1rr)rr{s@rTr-r-`s$ rVr-c4\rSrSr%\\S'SrSrSSjrSr g) IBertPreTrainedModeliprRibertc[U[[R45(akURR R SURRS9 URb%URR R5 gg[U[[R45(axURR R SURRS9 URb2URR URR5 gg[U[[R45(aJURR R5 URR R!S5 g[U["5(a%URR R5 gg)zInitialize the weightsg)meanstdNg?) isinstancerrr/weightdatanormal_rRinitializer_rangerzero_r Embeddingr'rrKfill_ IBertLMHead)rQmodules rT _init_weights"IBertPreTrainedModel._init_weightsusA f{BII6 7 7 MM   & &CT[[5R5R & S{{&   &&('  > ? ? MM   & &CT[[5R5R & S!!- ""6#5#56<<>. r|| < = = KK   " " $ MM   $ $S )  , , KK   " " $-rVNc[S5e)Nz6`resize_token_embeddings` is not supported for I-BERT.)NotImplementedError)rQnew_num_tokenss rTresize_token_embeddings,IBertPreTrainedModel.resize_token_embeddingss!"Z[[rVrr) rtrurvrwr__annotations__base_model_prefixrHrMryrrVrTr9r9ps %$\rVr9cf^\rSrSrSrSU4SjjrSrSrSr\ SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \S\ \S\ \S\\\\ R44Sjj5rSrU=r$) IBertModelia The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. c>[TU]U5 XlURUl[ U5Ul[ U5UlU(a [U5OSUl UR5 g)z^ add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r4r5rRr)r rkrencoderr-pooler post_init)rQrRadd_pooling_layerrSs rTr5IBertModel.__init__sX    ++)&1#F+ ->k&)D  rVc.URR$rrkr>rQs rTget_input_embeddingsIBertModel.get_input_embeddingss...rVc$XRlgrrZ)rQrs rTset_input_embeddingsIBertModel.set_input_embeddingss */'rVcUR5H7up#URRURR U5 M9 g)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrTrrr)rQheads_to_prunerrs rT _prune_headsIBertModel._prune_headss< +002LE LL  u % / / ; ;E B3rVrdrrer*rrfrr"r#returnc UbUOURRnUbUOURRnU bU OURRn UbUb [ S5eUb"UR X5 UR 5n O"UbUR 5SSn O [ S5eU upUb URO URn Uc[R"X4U S9nUc$[R"U [RU S9nURX*5nURXPRR5nURUUUUS9unnUR!UUUUUUU S9nUSnUR"bUR#U5OSnU (d UU4US S-$[%UUUR&UR(UR*S 9$) NzDYou cannot specify both input_ids and inputs_embeds at the same timer+z5You have to specify either input_ids or inputs_embeds)rZrX)rdr*rerf)rrrr"r#rr)r pooler_outputrrr)rRrr"use_return_dictr%warn_if_padding_and_no_attention_maskrarZrBonesrbrcget_extended_attention_mask get_head_maskrrkrTrUr rrr)rQrdrrer*rrfrr"r#rhrrrZextended_attention_maskembedding_outputembedding_output_scaling_factorencoder_outputssequence_outputr6s rTrnIBertModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y Q#..*K  &',,.s3KTU U!, %.%:!! @T@T  !"ZZ*)A6RN  !"[[EJJvVN150P0PQ_0m&&y++2O2OP <@OO%)' =L= 99 ,,  +2/!5#' *!,8< 8O O4UY #]3oab6II I;-')77&11,==   rV)rRrkrTrUr))T) NNNNNNNNN)rtrurvrwrxr5r\r_rdrrrB LongTensor FloatTensorboolrr r!rnryrzr{s@rTrRrRs "/0C156:59371559,0/3&*J E,,-J !!2!23J !!1!12 J u//0 J E--. J   1 12J $D>J 'tnJ d^J  ;U5CTCT=UU VJ J rVrRc^\rSrSrSS/rU4SjrSrSr\SS\ \ RS\ \ RS \ \ RS \ \ RS \ \ RS \ \ RS \ \ RS\ \ S\ \ S\ \ S\\\\ R44Sjj5rSrU=r$)IBertForMaskedLMizlm_head.decoder.biaszlm_head.decoder.weightc>[TU]U5 [USS9Ul[ U5UlUR 5 gNF)rW)r4r5rRr:rFlm_headrVrPs rTr5IBertForMaskedLM.__init__s6  %@ "6*  rVc.URR$r)r{decoderr[s rTget_output_embeddings&IBertForMaskedLM.get_output_embeddingss||###rVcZXRlURURlgr)r{r~r)rQnew_embeddingss rTset_output_embeddings&IBertForMaskedLM.set_output_embeddingss- *// rVrdrrer*rrflabelsrr"r#rfc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n SnUbF[ 5nU"U R SURR 5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrer*rrfrr"r#rr+rlosslogitsrr) rRrir:r{rrr;r rr)rQrdrrer*rrfrrr"r#rrrprediction_scoresmasked_lm_lossloss_fctrs rTrnIBertForMaskedLM.forwards(&1%1 'tn1 d^1  ~uU%6%677 81 1 rVrxc<^\rSrSrSrU4SjrSrSSjrSrU=r $)rFiLz)I-BERT Head for masked language modeling.c>[TU]5 [R"URUR5Ul[R "URURS9Ul[R"URUR5Ul [R"[R"UR55UlURURlg)N)r1)r4r5rr/r<rrKrJ layer_normr;r~ ParameterrBrbrrPs rTr5IBertLMHead.__init__Os YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@  II rVc URU5n[U5nURU5nURU5nU$r)rr rr~)rQfeatureskwargsxs rTrnIBertLMHead.forwardXs; JJx  G OOA  LLOrVcURRRRS:XaURURlgURRUlg)Nmeta)r~rrZtyper[s rT _tie_weightsIBertLMHead._tie_weightsbsC <<   # # ( (F 2 $ DLL  ))DIrV)rr~rr)rfN) rtrurvrwrxr5rnrryrzr{s@rTrFrFLs3&**rVrFz I-BERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ \R44Sjj5rSrU=r$)IBertForSequenceClassificationikc>[TU]U5 URUl[USS9Ul[ U5UlUR5 grz)r4r5 num_labelsrRr:IBertClassificationHead classifierrVrPs rTr5'IBertForSequenceClassification.__init__rsC   ++%@ 1&9 rVrdrrer*rrfrrr"r#rfc 0U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n SnUGbURRcUR S:XaSURlOoUR S:aNUR [R:XdUR [R:XaSURlOSURlURRS:XaI[5nUR S:Xa&U"U R5UR55nOU"X5nOURRS:Xa=[5nU"U RSUR 5URS55nO,URRS:Xa[5nU"X5nU (dU 4U S S-nUbU4U-$U$[UU U R U R"S 9$) ae labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrrr regressionsingle_label_classificationmulti_label_classificationr+rr)rRrir:r problem_typerrYrBrcrrsqueezerrrrrrrQrdrrer*rrfrrr"r#rrrrrrrs rTrn&IBertForSequenceClassification.forward|s(&1%B 'tnB d^B  'u/@/@)AA BB B rVrcl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ \R44Sjj5rSrU=r$)IBertForMultipleChoiceic>[TU]U5 [U5Ul[R "UR 5Ul[R"URS5Ul UR5 g)Nr) r4r5rRr:rrMrNrOr/r<rrVrPs rTr5IBertForMultipleChoice.__init__sV  ' zz&"<"<= ))F$6$6: rVrdrerrr*rrfrr"r#rfc \U bU OURRn UbURSOURSn Ub!URSUR S55OSn Ub!URSUR S55OSn Ub!URSUR S55OSnUb!URSUR S55OSnUb1URSUR S5UR S55OSnUR U U UUUUUU U S9 nUSnUR U5nURU5nURSU 5nSnUb[5nU"UU5nU (dU4USS-nUbU4U-$U$[UUURURS9$)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr+r)r*rerrrfrr"r#rr) rRrirrrar:rOrrrrr)rQrdrerrr*rrfrr"r# num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrr6rreshaped_logitsrrrs rTrnIBertForMultipleChoice.forwardsX&1%?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  ** *..,/!5#     ]3 / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rV)rrOr:r)rtrurvrwr5rrrBrtrurvrrr!rnryrzr{s@rTrrs$15596:-1371559,0/3&*W E,,-W !!1!12W !!2!23 W ))* W u//0 W E--.W   1 12W $D>W 'tnW d^W  (%0A0A*BB CW W rVrcl^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\ S \\ S \\ S \ \ \ \R44Sjj5rSrU=r$)IBertForTokenClassificationi)c.>[TU]U5 URUl[USS9Ul[ R "UR5Ul[ R"URUR5Ul UR5 grz) r4r5rrRr:rrMrNrOr/r<rrVrPs rTr5$IBertForTokenClassification.__init__+sk   ++%@ zz&"<"<= ))F$6$68I8IJ rVrdrrer*rrfrrr"r#rfc U bU OURRn URUUUUUUUU U S9 n U Sn URU 5n UR U 5n SnUb<[ 5nU"U R SUR5UR S55nU (dU 4U SS-nUbU4U-$U$[UU U RU RS9$)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr+rr) rRrir:rOrrrrrrrrs rTrn#IBertForTokenClassification.forward6s$&1%2 'tn2 d^2  $eE,=,=&>> ?2 2 rVrc2^\rSrSrSrU4SjrSrSrU=r$)rilz-Head for sentence-level classification tasks.c,>[TU]5 [R"URUR5Ul[R "UR5Ul[R"URUR5Ul gr) r4r5rr/r<rrMrNrOrout_projrPs rTr5 IBertClassificationHead.__init__os` YYv1163E3EF zz&"<"<=  &"4"4f6G6GH rVc USS2SSS24nURU5nURU5n[R"U5nURU5nUR U5nU$r4)rOrrBtanhr)rQrrrs rTrnIBertClassificationHead.forwardus^ Aq)  ]3  =1  =1  ]3  m4 rV)rrOr) rtrurvrwrxr5rnryrzr{s@rTrrls7I rVrc^\rSrSrU4Sjr\SS\\RS\\RS\\RS\\RS\\RS\\RS \\RS \\RS \\ S \\ S \\ S\ \ \ \R44Sjj5rSrU=r$)IBertForQuestionAnsweringic>[TU]U5 URUl[USS9Ul[ R "URUR5UlUR5 grz) r4r5rrRr:rr/r< qa_outputsrVrPs rTr5"IBertForQuestionAnswering.__init__sU   ++%@ ))F$6$68I8IJ rVrdrrer*rrfstart_positions end_positionsrr"r#rfc $U bU OURRn URUUUUUUU U U S9 n U Sn URU 5nUR SSS9unnUR S5R 5nUR S5R 5nSnUbUb[UR55S:aUR S5n[UR55S:aUR S5nURS5nURSU5nURSU5n[US9nU"X5nU"UU5nUU-S- nU (dUU4U SS-nUbU4U-$U$[UUUU RU RS9$) Nrrrr+r) ignore_indexr)r start_logits end_logitsrr)rRrir:rsplitrrrraclamprrrr)rQrdrrer*rrfrrrr"r#rrrrrr total_loss ignored_indexr start_lossend_lossrs rTrn!IBertForQuestionAnswering.forwards&1% E,,-> !!2!23> !!1!12 > u//0 > E--. >   1 12> "%"2"23>   0 01> $D>> 'tn> d^>  +U53D3D-EE F> > rVrcURU5R5n[R"USS9R U5U-U-nUR 5U-$)a1 Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's *utils.make_positions*. Args: input_ids (`torch.LongTensor`): Indices of input sequence tokens in the vocabulary. Returns: torch.Tensor rr)nerrBcumsumtype_asrc)rdr'rgmaskincremental_indicess rTr^r^sW << $ ( ( *D <<!4<rs$ " AA.Q,,cc   H %w=biiw=tK.K.\,;bii,;^./RYY./b; ;D,;")),;^7979t: 299: z "))  \?\\4t %t t nE +E E P*"))*> N %9N  N bc 1c c L? "6? ? Dbii&J 4J J Z4" rV