tj:^ddlZddlZddlmZmZddlmZddlm Z ddl m Z ddl Z ddlmZddlmZe Z eeegeee ffZGd d Zd Zd+d eedeee ffdZeGddeZd eedeee ffdZd eedeee ffdZeGddZeGddeZd,dedzfdZ d,dedzfdZ!eGddeZ"eGddZ#eGdd eZ$eGd!d"e$Z%dee fd#Z&de j'e fd$Z(eGd%d&e$Z)eGd'd(eZ*eGd)d*eZ+dS)-N)CallableMapping) dataclass)randint)Any)PreTrainedTokenizerBase)PaddingStrategyc"eZdZddedzfdZdS)DataCollatorMixinNreturn_tensorsc||j}|dkr||S|dkr||Std|d)Nptnpz Framework 'z' not recognized!)r torch_call numpy_call ValueErrorselffeaturesr s ]/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/data/data_collator.py__call__zDataCollatorMixin.__call__&sd  !!0N T ! !??8,, , t # #??8,, ,L>LLLMM MN)__name__ __module__ __qualname__strrrrr r %s@NNtNNNNNNrr ct|ds |j|i|S|jdd}d|jd< |j|i|}||jd<n#||jd<wxYw|S)zz Pads without triggering the warning about how using the pad function is sub-optimal when using a fast tokenizer. deprecation_warningszAsking-to-pad-a-fast-tokenizerFT)hasattrpadr!get) tokenizerpad_args pad_kwargs warning_statepaddeds r"pad_without_fast_tokenizer_warningr*1s 94 5 56y}h5*5552667WY^__MGKI"#CDY7J77LY &'GHH= &'GHXXXX Ms  A A(rrreturncZ|dkrt|S|dkrt|SdS)a Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. rrN)torch_default_data_collatornumpy_default_data_collator)rr s rdefault_data_collatorr/Gs>"*8444 4  *8444 rc`eZdZUdZdZeed<ddeeee fdeee ffdZ dS) DefaultDataCollatora$ Very simple data collator that simply collates batches of dict-like objects and performs special handling for potential keys named: - `label`: handles a single value (int or float) per object - `label_ids`: handles a list of values per object Does not do any additional preprocessing: property names of the input object will be used as corresponding inputs to the model. See glue and ner for example of how it's useful. This is an object (like other data collators) rather than a pure function like default_data_collator. This can be helpful if you need to set a return_tensors value at initialization. Args: return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". rr Nrr+c4||j}t||Sr)r r/rs rrzDefaultDataCollator.__call__ts  !!0N$X~>>>rr) rrr__doc__r r__annotations__listdictrrrrrr1r1^sq$NC??d38n!5?tTWY\T\~??????rr1cddl}t|dts d|D}|d}i}d|vr|dt|d|jr|dn|d}t|t r|jn|j}|d|D||d<nd|vr|dt|d|jr#| d|D|d<nSt|ddt r|jn|j}|d |D||d<| D]\}d vr|t|tst||jr%| fd |D|<Zt|tj r7|tj fd |D|<|fd |D|<|S)Nrc,g|]}t|Srvars.0fs r z/torch_default_data_collator..~...DGG...rlabelcg|] }|d Sr@rr;s rr>z/torch_default_data_collator..s'E'E'Eq' 'E'E'Erdtypelabels label_idscg|] }|d SrFrr;s rr>z/torch_default_data_collator..s*L*L*La1[>*L*L*Lrcg|] }|d SrHrr;s rr>z/torch_default_data_collator..s+M+M+MqAkN+M+M+Mrr@rFc g|] }| Srrr<r=ks rr>z/torch_default_data_collator..s'?'?'?!'?'?'?rc g|] }| SrrrLs rr>z/torch_default_data_collator..s5M5M5Mqad5M5M5Mrc g|] }| SrrrLs rr>z/torch_default_data_collator..s(@(@(@!1(@(@(@r)torch isinstancerTensoritemintlongfloattensorstackitemsrrndarray from_numpy)rrPfirstbatchr@rDvrMs @rr-r-zsvLLL hqk7 + +/..X... QKE E %E'N6)3E'NEL)Q)Qeg##%%%W\]dWe(44E %+,,'E'EH'E'E'EU,SSh   % "4"@ eK(%, 7 7 \#kk*L*L8*L*L*LMME(OO",U;-?-BC"H"HYEJJekE#ll+M+MH+M+M+MUZl[[E(O BB1 * * *q}ZPQSVEWEW}!U\** B ;;'?'?'?'?h'?'?'?@@aArz** B ++BH5M5M5M5MH5M5M5M,N,NOOa <<(@(@(@(@x(@(@(@AAa Lrc(t|dts d|D}|d}i}d|vr|dt|dtjr|dn|d}t|t r tjn tj}tjd|D||d<nd|vr|dt|dtjr"tj d|D|d<n\t|ddt r tjn tj}tjd |D||d<| D]\}d vrx|vt|tsat|tjr$tj fd |D|<^tjfd |D|<|S) Nrc,g|]}t|Srr9r;s rr>z/numpy_default_data_collator..r?rr@cg|] }|d SrBrr;s rr>z/numpy_default_data_collator..s#A#A#A1AgJ#A#A#ArrCrErFcg|] }|d SrHrr;s rr>z/numpy_default_data_collator..'I'I'I1+'I'I'Ircg|] }|d SrHrr;s rr>z/numpy_default_data_collator..rcrrJc g|] }| SrrrLs rr>z/numpy_default_data_collator..$<$<$z/numpy_default_data_collator..rfr) rQrrrZrSrTint64float32arrayrXrYr)rr\r]r@rDr^rMs @rr.r.s hqk7 + +/..X... QKE E %E'N6)3E'NBJ)O)Ocg##%%%UZ[bUc&uc22B (#A#A#A#A#AOOOh   % "4"@ eK("* 5 5 X h'I'I'I'I'IJJE(OO *5+=a+@# F FVBHHBJE h'I'I'I'I'IQVWWWE(O >>1 * * *q}ZPQSVEWEW}!RZ(( >8$<$<$<$<8$<$<$<==a8$<$<$<$<8$<$<$<==a LrceZdZUdZeed<dZeeze zed<dZ e dzed<dZ e dzed<dZ eed <d eeeefd eeeffd ZdS) DataCollatorWithPaddinga Data collator that will dynamically pad the inputs received. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". r%TpaddingN max_lengthpad_to_multiple_ofrr rr+ct|j||j|j|j|j}d|vr|d|d<|d=d|vr|d|d<|d=|S)Nrmrnror r@rErF)r*r%rmrnror )rrr]s rrz DataCollatorWithPadding.__call__s{2 N L#6.     e  #GnE(Og %  #K0E(Ok" r)rrrr3r r4rmboolrr rnrTror r5r6rrrrrrlrls4'&&&,0GTCZ/ )000!Jd !!!%)d )))NCd38n!5$sCx.rrlceZdZUdZeed<dZeeze zed<dZ e dzed<dZ e dzed<dZ e ed <d Zeed <d Zd ZdS)"DataCollatorForTokenClassificationa} Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignore by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". r%TrmNrnrolabel_pad_token_idrr c ddl d|dvrdnd|dvrfd|Dnd}fd|D}tj|jjjd}||S|djd jj} fd |d krfd |D|<nfd |D|< | j |<|S)Nrr@rEc g|] }| Srrr<feature label_names rr>zADataCollatorForTokenClassification.torch_call..>>>''*%>>>rcPg|]"}fd|D#S)c(i|]\}}|k ||Srrr<rMr^r{s r zLDataCollatorForTokenClassification.torch_call...s$SSS11 ??q!???rrYrys rr>zADataCollatorForTokenClassification.torch_call..s7lllX_SSSS SSSlllrrrq input_idsctt|jr|St|Sr)rQrRtolistr5)tensor_or_iterablerPs rto_listz>DataCollatorForTokenClassification.torch_call..to_list.s9,el;; 3)00222*++ +rrightc^g|])}|jgt|z zz*Srrvlenr<r@rsequence_lengthrs rr>zADataCollatorForTokenClassification.torch_call..4sM!!!`e$"9!:oPSTYPZPZ>Z![[!!!rc^g|])}jgt|z z|z*Srrrs rr>zADataCollatorForTokenClassification.torch_call..8sJ!!!`e()_s5zz-IJWWUZ^^[!!!rrC) rPr*r%rmrnroshape padding_siderWrh) rrrEno_labels_featuresr]rr{rrrPs ` @@@@rrz-DataCollatorForTokenClassification.torch_calls '8A; 6 6WWH BLPXYZP[B[B[>>>>X>>>>aellllcklll2 N L#6     >L ,215~2  , , , , , 7 " "!!!!!!io!!!E*  !!!!!!io!!!E* "LLz):%+LNNj rcd|dvrdnd|dvrfd|Dnd}tj|jjj|dnd}||St j|djdjj}|d krfd |D|d<nfd |D|d<d | D}|S) Nr@rrEc g|] }| Srrrys rr>zADataCollatorForTokenClassification.numpy_call..Ar|rrrqrrrcfg|]-}t|jgt|z zz.Sr)r5rvrr<r@rrs rr>zADataCollatorForTokenClassification.numpy_call..RsH]bU t67?SQVZZ;WXXrcfg|]-}jgt|z zt|z.Sr)rvrr5rs rr>zADataCollatorForTokenClassification.numpy_call..VsH]b()_s5zz-IJTRW[[XrcVi|]&\}}|tj|tj'SrCrrjrh)r<rMr^s rrzADataCollatorForTokenClassification.numpy_call..Zs/JJJDAqBHQbh///JJJr) r*r%rmrnrorrjrrrY)rrrEr]rr{rs` @@rrz-DataCollatorForTokenClassification.numpy_call?sH '8A; 6 6WWH BLPXYZP[B[B[>>>>X>>>>ae2 N L#6#)>44t    >L(5#566z(_torch_collate_batch..ds+HHH!ELL%*L55HHHrc3JK|]}|dkVdSrNsizer<xlength_of_firsts r z'_torch_collate_batch..js2!Q!Q1!&&))">!Q!Q!Q!Q!Q!Qr)dimCYou are attempting to pad samples but the tokenizer you are using () does not have a pad token.c3@K|]}|dVdSrrr<rs rrz'_torch_collate_batch..ws,111QVVAYY111111rrr)rPrQr5tuplerrZrallrRrX pad_tokenr __class__rmaxnew_fullr pad_token_id enumeraterr) examplesr%roare_tensors_same_lengthrnresultiexamplerrPs @@r_torch_collate_batchr^sLLL(1+eRZ899IHHHHxHHHqk&&q))O"!Q!Q!Q!Q!Q!Q!QQQ0$6$>/TfBfjkBkBk(EL11 0;;xQ;// /" L$- L L L   1111111J%:8J+Ja+O+O!%771<@RR a[ ! !3x==*"=y?U V VF))55 7  !W , ,,3F1( a((( ) )-4F1w}Q''))) * * Mrct|dttfr d|D}t|dt fd|D}|r!| |zdkrt j|dS|jtd|j j dtd|D}|||zdkr ||zd z|z}t j t||f|j |dj }t|D]:\}}|jd kr|||d|jdf<%||||jd df<;|S) rrcNg|]"}tj|tj#Srrr<rs rr>z(_numpy_collate_batch..s)BBBABHQbh///BBBrc3>K|]}t|kVdSrrrs rrz'_numpy_collate_batch..s.!N!N#a&&O";!N!N!N!N!N!NrNaxisrrc34K|]}t|VdSrrrs rrz'_numpy_collate_batch..s(..SVV......rr)r fill_valuerDr)rQr5rrrrrXrrrrrfullrrDrrr) rr%rorrnrrrrs @r_numpy_collate_batchrs(1+e}--CBBBBB(1+&&O!!N!N!N!NX!N!N!NNN*$6$>/TfBfjkBkBkxq))))" L$- L L L   ..X.....J%:8J+Ja+O+O!%771<@RR WCMM:69CYaijkalar s s sF))55 7  !W , ,,3F1( a((( ) )-4F1w}Q''))) * * MrceZdZUdZeed<dZeeze zed<dZ e dzed<dZ e dzed<dZ eed <d eeeeffd ZdS) DataCollatorForMultipleChoicea Data collator that dynamically pads a batch of nested examples for multiple choice, so that all choices of all examples have the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences according to the model's padding side and padding index among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): Pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.5 (Volta). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". r%TrmNrnrorr rcddl}d|dvrdndfd|D}t|t|ddtfd|Dg}|j||j|j|jd }fd |D}| ||j |d<|S) Nrr@rEc:g|]}|Sr)pop)r<rr{s rr>z.s%BBBg'++j))BBBrrc3PK|]fdtDV dS)cRg|]"fdD#S)c(i|]\}}||Srr)r<rMr^rs rrzQDataCollatorForMultipleChoice.torch_call....s#33341aq!A$333rr)r<rrs @rr>zFDataCollatorForMultipleChoice.torch_call...s7 P P P33337==??333 P P PrN)range)r<r num_choicess @rrz;DataCollatorForMultipleChoice.torch_call..sB i iU\ P P P PU;=O=O P P P i i i i i ir)startrrqcFi|]\}}||dS))view)r<rMr^ batch_sizers rrz.s/RRRDAqAFF:{B77RRRrrC) rPrsumr%r#rmrnrorYrWrh) rrrPrE flat_examplesr]rr{rs @@@rrz(DataCollatorForMultipleChoice.torch_calls  !(8A; 6 6WWH BBBBBBB]] (1+k233  i i i i`h i i iqs   "" L#6 #  SRRRREKKMMRRR,,vU[,AAh r)rrrr3r r4rmrrrr rnrTror r5r6rrrrrrrs8'&&&,0GTCZ/ )000!Jd !!!%)d )))NC4S#X#7rrceZdZUdZeed<dZedzed<dZe e ze zed<dZ e dzed<dZe dzed<d Ze ed <d Ze ed <dd ZdS)DataCollatorForSeq2Seqa Data collator that will dynamically pad the inputs received, as well as the labels. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. model ([`PreTrainedModel`], *optional*): The model that is being trained. If set and has the *prepare_decoder_input_ids_from_labels*, use it to prepare the *decoder_input_ids* This is useful when using *label_smoothing* to avoid calculating loss twice. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single sequence is provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >= 7.0 (Volta). label_pad_token_id (`int`, *optional*, defaults to -100): The id to use when padding the labels (-100 will be automatically ignored by PyTorch loss functions). return_tensors (`str`, *optional*, defaults to `"pt"`): The type of Tensor to return. Allowable values are "np", or "pt". r%NmodelTrmrnrorurvrr c |j}d|dvrdnd |dvr fd|Dnd}|td|Drd} fd|D}tj|jjj|}jdupjtjk}||rDt|d trt||d<nd |D|d<njtj kojdu}|std |Dnj j jzd z jzjz jj t|d tr fd |D|d<n fd |D|d<|ddZ|dkr*ddl}||d|j|d<n/t%j|dt$j|d<nd|d<|Bj;t+jdr&j|d} | |d<|S)Nr@rrEc g|] }| Srrrys rr>z3DataCollatorForSeq2Seq.__call__..r|rc3K|]}|duV dSrrr<r@s rrz2DataCollatorForSeq2Seq.__call__..s&%H%Hetm%H%H%H%H%H%HrcPg|]"}fd|D#S)c(i|]\}}|k ||Srrrs rrz>DataCollatorForSeq2Seq.__call__...s$TTTAAOO1OOOrrrys rr>z3DataCollatorForSeq2Seq.__call__..s7mmmY`TTTTTTTmmmrrqFc:g|]}tj|ggSr)r concatenaters rr>z3DataCollatorForSeq2Seq.__call__...s&&W&W&Wur~ubk'B'B&W&W&Wrc34K|]}t|VdSrr)r<ls rrz2DataCollatorForSeq2Seq.__call__..1s(&>&>!s1vv&>&>&>&>&>&>rrcg|]D}dkr|jgt|z zznjgt|z z|zES)rrr<r@max_label_lengthrrs rr>z3DataCollatorForSeq2Seq.__call__..;s|'''"(722!8 9=MPSTYPZPZ=Z [[["56:JSQVZZ:WX[``'''rc Pg|]}dkrMtj|tjjgt |z ztjgnLtjtjjgt |z ztj|gS)rrC)rrrjrvrrhrs rr>z3DataCollatorForSeq2Seq.__call__..Bs'''"(722  % "$*A)BFVY\]bYcYcFc)dlnlt u u u ^ "$*A)BFVY\]bYcYcFc)dlnlt u u u %'''rrrC%prepare_decoder_input_ids_from_labels)rEdecoder_input_ids)r rr*r%rmrnror DO_NOT_PADrQr5 MAX_LENGTHrrr$rPrWrhrrjrr"r) rrr rEnon_labels_featuresr] no_padding max_paddingrPrr{rrs ` @@@rrzDataCollatorForSeq2Seq.__call__s"  !!0N '8A; 6 6WWH BLPXYZP[B[B[>>>>X>>>>ae  #%H%H%H%H%H"H"H Fmmmmdlmmm3 N L#6)    \U*Xdlo>X.X  ' hqk*5t<<X&*6llE(OO&W&WPV&W&W&WE(OO"lo.HHhT_dhMh FQ#f3&>&>v&>&>&>#>#>#>W[Wf *6)D,CCaG2312% $~: hqk*5t<<''''''&, '''E(OO''''''&,'''E(O$ 99Xt $ $ 0%% "',,uXek,"R"Rh"$(5?"("K"K"Kh"E(O   & $KLL'!% P PX]^fXg P h h ):E% & rr)rrrr3r r4rrrmrrrr rnrTrorvr rrrrrrsB'&&&E3:,0GTCZ/ )000!Jd !!!%)d )))""""NCVVVVVVrrc eZdZUdZeed<dZeed<dZeed<dZ e dzed <d Z e ed <d Z e ed <dZ edzed<dZeed<dZedzed<dZdZdZdeeeezeeefzdeeeffdZ d#dededzdedzdeeeffdZdeeeezeeefzdeeeffdZ d#dededzdedzdeeeffdZedejejeeefdejejedeejejeejejeffdZ ed ejejed!edefd"Z!dS)$DataCollatorForLanguageModelinga Data collator used for language modeling. Inputs are dynamically padded to the maximum length of a batch if they are not all of the same length. Args: tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]): The tokenizer used for encoding the data. mlm (`bool`, *optional*, defaults to `True`): Whether or not to use masked language modeling. If set to `False`, the labels are the same as the inputs with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked tokens and the value to predict for the masked token. whole_word_mask (`bool`, *optional*, defaults to `False`): Whether or not to mask whole words instead of individual tokens. mlm_probability (`float`, *optional*, defaults to 0.15): The probability with which to (randomly) mask tokens in the input, when `mlm` is set to `True`. mask_replace_prob (`float`, *optional*, defaults to 0.8): The probability with which masked tokens are replaced by the tokenizer's mask token (e.g., `[MASK]`). Defaults to 0.8, meaning 80% of the masked tokens will be replaced with `[MASK]`. Only works when `mlm` is set to `True`. random_replace_prob (`float`, *optional*, defaults to 0.1): The probability with which masked tokens are replaced by random tokens from the tokenizer's vocabulary. Defaults to 0.1, meaning 10% of the masked tokens will be replaced with random tokens. The remaining masked tokens (1 - mask_replace_prob - random_replace_prob) are left unchanged. Only works when `mlm` is set to `True`. pad_to_multiple_of (`int`, *optional*): If set, will pad the sequence to a multiple of the provided value. return_tensors (`str`): The type of Tensor to return. Allowable values are "np", or "pt". seed (`int`, *optional*): The seed to use for the random number generator for masking. If not provided, the global RNG will be used. For best performance, this data collator should be used with a dataset having items that are dictionaries or BatchEncoding, with the `"special_tokens_mask"` key, as returned by a [`PreTrainedTokenizer`] or a [`PreTrainedTokenizerFast`] with the argument `return_special_tokens_mask=True`. 1. Default Behavior: - `mask_replace_prob=0.8`, `random_replace_prob=0.1`. - Expect 80% of masked tokens replaced with `[MASK]`, 10% replaced with random tokens, and 10% left unchanged. 2. All masked tokens replaced by `[MASK]`: - `mask_replace_prob=1.0`, `random_replace_prob=0.0`. - Expect all masked tokens to be replaced with `[MASK]`. No tokens are left unchanged or replaced with random tokens. 3. No `[MASK]` replacement, only random tokens: - `mask_replace_prob=0.0`, `random_replace_prob=1.0`. - Expect all masked tokens to be replaced with random tokens. No `[MASK]` replacements or unchanged tokens. 4. Balanced replacement: - `mask_replace_prob=0.5`, `random_replace_prob=0.4`. - Expect 50% of masked tokens replaced with `[MASK]`, 40% replaced with random tokens, and 10% left unchanged. Note: The sum of `mask_replace_prob` and `random_replace_prob` must not exceed 1. If their sum is less than 1, the remaining proportion will consist of masked tokens left unchanged. r%TmlmFwhole_word_maskg333333?Nmlm_probability皙?mask_replace_probg?random_replace_probrorr seedc|jra|jjtd|j|jdks |jdkrtdt |j|_n|jrtd|j|jzdkrtd|jdks |jdkrtd|jdks |jdkrtd|jrS|jj stj d t|jdkr"tj d d|_d|_t |j|_t |j|_d|_ dS) NzThis tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.rrz*mlm_probability should be between 0 and 1.zmWhole word masking can only be used with mlm=True.If you want to use whole word masking, please set mlm=True.zHThe sum of mask_replace_prob and random_replace_prob should not exceed 1z,mask_replace_prob should be between 0 and 1.z.random_replace_prob should be between 0 and 1.zcWhole word masking depends on offset mapping which is only natively available with fast tokenizers.zbRandom token replacement is not supported with whole word masking. Setting mask_replace_prob to 1.)rr% mask_tokenrrrVrrris_fastwarningswarn UserWarning generator)rs r __post_init__z-DataCollatorForLanguageModeling.__post_init__s 8 ~(0 `#+t/Ca/G/G4K_bcKcKc !MNNN#()=#>#>D  ! N   !D$< )  y %)) 6*+&+,(!&t'=!>!>#()A#B#B rc|jdkr+ddl}||Stj|S)Nrr)r rP Generator manual_seedrrandom default_rng)rrrPs r get_generatorz-DataCollatorForLanguageModeling.get_generatorsM  $ & & LLL??$$0066 69((.. .rc8tjjdkr!||j|_dSddl}|jj }|d}t|||j|j z|_dS)N MainProcessr)zZWorker process information is not available for seeding the generator. This may be becausezZyou are using multiprocessing without using a PyTorch DataLoader. The `seed` parameter canzVonly be used when using multiprocessing with a PyTorch DataLoader. Please either use azAsingle process or use a PyTorch DataLoader with multiple workers.) mpcurrent_processnamerrrrPutilsdataget_worker_inforid)rrP worker_info error_strings r create_rngz*DataCollatorForLanguageModeling.create_rngs    $ 5 5!// ::DNNN LLL+*::<|d }|jj d |||jj k<||d <|S) Nrrr rorrospecial_tokens_maskoffset_mappingrrrEru)rrrrQrr*r%rorrrtorch_mask_tokensclonerrrr]rrrEs rrz*DataCollatorForLanguageModeling.torch_calls9 9 / OO    hqk7 + + 6RVRiEE 1(DN_c_vwwwE $ii(=tDD#3T:: 8 %262H2Hk"8K\j3I33 /E+ h;'--//F~*6@Dv!<<=$E(O rinputsrrc\ddl}|}||jj}| fd|D}jrPt|t|\}}| ||j }nEt||j r| n| ||j }| |d||j } jr)||| } d|| <|||jjj | z} jjj|| <jd ks jdkr||fSd jz } j| z } |||j| j | z| z} |t1j|j|jj }|| || <||fS) S Prepare masked tokens inputs/labels for masked language modeling. rNcHg|]}j|dST)already_has_special_tokensr%get_special_tokens_maskr<valrs rr>zEDataCollatorForLanguageModeling.torch_mask_tokens..(<###ad66sW[6\\###rrCvalue)rrur)rDr)rPrrrrrr_calc_word_ids_and_prob_maskto_numpyrWrrrQrR masked_fill_ bernoullir BoolTensor_whole_word_maskrr%convert_tokens_to_idsrrrrrU)rrrrrPrEprobability_matrixword_ids no_mask_maskmasked_indicesindices_replacedremaining_probrandom_replace_prob_scaledindices_random random_wordss` rrz1DataCollatorForLanguageModeling.torch_mask_tokenss  "ZZ d6JKK  &####hnhuhuhwhw###    %)%F%F(((3F*G*G&& "Hl!<< EJ<GGLL15<@@I#((***\\"5UZ\HH  '' C'@@@);t~VV[[]]   _"--d.C.CHn.].]^^N"& OOEJJv|T5KLLX\XfO g g l l n n  $(>#G#GHa#b#b  !Q & &$*Ba*G*G6> !T33&*%=%N" OOEJJv|5OPP\`\jO k k p p r r    }}S%8%8&,ejdhdr}ss !-n!=~v~rc*|jr|j|t|dtrt |j|d|j}ndt||j|ji}| dd}| dd}|j r(| |d||\|d<|d <n>tj |d}|jjd |||jjk<||d <|S) NrrrrrrrrrEru)rrrrQrr*r%rorrrnumpy_mask_tokensrcopyrrs rrz*DataCollatorForLanguageModeling.numpy_call[s7 9 / OO    hqk7 + + 6RVRiEE 1(DN_c_vwwwE $ii(=tDD#3T:: 8 %262H2Hk"8K\j3I33 /E+ hWU;/00F~*6@Dv!<<=$E(O rctj|}tj|jj}| fd|D}jr4t|t|\}}nOt|tj r| tntj |t}d||<jr;jd||j t}n?tjd||j t}jr||}d||<jrCjdj|j t|z} nGtjdj|j t|z} jj|| <jdks jdkr||fSdjz } j| z } jrjd| |j t|z| z} jdt/jtj| tj} ntjd| |j t|z| z} tjdt/jtj| tj} | || <||fS) rNcHg|]}j|dSrrr!s rr>zEDataCollatorForLanguageModeling.numpy_mask_tokens..r#rrCrrrru)lowhighrrD)rr9rrrrrr'r(rQrZastyperrrjrbinomialrr,rr% mask_token_idrintegersr count_nonzerorhr)rrrrrEr.r/r0r1r2r3r4r5r6s` rr8z1DataCollatorForLanguageModeling.numpy_mask_tokenszsWV\43GHH  &####hnhuhuhwhw###    %)%F%F(((3F*G*G&& "Hll 12:>>?#**4000X1>>>  ,-<( > s!^44Q8JQcQi4jjqqrvwwNNY//3EL^Ld/eellmqrrN   M!228^LLN"& > ''4+A 'UU\\]abbess    ""1d&<6<"PPWWX\]]`nn $(>#?  !Q & &$*Ba*G*G6> !T33&*%=%N" > ''+EFL'YY``aeff !##$   >22C//b6F~6V6V^`^f3LL  ""1&@v|"TT[[\`aa !##$  9,,C//b6F~6V6V^`^f-L".~v~roffsetscX|dddddf}|dddddf}tj|dd}d|dddf<tj|dd}d|dddf<|t}|||k|dkzz}tj|d}d||<|}||fS)z Map tokens to word ids and create mask of tokens to not mask. Tokens that are part of the same word will have the same word id and we will only set a mask probability for the first token of each word. Nrrrr)rrollr>rrcumsum) rCr token_starts token_endsprev_token_endsprev_token_special is_new_wordr/ prob_masks rr'z>++1PUgklUl0mn 9[q111(*$% L ""rr/maskct|}|dk}|dddddf|dddddfk|dddddfz|dddddfz}tj||dddddfzdS)z> Mask whole words based on word ids and mask. rNrr)r(rany)r/rM valid_ids same_words rr,z0DataCollatorForLanguageModeling._whole_word_masks ~~N aaaDj)Xaaaqqqj-AAYqqqRSRSRSUYzEZZ]fghghghjnpqpqpqgq]rr vi$qqq$z"22;;;;rNN)"rrrr3r r4rrrrrrVrrrorTr rrrrrr5rr6rrrrr8 staticmethodrrZr'r,rrrrrjs<<|'&&&C!OT!!!$(OUT\((("u"""!$$$$%)d )))NCD#*(((T///LLL.4S C$sCx.(H#IdSVX[S[n@ae===03d =SVY]S]= sCx====~4S C$sCx.(H#IdSVX[S[nD+/%) PPP!4ZPd P sCx PPPPd#BJuS#X78#OQzZ\ZdehZiOj# rz"*S/*BJrz#,GG H###\#: <2:bjo#> z/DataCollatorForSOP.__call__..3sBBBgW[)BBBrcg|] }|d S)token_type_idsrrks rr>z/DataCollatorForSOP.__call__..7sLLL'"23LLLrT) batch_first padding_valuecg|] }|d S)sentence_order_labelrrks rr>z/DataCollatorForSOP.__call__..;sRRRg'"89RRRr)rrEattention_maskrmrq)rPtorch.nn.utils.rnnrhrr% mask_tokensrrX) rrrPrhrrErrrmsop_label_listrqs rrzDataCollatorForSOP.__call__/s 333333BBBBB (DNCC ,0,<,LL8LLL%n$VZVdVqrrrRRRRR${{>::#,,$8    rrcddl}jjtd|}||jj}fd|D}| | ||j djj 6| jj}| |d|| }|}jj 6| jj} | | dd ||<|||jd  |z} jjj|| <|||jd  |z| z} |t'j|j|j} | | || <|||fS) z Prepare masked tokens inputs/labels/attention_mask for masked language modeling: 80% MASK, 10% random, 10% original. N-gram not applied yet. rNzThis tokenizer does not have a mask token which is necessary for masked language modeling. Remove the --mlm flag if you want to use this tokenizer.cHg|]}j|dSrrr!s rr>z2DataCollatorForSOP.mask_tokens..Vs<   ]`DN 2 23SW 2 X X   rrCr$r%g?rurg?)rPr%rrrrrrrr)rWrrreqrr*rVr-rrrU) rrrPrEr.r padding_maskr1rrattention_padding_maskr2r5r6s ` rrtzDataCollatorForSOP.mask_tokensFsI  > $ ,A  "ZZ d6JKK    djdqdqdsds    '' 5HPUPZ ([([cf'ggg > # /!99T^%@AAL  + +L + D D D);<<AACC)/0022 > # /%+YYt~/J%K%K "  ' '(>c ' J J J"&!??5::flC+H+HIINNPPSaa#'>#G#GHa#b#b FL#)F)FGGLLNNQ__csbss}}S%8%8&,ej}YY !-n!=~v~--rN) rrrr3rYr5r6rrrrrtrrrreres    d38n!5 $sCx.    .).#).%S# *>).).).).).).rrec8eZdZUdZeed<dZeed<dZe ed<dZ e ed<d e e e e zee e fzd ee e ffd Zd e e e e zee e fzd ee e ffd Zd e d ee e e e ffdZd e d ee e e e ffdZdS)*DataCollatorForPermutationLanguageModelingz Data collator used for permutation language modeling. - collates batches of tensors, honoring their tokenizer's pad_token - preprocesses batches for permutation language modeling with procedures specific to XLNet r%gUUUUUU?plm_probabilitymax_span_lengthrr rr+ct|dtr d|D}t||j}||\}}}}||||dS)Nrcg|] }|d Srjrrs rr>zIDataCollatorForPermutationLanguageModeling.torch_call..9991+999rr perm_masktarget_mappingrE)rQrrr%rrrr]rrrrEs rrz5DataCollatorForPermutationLanguageModeling.torch_callk hqk7 + + :99999H$Xt~>>484J4J54Q4Q1 >6#)~ioppprct|dtr d|D}t||j}||\}}}}||||dS)Nrcg|] }|d Srjrrs rr>zIDataCollatorForPermutationLanguageModeling.numpy_call..rrr)rQrrr%r8rs rrz5DataCollatorForPermutationLanguageModeling.numpy_callrrrcddl}jjtd|ddzdkrtd|}||jd|j}| |d|d|df|j }t|dD]}d}|d}||kr| dj dzd} t| jz } || | | z dzdz} d||| | | zf<|| z }||k||d||<|fd |D|j} || d jj6|jj} || d | | z}jj||<d ||<| |d|d|df|j }t|dD]N}||d}|d |ddzfdd}|||ddz}||dd}|||||zd ||ddf|d|dfk||z||<P||||fS)g The masked tokens to be predicted for a particular sequence are determined by the following algorithm: 0. Start from the beginning of the sequence by setting `cur_len = 0` (number of tokens processed so far). 1. Sample a `span_length` from the interval `[1, max_span_length]` (length of span of tokens to be masked) 2. Reserve a context of length `context_length = span_length / plm_probability` to surround span to be masked 3. Sample a starting point `start_index` from the interval `[cur_len, cur_len + context_length - span_length]` and mask tokens `start_index:start_index + span_length` 4. Set `cur_len = cur_len + context_length`. If `cur_len < max_len` (i.e. there are tokens remaining in the sequence to be processed), repeat from Step 1. rNThis tokenizer does not have a mask token which is necessary for permutation language modeling. Please add a mask token if you want to use this tokenizer.rrThis collator requires that sequence lengths be even to create a leakage-free perm_mask. Please see relevant comments in source code for details.rC)rcHg|]}j|dSrrr!s rr>zPDataCollatorForPermutationLanguageModeling.torch_mask_tokens..0 u u u^aT^ 3 3CTX 3 Y Y u u urr$r%rur)rPr%rrrrrrrrzerosrirrrrSrTr}eyerWrr)rrxrr@arangereshape transposerandpermflattenrU)rrrPrEr1rrcur_lenmax_len span_lengthcontext_length start_indexrry non_func_maskr perm_indexs` rrz $ ,N  ;;q>>A  " "A  FL!5:FFfkk!nnfkk!nnfkkRSnn%U]b]jkkv{{1~~&& : :AGkk!nnGG###mmAt/Ca/GNNSSUU !$[43G%G!H!H% n{6RUV6VX\(](](b(b(d(dd MNq+ k0I"IIJ>)G##!& &++a.. 9 9N1  #ll u u u uekereretet u u u*+   ##$7s#CCC > # /!99T^%@AAL  ' ' C ' @ @ @')<<= !%!=~"&KKQQQ PX]XeKff v{{1~~&& " "Afkk!nn55J#++RQ11D,EFFPPQRTUVVJ#ENN6;;q>>Q3F$G$GHJz';';Aq'A'ABBJ  # #^A%6$6q9I$I2 N N N ""FKKNNA#677:;M;MqRXR]R]^_R`R`Na;b;bbq!"IaLL{{}}iFFrcjjtd|jddzdkrtdt j|}t j|jdt}t j|jd|jd|jdftj }t|jdD]}d}|jd}||kr_tdj dz}t|jz } |td| |z dzz} d||| | |zf<|| z }||k_t j|jd||<t jfd|Dt} d|| <jj|jjk} d || <| | z} jj||<d ||<t j|jd|jd|jdftj }t|jdD]}t j|jd}|d |jddzfj}tj||j}d |||| |z<||jddf|d|jdfk||z||<|tj|||tjfS) rNrrrrrrCcHg|]}j|dSrrr!s rr>zPDataCollatorForPermutationLanguageModeling.numpy_mask_tokens..$rrr$rur)r%rrrrr9rrrrrirrrrTr}rrjrrrr@rrTrshufflerr>rh)rrrEr1rrrrrrrrryrrrs` rr8z $ ,N  <?Q ! # #A  q===6<?FLOV\RS_"U]_]ghhhv|A'' 8 8AGl1oGG##%a)=)ABB !$[43G%G!H!H%>K3ORS3S(T(TT MNq+ k0I"IIJ>)G##!#v|A 7 7N1   h u u u uekereretet u u u   /0*+ > # /!T^%@@L+.N< (')<<= !%!=~"&Hfl1ov|A QPXZXbccc v|A'' " "A6<?33J#++RaA1E,FGGIJ I  j ) ) )#--//J ACJq))M!,<< = ""FLOQ#788J6==QSQYCZCZZZrN)rrrr3r r4r}rVrrTr rr5rr6rrrrr8rrrr|r|rsu'&&&"OU"""OSNCq4S C$sCx.(H#IqdSVX[S[nqqqqq4S C$sCx.(H#IqdSVX[S[nqqqqaGaGc3S6H0IaGaGaGaGF_[_[c3S6H0I_[_[_[_[_[_[rr|c6eZdZdZdddddfd Zd dZxZS) DataCollatorWithFlatteninga Data collator used for padding free approach. Does the following: - concatenates the entire mini batch into single long sequence of shape [1, total_tokens] - uses `separator_id` to separate sequences within the concatenated `labels`, default value is -100 - no padding will be added, returns `input_ids`, `labels` and `position_ids` by default - optionally returns the kwargs contained in FlashAttentionKwargs - optionally returns seq_idx indicating which sequence each token belongs to Using `DataCollatorWithFlattening` will flatten the entire mini batch into single long sequence. Make sure your attention computation is able to handle it! TruF)return_position_ids separator_idreturn_flash_attn_kwargsreturn_seq_idxctj|i|||_||_||_||_hd|_hd|_ddh|_dS)N>rEr position_ids>rEseq_idxrr max_length_q max_length_k) rXrYrrrr _int_64_keys_batch_dim_keys _py_int_keys)rrrrrrZr[rs rrYz#DataCollatorWithFlattening.__init__fsp $)&)))#6 ((@%,CCCQQQ+^<rNc h||j}||j}d|dv}ggd}|jr|dgi|jr|dgi|jrdg}d}t |D]s\}|d} t| dr| } |dxx| z cc<|rI|d} t| dr| } |dxx|g| ddzz cc<n|dxx|g| ddzz cc<|jr7|dxxttt| z cc<|jr6|dxxfd tt| Dz cc<|jrH| |d t| zt|t| }u|jr|x|d <|d <|x|d <|d<|dkrddl} | j} | j} | j}n>|dkr%t$j} t$j} t$j}nt)d|d|D]5\}}||jvr|g}||jvr| |||jvr| n|||<6|S)NrEr)rrErrrrrcg|]}Srr)r<_rs rr>z7DataCollatorWithFlattening.__call__..s$L$L$LW$L$L$Lrr cu_seq_lens_q cu_seq_lens_krrrrz;return_tensors must be one of ("pt", "np"), return_tensors=z not supportedrC)r rrupdaterrrr"rr5rrappendrrPrWrhint32rrjrrYrrr)rrr ris_labels_providedr] cu_seq_lensrnsamplerrErPdata_clsdtype_64dtype_32rMr^rs @rrz#DataCollatorWithFlattening.__call__xsQ  !!0N  ,L%!4 B//  # / LL."- . . .   * LL)R ) ) )  ( #KJ(22 = =OGV{+Iy(++ /%,,.. +   ) +   ! B)68,,-#]]__FhL>F122J#>>hL>IabbM#AA' En%%%eC NN.C.C)D)DD%%%" Mi   $L$L$L$LeC NN6K6K$L$L$LL   , ="";r?S^^#CDDD S^^<<  ( G>I IE/ "U?%;O9O9OXXU]^^^a rrR)rrrr3rYrr\r]s@rrrSsn(!!& =======$>>>>>>>>rr)rr),multiprocessingrrcollections.abcrr dataclassesrrrtypingrr_rtokenization_utils_baser r r InputDataClassr5r6r DataCollatorr r*r/r1r-r.rlrtrTrrrrrrUrrZr(rer|rrrrrs--------!!!!!!======######n-.S#X>?  N N N N N N N N,55D$85RVWZ\_W_R`5555. ?????+?? ?6!$~*>!4S>!!!!H$~*>4S>@ 0000000 0f hhhhh):hh hV""#*""""J#*@ ?????$5?? ?D @@@@@@@ @F L<L<L<L<L<&7L<L< L<^  $$$$$#B$$ $&c2:c? O.O.O.O.O.8O.O. O.d ][][][][][1B][][ ][@ bbbbb!4bb bbbr