tjddlZddlZddlZddlmZmZddlmZddlZddl m Z ddl m Z ddl mZddlmZmZd d lmZmZmZd d lmZejeZeGd d ZGddeZGdde ZdS)N) dataclassfield)Enum)FileLock)Dataset)PreTrainedTokenizerBase)check_torch_load_is_safelogging)!glue_convert_examples_to_featuresglue_output_modesglue_processors) InputFeaturesceZdZUdZedddejziZe e d<eddiZ e e d<ed dd i Z e e d <ed ddi Zee d<dZdS)GlueDataTrainingArgumentsz Arguments pertaining to what data we are going to input our model for training and eval. Using `HfArgumentParser` we can turn this class into argparse arguments to be able to specify them on the command line. helpz"The name of the task to train on: z, )metadata task_namezUThe input data dir. Should contain the .tsv files (or other data files) for the task.data_dirzThe maximum total input sequence length after tokenization. Sequences longer than this will be truncated, sequences shorter will be padded.)defaultrmax_seq_lengthFz1Overwrite the cached training and evaluation setsoverwrite_cachecB|j|_dSN)rlowerselfs ]/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/data/datasets/glue.py __post_init__z'GlueDataTrainingArguments.__post_init__<s--//N)__name__ __module__ __qualname____doc__rjoinrkeysrstr__annotations__rrintrboolr!r"r rr"sUV-QTXT]T]^r^m^r^t^tTuTu-u$vwwwIswwwEqrHc % Q NC"E)\ ]OT00000r"rceZdZdZdZdZdS)SplittraindevtestN)r#r$r%r0r1r2r-r"r r/r/@s E C DDDr"r/c eZdZUeed<eed<eeed<dej dfdede de dzdeezdedzf d Z d Z d efd Zd ZdS) GlueDatasetargs output_modefeaturesN tokenizer limit_lengthmode cache_dirc Ztjdt||_t |j|_t|j|_t|tr, t|}n#t$rtdwxYwtj||n|jd|jd|jjd|jd|j}|j}|jdvr%|jjdvr|d|dc|d<|d<||_|d z}t/|5tj|rx|jsqt5j} t7t9j|d |_t> d |d t5j| z n3t> d|j|tj!kr |j"|j} nO|tj#kr |j$|j} n|j%|j} | | d|} tM| ||j||j|_t5j} t9j'|j|t> d|dt5j| z ddddddS#1swxYwYdS)NaThis dataset will be removed from the library soon, preprocessing should be handled with the Hugging Face Datasets library. You can have a look at this example script for pointers: https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.pyzmode is not a valid split namecached__)mnlizmnli-mm)RobertaTokenizerXLMRobertaTokenizer BartTokenizerBartTokenizerFastr z.lockT) weights_onlyz"Loading features from cached file z [took %.3f s]z'Creating features from dataset file at ) max_length label_listr6z!Saving features into cached file z [took z.3fz s])(warningswarn FutureWarningr5rr processorrr6 isinstancer)r/KeyErrorospathr'rvalue __class__r#r get_labelsrGrexistsrtimer torchloadr7loggerinfor1get_dev_examplesr2get_test_examplesget_train_examplesr save) rr5r8r9r:r;cached_features_filerG lock_pathstartexampless r __init__zGlueDataset.__init__Ks   u      (8::,T^< dC  A AT{ A A A?@@@ A "w||".IIDM hdj h h9#6#? h h$BU h hX\Xf h h  ^..00 >0 0 0Y5H5QV 6 6 ,6a=*Q- (JqM:a=$)72 i   w~~233 D>t}MMHHUZ''#~?? NNHH#~@@OOH+'  6H A#2) $ 0 !!!   4=*>??? q8LqqUYU^U`U`chUhqqqq;                  s- A;;BG L  L$'L$c*t|jSr)lenr7rs r __len__zGlueDataset.__len__s4=!!!r"returnc|j|Sr)r7)ris r __getitem__zGlueDataset.__getitem__s}Qr"c|jSr)rGrs r rRzGlueDataset.get_labelss r")r#r$r%rr*r)listrr/r0r r+rardrhrRr-r"r r4r4Fs ####=!!!! $(!K $ HH'H+HDj H Ek H : HHHHT"""     r"r4)rNrTrH dataclassesrrenumrrUfilelockrtorch.utils.datartokenization_utils_baser utilsr r processors.gluer rrprocessors.utilsr get_loggerr#rWrr/r4r-r"r rts (((((((( $$$$$$>>>>>>66666666cccccccccc,,,,,,  H % % 0000000 0:D VVVVV'VVVVVr"