o sh @sFddlZddlZddlZddlmZGdddZGdddeZdS)N) InputExamplec@s6eZdZdZddddejdddfdd Zd d d Zd S) STSDataReadera1Reads in the STS dataset. Each line contains two sentences (s1_col_idx, s2_col_idx) and one label (score_col_idx) Default values expects a tab separated file with the first & second column the sentence pair and third column the score (0...1). Default config normalizes scores from 0...5 to 0...1 rr Tc Cs:||_||_||_||_||_||_||_||_| |_dS)N) dataset_folder score_col_idx s1_col_idx s2_col_idx delimiterquotingnormalize_scores min_score max_score selfrr r r r r rrrri/var/www/html/alpaca_bot/venv/lib/python3.10/site-packages/sentence_transformers/readers/STSDataReader.py__init__s  zSTSDataReader.__init__c Cs tj|j|}|drtj|dddnt|dda}tj||j |j d}g}t |D]?\}}t ||j } |jrE| |j|j|j} ||j} ||j} |t|t|| | g| d|dkrkt||krkn q,Wd |SWd |S1swY|S) zJfilename specified which data split to use (train.csv, dev.csv, test.csv).z.gzrtutf8)encodingzutf-8)r r )guidtextslabelrN)ospathjoinrendswithgzipopencsvreaderr r enumeratefloatr rrrr r appendrstrlen) rfilename max_examplesfilepathfIndataexamplesidrowscores1s2rrr get_examples$s2     zSTSDataReader.get_examplesN)r)__name__ __module__ __qualname____doc__r" QUOTE_NONErr4rrrrrs rcs4eZdZdZddddejdddffdd ZZS) STSBenchmarkDataReaderzReader especially for the STS benchmark dataset. There, the sentences are in column 5 and 6, the score is in column 4. Scores are normalized from 0...5 to 0...1 rrTrc s"tj||||||||| d dS)N) rr r r r r rrr)superrr __class__rrr@s  zSTSBenchmarkDataReader.__init__)r5r6r7r8r"r9r __classcell__rrr>rr:;sr:)r"r rrrr:rrrrs  3