tjCddlZddlZddlZddlmZddlmZmZm Z ddl m Z m Z m Z mZmZer ddlZddlmZmZGdd e Zeed Gd d e ZdS)N)GenerationConfig)add_end_docstringsis_torch_availablerequires_backends)ArgumentHandlerDatasetPipelinePipelineExceptionbuild_pipeline_init_args),MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPING_NAMES0MODEL_FOR_TABLE_QUESTION_ANSWERING_MAPPING_NAMESceZdZdZddZdS)%TableQuestionAnsweringArgumentHandlerzB Handles arguments for the TableQuestionAnsweringPipeline Nc t|dddl}|td|]t|tr/|d|d|g}nt|t rt|dkrtd|Dstdd|D|dd|dd|}ntd |d d tt|tst|tj r|Std t|d ||d g}|D]R}t|d|js5|dtd||d|d<S|S)Npandasrz(Keyword argument `table` cannot be None.querytablec3@K|]}t|tVdSN) isinstancedict.0ds m/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/pipelines/table_question_answering.py zATableQuestionAnsweringArgumentHandler.__call__..-s,>>1:a..>>>>>>z:Keyword argument `table` should be a list of dict, but is c34K|]}t|VdSr)typers rrzATableQuestionAnsweringArgumentHandler.__call__../s-UmUmbcVZ[\V]V]UmUmUmUmUmUmrzIf keyword argument `table` is a list of dictionaries, each dictionary should have a `table` and `query` key, but only dictionary has keys z `table` and `query` keys.zZInvalid input. Keyword argument `table` should be either of type `dict` or `list`, but is ))rrzTable cannot be None.)rr ValueErrorrrgetlistlenallkeysr types GeneratorTyper! DataFrame)selfrrkwargspdtqa_pipeline_inputstqa_pipeline_inputs r__call__z.TableQuestionAnsweringArgumentHandler.__call__s2 $))) =GHH H ]%&& 599W+=+=+IeiiX_N`N`Nl',g##E4(( SZZ!^^>>>>>>>$oUmUmglUmUmUmoo8<<((4qg9N9N9Z*/''$vJOPQ(--//vvv$E7)C)C$zRWY^YlGmGm$  )u++))) .3U#C#C"D "5 X X 092<HH X%g.6$%<===.0ll;Mg;V.W.W"7+""r)NN)__name__ __module__ __qualname____doc__r1rrrrs2-#-#-#-#-#-#rrT) has_tokenizerceZdZdZdZdZdZdZdZdZ e dZ e ffd Z dZd Zfd Zdd Zdd ZddZdZxZS)TableQuestionAnsweringPipelinea Table Question Answering pipeline using a `ModelForTableQuestionAnswering`. This pipeline is only available in PyTorch. Unless the model you're using explicitly sets these generation parameters in its configuration files (`generation_config.json`), the following default values will be used: - max_new_tokens: 256 Example: ```python >>> from transformers import pipeline >>> oracle = pipeline(model="google/tapas-base-finetuned-wtq") >>> table = { ... "Repository": ["Transformers", "Datasets", "Tokenizers"], ... "Stars": ["36542", "4512", "3934"], ... "Contributors": ["651", "77", "34"], ... "Programming language": ["Python", "Python", "Rust, Python and NodeJS"], ... } >>> oracle(query="How many stars does the transformers repository have?", table=table) {'answer': 'AVERAGE > 36542', 'coordinates': [(0, 1)], 'cells': ['36542'], 'aggregator': 'AVERAGE'} ``` Learn more about the basics of using a pipeline in the [pipeline tutorial](../pipeline_tutorial) This tabular question answering pipeline can currently be loaded from [`pipeline`] using the following task identifier: `"table-question-answering"`. The models that this pipeline can use are models that have been fine-tuned on a tabular question answering task. See the up-to-date list of available models on [huggingface.co/models](https://huggingface.co/models?filter=table-question-answering). z table,queryTF)max_new_tokensc tjdi|||_tj}|t ||t|j j ddot|j j dd|_ t|j j drdnd|_ dS)Naggregation_labelsnum_aggregation_labelstapasr6)super__init__ _args_parserrcopyupdatercheck_model_typegetattrmodelconfig aggregatehasattrr!)r, args_parserr-mapping __class__s rrAz'TableQuestionAnsweringPipeline.__init__}s""6"""'BGIICDDD g&&& !24H$OO T[ J 7U U  'tz'8:NOOYGGUY rc |jdi|S)Nr6)rG)r,inputss rbatch_inferencez.TableQuestionAnsweringPipeline.batch_inferencestz##F###rc g}g}d}|djd}|d|j}|d|j}|d|j}d} t|D]h} |W| dddf} t j| } || } t| jdD]} | dddf| }| dddf| dz }| dddf| dz }|dkr&|dkr |dkrt|||f| | <tj |  tj |j| dddf<|| }|| }|| } ||d|d| d }|j}|jr||j||tj| }|j| tj|jjz}t1jt4t7|D]\} }| dddf| }| dddf| dz }| dddf| dz }|dkr)|dkr#|dkr||f|fd D}jtjt=|d}|js|fn#|tjt=|dfS) z Inference used for models that need to process sequences in a sequential fashion, like the SQA models which handle conversational query related to a table. N input_idsrattention_masktoken_type_idsrr)rRrSrT)logitscri|]3}|tj|dk4S)g?)nparraymean)rkeycoords_to_probss r zGTableQuestionAnsweringPipeline.sequential_inference..s<hhhQTC/#*>!?!?!D!D!F!F!Lhhhr)shapetodevicerangerX zeros_likecpunumpytolistinttorch from_numpyr!longrG unsqueezerVrIappendlogits_aggregation distributions Bernoulliprobsfloat32 collections defaultdictr% enumeratesqueezecattuple)r,rO all_logitsall_aggregations prev_answers batch_sizerRrSrTtoken_type_ids_exampleindexprev_labels_example model_labelsi segment_idcol_idrow_idinput_ids_exampleattention_mask_exampleoutputsrVdist_per_token probabilitiespcolrow logits_batchr\s @rsequential_inferencez3TableQuestionAnsweringPipeline.sequential_inferences?   K(.q1 ;'**4;77  0144T[AA 0144T[AA!%:&&. i. iE'&&|1!455NNA!71!=!D!D!F!Fq!IJ3AAAqD9@@BB1EIF3AAAqD9@@BB1EIF{{v{{zQ*-lFF;K.L*M*M Q/4/? /M/M/R/RSXS]/^/^/a/abfbm/n/n&qqq!t, )% 0 %3E%: "%3E%: "jj+55a885??BB5??BB!G ^F~ D ''(BCCC   f % % %"0::&:IIN*03I3N3Nu}3]3]3`3`$+44M*5d;;O!-"7"7"9"9"@"@"B"BCC : :13AAAqD9@@BB1E ,QQQT299;;A>B,QQQT299;;A>B!88qZ1__#S#J/66q999hhhhXghhhLLyz!2!2A66 &*no <SXYiSjSjlmInIn:oorc|j|i|}tj|fi|}t|dkr|dS|S)a Answers queries according to a table. The pipeline accepts several types of inputs which are detailed below: - `pipeline(table, query)` - `pipeline(table, [query])` - `pipeline(table=table, query=query)` - `pipeline(table=table, query=[query])` - `pipeline({"table": table, "query": query})` - `pipeline({"table": table, "query": [query]})` - `pipeline([{"table": table, "query": query}, {"table": table, "query": query}])` The `table` argument should be a dict or a DataFrame built from that dict, containing the whole table: Example: ```python data = { "actors": ["brad pitt", "leonardo di caprio", "george clooney"], "age": ["56", "45", "59"], "number of movies": ["87", "53", "69"], "date of birth": ["7 february 1967", "10 june 1996", "28 november 1967"], } ``` This dictionary can be passed in as such, or can be converted to a pandas DataFrame: Example: ```python import pandas as pd table = pd.DataFrame.from_dict(data) ``` Args: table (`pd.DataFrame` or `Dict`): Pandas DataFrame or dictionary that will be converted to a DataFrame containing all the table values. See above for an example of dictionary. query (`str` or `list[str]`): Query or list of queries that will be sent to the model alongside the table. sequential (`bool`, *optional*, defaults to `False`): Whether to do inference sequentially or as a batch. Batching is faster, but models like SQA require the inference to be done sequentially to extract relations within sequences, given their conversational nature. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, `str` or [`TapasTruncationStrategy`], *optional*, defaults to `False`): Activates and controls truncation. Accepts the following values: - `True` or `'drop_rows_to_fit'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate row by row, removing rows from the table. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). Return: A dictionary or a list of dictionaries containing results: Each result is a dictionary with the following keys: - **answer** (`str`) -- The answer of the query given the table. If there is an aggregator, the answer will be preceded by `AGGREGATOR >`. - **coordinates** (`list[tuple[int, int]]`) -- Coordinates of the cells of the answers. - **cells** (`list[str]`) -- List of strings made up of the answer cell values. - **aggregator** (`str`) -- If the model has an aggregator, this returns the aggregator. rr)rBr@r1r&)r,argsr-pipeline_inputsresultsrMs rr1z'TableQuestionAnsweringPipeline.__call__s[V,$+T>>>#/997;7M 34)dj)LLLLOLLG)5RYZZ rc|d}|d|d}jdkrqjrw|dd\}}j|||}|\}}fdt |Djjjfdt |D} n/|d}j||}|d}ii} g} t |D]\} } fd | D} | d }| | d }|d | z| fd | Dd }|r||d<| |t|dkrtdjj dn&dj|dD} t| dkr| n| dS)Nrrrr?rcFi|]\}}|jjj|Sr6)rGrHr=)rrpredr,s rr]z>TableQuestionAnsweringPipeline.postprocess.._s/wwwQXQRTXq$*"3"Ft"Lwwwrc:i|]\}}|k ||dzS)z > r6)rrr aggregatorsno_agg_label_indexs rr]z>TableQuestionAnsweringPipeline.postprocess..bs;&&&29!T[_cu[u[uA{1~-[u[u[urrc*g|]}j|Sr6iatr coordinaters r z>TableQuestionAnsweringPipeline.postprocess..ms MMM::.MMMrrz, c*g|]}j|Sr6rrs rrz>TableQuestionAnsweringPipeline.postprocess..ss RRR ei 3RRRr)answer coordinatescells aggregatorzTable question answeringz Empty answercg|]}d|iS)rr6)rrs rrz>TableQuestionAnsweringPipeline.postprocess..|swwwf&)wwwrT)skip_special_tokensr)r!rIrconvert_logits_to_predictionsrsrGrHno_aggregation_label_indexr$joinrkr&r name_or_path batch_decode)r,rrOrrV logits_agg predictionsanswer_coordinates_batchagg_predictionsaggregators_prefixanswersr|rrraggregator_prefixrrrrs` @@@r postprocessz*TableQuestionAnsweringPipeline.postprocessVsV~.g& * 9  ~ (%,RaR[" "nJJ6SY[eff ( %'"G&/0H&I&I ' '"{MMMMMMM(__UB77 $6$:$:5"$E$E!/$))E2B2BB#.RRRRkRRR 6+5F<(v&&&&6{{a'(BDJD[]klll xw8S8ST[qu8S8v8vwwwGg,,**ww :r)NNN)TN)F)r2r3r4r5default_input_names_pipeline_calls_generate_load_processor_load_image_processor_load_feature_extractor_load_tokenizerr_default_generation_configrrArPrr1rrrr __classcell__)rMs@rr9r9Ms+  D(#O!#O!1!1"""$I#H#J#J Z Z Z Z Z Z$$$ApApApFPPPPPd5555& "(;(;(;(;(;(;(;rr9)rqr)rdrX generationrutilsrrrbaser r r r r rgmodels.auto.modeling_autorrrr9r6rrrsl )))))) baaaaaaaaaaaaaLLL 2#2#2#2#2#O2#2#2#j,,4@@@AAp;p;p;p;p;Xp;p;BAp;p;p;r