tjNtdZddlmZddlmZddlmZmZmZddl m Z e GddeZ dgZ dS) z Processor class for MarkupLM. ) TensorType)ProcessorMixin) BatchEncodingPaddingStrategyTruncationStrategy)auto_docstringc eZdZdZfdZe ddedeezezdeeze zd e dzd e d e dzd edzd edzdedededededee zdzde fdZ xZS)MarkupLMProcessorTcLt||dS)N)super__init__)selffeature_extractor tokenizer __class__s n/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/markuplm/processing_markuplm.pyr zMarkupLMProcessor.__init__s$ *I66666NFadd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturnc |jrL|td|||td||}|d}|d}n$|td||td||jrt|tr|g}|jdid||n|d ||ndd|d |d |d |d |d| d| d| d| d| d|d|d|d|d|d||}|S)a html_strings (`str` or `list[str]`, *optional*): Raw HTML strings to parse and process. When `parse_html=True` (default), these strings are parsed to extract nodes and xpaths automatically. If provided, `nodes`, `xpaths`, and `node_labels` should not be provided. Required when `parse_html=True`. nodes (`list[list[str]]`, *optional*): Pre-extracted HTML nodes as a list of lists, where each inner list contains the text content of nodes for a single document. Required when `parse_html=False`. Should not be provided when `parse_html=True`. xpaths (`list[list[str]]`, *optional*): Pre-extracted XPath expressions corresponding to the nodes. Should be a list of lists with the same structure as `nodes`, where each XPath identifies the location of the corresponding node in the HTML tree. Required when `parse_html=False`. Should not be provided when `parse_html=True`. node_labels (`list[list[int]]`, *optional*): Labels for the nodes, typically used for training or fine-tuning tasks. Should be a list of lists with the same structure as `nodes`, where each label corresponds to a node. Optional and only used when `parse_html=False`. questions (`str` or `list[str]`, *optional*): Question strings for question-answering tasks. When provided, the tokenizer processes questions as the first sequence and nodes as the second sequence (text_pair). If a single string is provided, it is converted to a list to match the batch dimension of the parsed HTML. NzDMake sure to pass HTML strings in case `parse_html` is set to `True`zUPlease don't pass nodes, xpaths nor node labels in case `parse_html` is set to `True`nodesxpathsz@You have passed HTML strings but `parse_html` is set to `False`.zIMake sure to pass nodes and xpaths in case `parse_html` is set to `False`text text_pair node_labelsrrrrrrrrrrrr r!r") parse_html ValueErrorr isinstancestrr)r html_stringsr%r&r) questionsrrrrrrrrrrrr r!r"kwargsfeaturesencoded_inputss r__call__zMarkupLMProcessor.__call__s\ ? n# !ghhh F$6+:Q k--l;;HW%Eh'FF' !cddd} !lmmm  T_ )S)) (&K '   '3 (4ee$ 6 $  21  G  "z "z 6  21 #8"7 #8"7 '@&? (B'A $:#9 (-! "G# $*>'  ,r)NNNNNTFNNrNNNFFFFTN)__name__ __module__ __qualname__r+r rboolr.rrintrrr4 __classcell__)rs@rr r sxJ77777#'056:!%)--1-1*/+0',#26)ZZ!Zo-Z3J!33Z$JZZ $JZ $d{Z $d{Z$(Z %)!Z"!%#Z$%Z&'Z(j(4/)Z, -ZZZ^ZZZZZrr N) __doc__ file_utilsrprocessing_utilsrtokenization_utils_baserrrutilsrr __all__r*rrrAs%$$$$$......YYYYYYYYYY######aaaaaaaaH  r