tj^F U d Z ddlZddlZddlmZ ddlmZ ddlmZ ddl m
Z
m Z ddl m
Z
dd lmZ dd
lmZ dd lmZ dd lmZmZmZmZmZ dd
lmZ ddl m Z ddl m Z ddl m!Z!m"Z"m#Z#m$Z$ ddl%m&Z& ddl'm(Z( ddl)m*Z* ddl+m,Z, ej- e. Z/er e Z ee0e0dz f e1d< n eg d Z2 e j3 e2 e e!e Z4de0fdZ5 G d d Z6d dgZ7dS ) zAutoProcessor class. N) OrderedDict)
TYPE_CHECKING )PreTrainedConfig) get_class_from_dynamic_moduleresolve_trust_remote_code)FeatureExtractionMixin)ImageProcessingMixin)ProcessorMixin)TOKENIZER_CONFIG_FILE)FEATURE_EXTRACTOR_NAMEPROCESSOR_NAMEVIDEO_PROCESSOR_NAME cached_filelogging)BaseVideoProcessor )_LazyAutoMapping)PROCESSOR_MAPPING_NAMES)CONFIG_MAPPING_NAMES
AutoConfigmodel_type_to_module_name!replace_list_option_in_docstrings)AutoFeatureExtractor)AutoImageProcessor)
AutoTokenizer)AutoVideoProcessorr ) )aimv2
CLIPProcessor)diffusion_gemmaGemma4Processor)edgetam
Sam2Processor) glm4v_moeGlm4vProcessor)granite_speech_plusGraniteSpeechProcessor)groupvitr )hubertWav2Vec2Processor)inkling_mm_modelInklingProcessor)lasr_ctc
LasrProcessor) lasr_encoderr. )
metaclip_2r )mistral3PixtralProcessor)zmm-grounding-dinoGroundingDinoProcessor) modernvbertIdefics3Processor) moonshiner* ) parakeet_ctcParakeetProcessor)
parakeet_rnntr8 ) parakeet_tdtr8 )qwen3_5Qwen3VLProcessor) qwen3_5_moer< ) qwen3_vl_moer< )sam3_lite_text
Sam3Processor)sewr* )zsew-dr* )t5gemma2Gemma3Processor)t5gemma2_encoderrC ) unispeechr* )z
unispeech-satr* )vipllavaLlavaProcessor)zwav2vec2-conformerr* )wavlmr*
class_namec t j D ]S\ }}| |v rJt | }t j d| d } t || c S # t $ r Y Ow xY wTt j D ]}t |dd | k r|c S t j d }t || rt || S d S )N.ztransformers.models__name__ transformers) r itemsr importlib
import_modulegetattrAttributeErrorPROCESSOR_MAPPING_extra_contentvalueshasattr)rI module_name
processorsmodule processor main_modules f/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/auto/processing_auto.pyprocessor_class_from_namer] Z s #:#@#B#B Z # #3K@@K ,->->->@UVVF
vz22222!
$ '5<<>> 9j$ / /: = = >
).99K{J'' 0{J/// 4s A
A$#A$c h e Zd ZdZd Ze ee d Ze dd Z
dS )
AutoProcessora
This is a generic processor class that will be instantiated as one of the processor classes of the library when
created with the [`AutoProcessor.from_pretrained`] class method.
This class cannot be instantiated directly using `__init__()` (throws an error).
c t d )Nz}AutoProcessor is designed to be instantiated using the `AutoProcessor.from_pretrained(pretrained_model_name_or_path)` method.)OSError)selfs r\ __init__zAutoProcessor.__init__z s
_
c dd } dd }dd< d}d}d}fd|D }| dddd t |t fi |} | Pt j |fi \ }
} |
d
}d |
d i v r|
d d }|8t |t fi |} | Qt j |fi \ }
} |
d
d }d |
d i v r|
d d }| ft |t fi |} | Qt j |fi \ }
} |
d
d }d |
d i v r|
d d }| ht |t fi |} | S|Qt j |fi \ }
} |
d
d }d |
d i v r|
d d }|t |t fi |}
|
yt! |
d
5 }t# j | }
ddd n # 1 swxY w Y |
d
d }d |
d i v r|
d d }|r t' |t( st+ j |fd|i}t/ |d
d }t1 |d rd |j v r
|j d }n# t4 $ r Y nw xY w|t7 | }|du}|dupt9 | t: v }|o4|pt: t9 | j d }|s7|r5d|v r | d d }nd}tC ||||| }|rL|rJ|sHtE ||fi } dd } |# |j |fd|iS | |j |fd|iS t9 | t: v r%t; t9 | j |fi S tH tJ tL tN fD ]$} |j |fd|ic S # tP $ r Y !w xY wt5 d| d )a3
Instantiate one of the processor classes of the library from a pretrained model vocabulary.
The processor class to instantiate is selected based on the `model_type` property of the config object (either
passed as an argument or loaded from `pretrained_model_name_or_path` if possible):
List options
Params:
pretrained_model_name_or_path (`str` or `os.PathLike`):
This can be either:
- a string, the *model id* of a pretrained feature_extractor hosted inside a model repo on
huggingface.co.
- a path to a *directory* containing a processor files saved using the `save_pretrained()` method,
e.g., `./my_model_directory/`.
cache_dir (`str` or `os.PathLike`, *optional*):
Path to a directory in which a downloaded pretrained model feature extractor should be cached if the
standard cache should not be used.
force_download (`bool`, *optional*, defaults to `False`):
Whether or not to force to (re-)download the feature extractor files and override the cached versions
if they exist.
proxies (`dict[str, str]`, *optional*):
A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',
'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request.
token (`str` or *bool*, *optional*):
The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
when running `hf auth login` (stored in `~/.huggingface`).
revision (`str`, *optional*, defaults to `"main"`):
The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
identifier allowed by git.
return_unused_kwargs (`bool`, *optional*, defaults to `False`):
If `False`, then this function returns just the final feature extractor object. If `True`, then this
functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary
consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of
`kwargs` which has not been used to update `feature_extractor` and is otherwise ignored.
trust_remote_code (`bool`, *optional*, defaults to `False`):
Whether or not to allow for custom models defined on the Hub in their own modeling files. This option
should only be set to `True` for repositories you trust and in which you have read the code, as it will
execute code present on the Hub on your local machine.
kwargs (`dict[str, Any]`, *optional*):
The values in kwargs of any keys which are feature extractor attributes will be used to override the
loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is
controlled by the `return_unused_kwargs` keyword parameter.
Passing `token=True` is required when you want to use a private model.
Examples:
```python
>>> from transformers import AutoProcessor
>>> # Download processor from huggingface.co and cache.
>>> processor = AutoProcessor.from_pretrained("facebook/wav2vec2-base-960h")
>>> # If processor files are in a directory (e.g. processor was saved using *save_pretrained('./test/saved_model/')*)
>>> # processor = AutoProcessor.from_pretrained("./test/saved_model/")
```configNtrust_remote_codeT
_from_auto) cache_dirforce_downloadproxiestokenrevisionlocal_files_only subfolder repo_type
user_agentc * i | ]}|v || S rs ).0keykwargss r\
z1AutoProcessor.from_pretrained.. s% ] ] ]3sV\}}c6#;}}}rd F) _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errorsprocessor_classr_ auto_mapzutf-8)encodingz
transformers.z--r
code_revisionz!Unrecognized processing class in z. Can't instantiate a processor, a tokenizer, an image processor, a video processor or a feature extractor for this model. Make sure the repository contains the files of at least one of those processing classes.))popupdater r r get_processor_dictgetr
r
get_image_processor_dictr r get_video_processor_dictr get_feature_extractor_dictr openjsonload
isinstancer r from_pretrainedrQ rV r|
ValueErrorr] typerS
__module__
startswithsplitr r register_for_auto_classr r r r Exception)cls pretrained_model_name_or_pathrv rf rg r{ processor_auto_map_hub_valid_kwargscached_file_kwargsprocessor_config_file config_dict_preprocessor_config_filetokenizer_config_filereaderhas_remote_codehas_local_codeexplicit_local_code
upstream_repoklasss ` r\ r z AutoProcessor.from_pretrained s D Hd++ "JJ':D A A #| !
^ ] ] ]:K ] ] ]!!499>;@
!,,I> p p]o p p , + >?\ g g`f g gNK )oo.? @ @O +//*b"A"AAA%0%<_%M" "'2-/E( (I[( ( $ (3!5!NOl!w!wpv!w!w Q"-//2CT"J"J"kooj"&E&EEE)4Z)@ )Q& (/+613G, ,K], ,( ,7%7%P5& &9?& &NK '2oo6G&N&NO&+//*b*I*III-8-D_-U*'/+613I, ,M_, ,( ,7O 0f >
ZZ0021 / ))) 2 8 8 > >q A
$
9!#@.Racp! ! l 0 l9L l ;"$A EK O
?D11A 3 3 5 5 52?2- ARV\
(2?2- ARV\
&\\ .
.
.$T&\\2BC`kkdjkk k $%79KMab E
,u,1 EVZ`
g0M
g
g
g
s7 (I I
I
A K4 4
L LQ--
Q:9Q:Fc @ t | || dS )a
Register a new processor for this class.
Args:
config_class ([`PreTrainedConfig`]):
The configuration corresponding to the model to register.
processor_class ([`ProcessorMixin`]): The processor to register.
)exist_okN)rS register) config_classr{ r s r\ r zAutoProcessor.register_ s% ""< 8"TTTTTrd N)F) rL r __qualname____doc__rc classmethodr r r staticmethodr rs rd r\ r_ r_ r s
&&'>??[
[
@? [[
z U U U \ U U Urd r_ rS )8r rO r collectionsr typingr configuration_utilsr dynamic_module_utilsr r feature_extraction_utilsr image_processing_utilsr
processing_utilsr tokenization_pythonr utilsr
r r r r video_processing_utilsr auto_factoryr
auto_mappingsr configuration_autor r r r feature_extraction_autor image_processing_autor tokenization_autor video_processing_autor
get_loggerrL loggerstr__annotations__ MISSING_PROCESSOR_MAPPING_NAMESr rS r] r_ __all__rs rd r\ r s # # # # # # 4 3 3 3 3 3 \ \ \ \ \ \ \ \ > > > > > > : : : : : : . . . . . . 8 8 8 8 8 8 g g g g g g g g g g g g g g 8 8 8 8 8 8 * * * * * * 2 2 2 2 2 2 : 9 9 9 9 9 5 5 5 5 5 5 , , , , , , 5 5 5 5 5 5
H % % (D =HKMM[cDj 9IIII&1k
"' "'#F #"#BCCC$$%9;RSS # 0wU wU wU wU wU wU wU wUt
0rd