tj%dZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z ddl mZmZmZmZmZddlmZerddlZejeZdd iZd Zd Zed Gdde ZdgZdS)z!Tokenization class for Pop2Piano.N) BatchFeature) AddedToken BatchEncodingPaddingStrategyPreTrainedTokenizerTruncationStrategy) TensorTypeis_pretty_midi_availableloggingrequires_backendsto_numpy)requiresvocabz vocab.jsonc4||z }|t||}|SN)minnumbercutoff_time_idx current_idxs r/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/pop2piano/tokenization_pop2piano.pytoken_time_to_noter&s'6K"+77 c||9||}||kr*|}|||||g|dkrdn|}|||<n|||<|S)Nr)append) rcurrent_velocitydefault_velocitynote_onsets_readyrnotes onset_idx offset_idx onsets_readys rtoken_note_to_noter$.sr ,%f- { " "$J LL)Z9IJ K K K#3q#8#844kL(4 f %$/&! Lr) pretty_miditorch)backendsceZdZdZddgZeZ d5fd Zed Z d Z d e de fdZ d6de fdZdejde de de fdZ d7dejdejde de de f dZd8dejdededzfdZd9dejdejd e fd!Zd8d"ed#edzdeefd$Z d:deje ejzd%edzd&e dzdefd'Z d:deje ejzd%edzd&e dzdefd(Z d;deje ejze e ejzd+eeze zd,eezezd&e dzd-e dzd.edzd/ee!zdzd0edefd1Z" dz/Pop2PianoTokenizer.__init__..ss>>>A1>>>r) unk_token eos_token pad_token bos_tokenr7) isinstancestrrrnum_barsopenjsonloadencoderitemsdecodersuper__init__) selfrrrBr<r=r>r?kwargsfile __class__s rrJzPop2PianoTokenizer.__init__[sJTT]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir 0  %   +$9T??DL + + + + + + + + + + + + + + +?>););)=)=>>>           sC**C.1C.c*t|jS)z-Returns the vocabulary size of the tokenizer.)lenrFrKs r vocab_sizezPop2PianoTokenizer.vocab_size}s4<   rc0t|jfi|jS)z(Returns the vocabulary of the tokenizer.)dictrFadded_tokens_encoderrQs r get_vocabzPop2PianoTokenizer.get_vocabsDL>>D$=>>>rtoken_idreturnc|j||jd}|d}d|ddt |d}}||gS)a? Decodes the token ids generated by the transformer into notes. Args: token_id (`int`): This denotes the ids generated by the transformers to be converted to Midi tokens. Returns: `List`: A list consists of token_type (`str`) and value (`int`). _TOKEN_TIME_Nr)rHgetr<splitjoinint)rKrWtoken_type_value token_typevalues r_convert_id_to_tokenz'Pop2PianoTokenizer._convert_id_to_tokenst <++H6T6T6TUU+11#66HH%5abb%9::C@PQR@S+AA+EEJ.; 66$ 07F 6{{avQ??? =I rr beatstepc|dn|}|||||}|||||}|S)al Converts tokens to Midi. This method calls `relative_batch_tokens_ids_to_notes` method to convert batch tokens to notes then uses `notes_to_midi` method to convert them to Midi. Args: tokens (`numpy.ndarray`): Denotes tokens which alongside beatstep will be converted to Midi. beatstep (`np.ndarray`): We get beatstep from feature extractor which is also used to get Midi. beat_offset_idx (`int`, *optional*, defaults to 0): Denotes beat offset index for each note in generated Midi. bars_per_batch (`int`, *optional*, defaults to 2): A parameter to control the Midi output generation. cutoff_time_idx (`int`, *optional*, defaults to 12): Denotes the cutoff time index for each note in generated Midi. Nr)rirjrkr) offset_sec)ry notes_to_midi)rKrir{rjrkrr midis r!relative_batch_tokens_ids_to_midiz4Pop2PianoTokenizer.relative_batch_tokens_ids_to_midis^0 /6!!O77+)+ 8   !!%h>W!XX rNrnc fd|D}|}d}dttdjDdzD}g}|D]e\} } | dkr | dkrnS| dkrt| ||}-| d kr| }6| d krt | |j||| }Wt d t|D]P\} } | I|| dz} nt|| dz} t|| }| | || jgQt|dkrgStj |}|d d dfdz|d d dfz}|| }|S)a Converts relative tokens to notes which will then be used to create Pretty Midi objects. Args: tokens (`numpy.ndarray`): Relative Tokens which will be converted to notes. start_idx (`float`): A parameter which denotes the starting index. cutoff_time_idx (`float`, *optional*): A parameter used while converting tokens to notes. c:g|]}|Sr7)rd)r8rgrKs r zCPop2PianoTokenizer.relative_tokens_ids_to_notes..s'FFFe**511FFFrrcg|]}dSrr7r8is rrzCPop2PianoTokenizer.relative_tokens_ids_to_notes..sdddaTdddrc3@K|]}|dVdS)NOTEN)endswith)r8r9s r zBPop2PianoTokenizer.relative_tokens_ids_to_notes..s.4^4^AQZZ5G5G4^4^4^4^4^4^rr\ TOKEN_SPECIALrerTOKEN_VELOCITY TOKEN_NOTE)rrrrrr zToken type not understood!N)rpsumrFrr$r ValueError enumeratemaxrrPrrarrayargsort)rKrirnrwordsrrrr rbrpitch note_onsetcutoffr" note_orders` rrqz/Pop2PianoTokenizer.relative_tokens_ids_to_notessGFFFvFFF dd54^4^QUQ]4^4^4^1^1^ab1b+c+cddd"' ? ? J_,,Q;;E|++0!?P[ ///#)  |++*!%5%)%:&7 + !!=>>>!*+>CCtruncation_strategy max_lengthc `t|dgt|dtjr2t jd|Ddd}t j|tj }|ddddf }dt|d zD}|D]A\}}} } || | | g|| | dgBg} d} t|D]\} }t|dkr| || d |D]r\} } t!| dk} | | kr+| } | || d | || d st| }|t"jkr |r||kr|jd| ||z |d |\} }}t)d| iS)a This is the `encode_plus` method for `Pop2PianoTokenizer`. It converts the midi notes to the transformer generated token ids. It only works on a single batch, to process multiple batches please use `batch_encode_plus` or `__call__` method. Args: notes (`numpy.ndarray` of shape `[sequence_length, 4]` or `list` of `pretty_midi.Note` objects): This represents the midi notes. If `notes` is a `numpy.ndarray`: - Each sequence must have 4 values, they are `onset idx`, `offset idx`, `pitch` and `velocity`. If `notes` is a `list` containing `pretty_midi.Note` objects: - Each sequence must have 4 attributes, they are `start`, `end`, `pitch` and `velocity`. truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`], *optional*): Indicates the truncation strategy that is going to be used during truncation. max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). Returns: `BatchEncoding` containing the tokens ids. r%rcBg|]}|j|j|j|jgSr7)rrrr)r8 each_notes rrz2Pop2PianoTokenizer.encode_plus..s+nnn[d)/9=)/9CUVnnnrrmNr-cg|]}gSr7r7rs rrz2Pop2PianoTokenizer.encode_plus..s555555rr\rerr)idsnum_tokens_to_removerr*r7)r r@r%rrrrreshaperoundastypeint32rrprrrPrhr`r DO_NOT_TRUNCATEtruncate_sequencesr)rKr rrrL max_time_idxtimesonsetoffsetrrrirrtime total_lenr[s r encode_pluszPop2PianoTokenizer.encode_plusksy6 $000 eAh 0 1 1 Hnnhmnnngb!nn  &&rx00QQQU|'')) 55U= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~file_utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. Returns: `BatchEncoding` containing the token_ids. rr)rrrrrNT)r rr)rrrrrrr7) r@rrndarrayndimlist"_get_padding_truncation_strategiesrrpad)rKr rrrrrrrrL is_batchedpadding_strategyrr*s r__call__zPop2PianoTokenizer.__call__s9Z)35"*(E(EeUZ1__:V[\]V^`dKeKe ElDDkE !!1 E E  E E A-z6  ,A,IDDOd !..$7% II)($7% IHH $!1"7)  rfeature_extractor_output return_midic 8tt|dot|dot|d}|s&|djddkrtd|rt |ddddfdk|djdks(|djd|d jdkrEtd |jdd |djdd |d jd|djd|jdkr1td |djdd|jdnf|djddks|d jddkr8td|djdd|d jdd|r/t j|ddddfdkd}n|jdg}g}g}d}t|D]\} } ||| } | dddt jt j| t|j kddzf} |d| } |d | } |r|d| }|d| }| dt jt j|dkddz} | dt jt j|dkddz} t| } t| } t| } | | | |j |j dzdz}|jdjD]C}|xj| dz c_|xj| dz c_||D|||| dzz }|rt'||dSt'd|iS)a; This is the `batch_decode` method for `Pop2PianoTokenizer`. It converts the token_ids generated by the transformer to midi_notes and returns them. Args: token_ids (`Union[np.ndarray, torch.Tensor]`): Output token_ids of `Pop2PianoConditionalGeneration` model. feature_extractor_output (`BatchFeature`): Denotes the output of `Pop2PianoFeatureExtractor.__call__`. It must contain `"beatstep"` and `"extrapolated_beatstep"`. Also `"attention_mask_beatsteps"` and `"attention_mask_extrapolated_beatstep"` should be present if they were returned by the feature extractor. return_midi (`bool`, *optional*, defaults to `True`): Whether to return midi object or not. Returns: If `return_midi` is True: - `BatchEncoding` containing both `notes` and `pretty_midi.pretty_midi.PrettyMIDI` objects. If `return_midi` is False: - `BatchEncoding` containing `notes`. r+attention_mask_beatsteps$attention_mask_extrapolated_beatstep beatstepsrr\zattention_mask, attention_mask_beatsteps and attention_mask_extrapolated_beatstep must be present for batched inputs! But one of them were not present.Nextrapolated_beatstepzbLength mistamtch between token_ids, beatsteps and extrapolated_beatstep! Found token_ids length - z, beatsteps shape - z$ and extrapolated_beatsteps shape - z!Found attention_mask of length - z but token_ids of length - zLength mistamtch of beatsteps and extrapolated_beatstep! Since attention_mask is not present the number of examples must be 1, But found beatsteps length - z", extrapolated_beatsteps length - .rm)rir{rkr)r pretty_midi_objectsr )boolhasattrshaperrrrwhererrr`r=rrrBrr rrrr)rKr*rrattention_masks_present batch_idx notes_listpretty_midi_objects_listrnrtend_idxeach_tokens_idsrrrrpretty_midi_objectnotes r batch_decodezPop2PianoTokenizer.batch_decodePsF8#' ,.> ? ? Z02LMM Z02XYY# # ' +CK+P+VWX+Y\]+]+]H  # ,-=>qqq!tDIJJ+K8>qABB+K8>qA+,CDJ1MNN!w*3/!*<wwRjkvRwR}~SAww:RSj:k:qrs:tww ((89?BioVWFXXX ]8PQa8b8hij8k]]IRIXYZI[]]Y)5;A>!CC+,CDJ1MQRRR D4L[4Y4_`a4bDDG_`wGxG~@GADDD # -!9:J!KAAAqD!QUV!VWWXYZII"+,I #%  ' 22# %# %NE7' '(9:O-aaa1r26"(?VYZ^ZhViViCi:j:jkl:m3n3nqr3r1r.rsO0=eDI$<=T$UV[$\ !' +CD^+_`e+f(7O:884&&^rx8PTU8U/V/VWX/Y(Z(Z]^(^&^_ (=XbfRX&Ja&OPPQRSTTWXXX)%'77O ++I$,-B$C$C !!%!G!G&.#}!%!2a 7 "H"" +6q9? ( ( il* IaL(!!$'''' $ + +,> ? ? ? 1 $II  i :Nf!g!ghh hgz2333r)r,r-r.r/r0r1)re)rr-rzr)r)NN)FNNNNNT)T)&__name__ __module__ __qualname____doc__model_input_namesrvocab_files_namesrJpropertyrRrVr`rrdrhrrrryrfloatrqr~rAtuplerr%rr rrrrrr rrr __classcell__)rNs@rr)r)=s 2%&67)        D!!X!???#S#T####$NNcNNNN * ** *  ****` !!    *         H::2::%:bgjnbn::::x2:QT@!!c!C$J!Z_`cZd!!!!4:>!% E4E4zD!122E40$6E4$J E4  E4E4E4E4T:>!% %E%EzD!122%E0$6%E$J %E  %E%E%E%ET166:!%)--126uuzD!122T${?O:P5QQuo-u3J!33 u $J u  $J u $d{uj(4/uu uuuuv! w4w4#/w4 w4w4w4w4w4w4w4w4rr))rrDrnumpyrrfeature_extraction_utilsrtokenization_pythonrrrrr utilsr r r r rutils.import_utilsrr% get_loggerrrrrr$r)__all__r7rrr s^(' 444444vvvvvvvvvvvvvv______________******  H % % \     +,,,I 4I 4I 4I 4I 4,I 4I 4-,I 4X r