tjLUdZddlmZddlZddlmZmZddlmZddlm Z ddl m Z m Z m Z d d lmZd d lmZe jeZe d eGdde Ze d eGdde Ze d eGdde Ze Gdde ZGddeZGddeZGddeZGddeZGdd eZe d!Gd"d#eZe d$Gd%d&eZe d'Gd(d)eZ gd*Z!dS)+z5PyTorch DPR model for Open Domain Question Answering.) dataclassN)Tensornn)BaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringlogging) BertModel) DPRConfigz6 Class for outputs of [`DPRQuestionEncoder`]. ) custom_introceZdZUdZejed<dZeejdfdzed<dZ eejdfdzed<dS)DPRContextEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the context representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed contexts for nearest neighbors queries with questions embeddings. pooler_outputN. hidden_states attentions __name__ __module__ __qualname____doc__torch FloatTensor__annotations__rtuplerb/var/www/html/CA-Chatbot/venv/lib/python3.11/site-packages/transformers/models/dpr/modeling_dpr.pyrr(q$$$$:>M5*C/047>>>7;Je',-4;;;;;r rceZdZUdZejed<dZeejdfdzed<dZ eejdfdzed<dS)DPRQuestionEncoderOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, embeddings_size)`): The DPR encoder outputs the *pooler_output* that corresponds to the question representation. Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer. This output is to be used to embed questions for nearest neighbors queries with context embeddings. rN.rrrrr r!r$r$;r"r r$ceZdZUdZejed<dZejdzed<dZejdzed<dZ e ejdfdzed<dZ e ejdfdzed<dS) DPRReaderOutputa start_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the start index of the span for each passage. end_logits (`torch.FloatTensor` of shape `(n_passages, sequence_length)`): Logits of the end index of the span for each passage. relevance_logits (`torch.FloatTensor` of shape `(n_passages, )`): Outputs of the QA classifier of the DPRReader that corresponds to the scores of each passage to answer the question, compared to all the other passages. start_logitsN end_logitsrelevance_logits.rr) rrrrrrrr(r)rrrrr r!r&r&Ns####+/J!D(///15e'$.555:>M5*C/047>>>7;Je',-4;;;;;r r&ceZdZdZdS)DPRPreTrainedModelTN)rrr_supports_sdparr r!r+r+fsNNNr r+ceZdZdZdeffd Z ddededzdedzd edzd ed ed ed ee edfzfdZ e d e fdZ xZS) DPREncoder bert_modelconfigcpt|t|d|_|jjjdkrt d|j|_|jdkr.tj |jjj|j|_ | dS)NF)add_pooling_layerrz!Encoder hidden_size can't be zero) super__init__r r/r0 hidden_size ValueErrorprojection_dimrLinear encode_proj post_initselfr0 __class__s r!r4zDPREncoder.__init__ns    #FeDDD ? ! - 2 2@AA A$3   " "!y)?)KVMbccD  r NF input_idsattention_masktoken_type_ids inputs_embedsoutput_attentionsoutput_hidden_states return_dictreturn.c ||||||||} | d} | dddddf} |jdkr|| } |s| | f| ddzSt| | | j| jS)Nr>r?r@rArBrCrDrr )last_hidden_staterrr)r/r7r9rrr) r<r>r?r@rArBrCrDkwargsoutputssequence_output pooled_outputs r!forwardzDPREncoder.forwardys//))'/!5#"  "!*'1aaa0   " " ,,];;M B#]3gabbkA A)-'!/)     r cR|jdkr |jjS|jjjS)Nr)r7r9 out_featuresr/r0r5)r<s r!embeddings_sizezDPREncoder.embeddings_sizes*   " "#0 0%11r )NNNFFF)rrrbase_model_prefixrr4rboolrrrMpropertyintrP __classcell__r=s@r!r.r.ks$ y      )-(,'+"'%*!" " "  "  " } "  " #" "  $eFCK&8 8" " " " H2222X22222r r.creZdZdZdeffd Z ddedededzd ed ed ed ee ed fzfdZ xZ S)DPRSpanPredictorencoderr0c*t|t||_t j|jjd|_t j|jjd|_| dS)Nr r) r3r4r.rYrr8rP qa_outputs qa_classifierr:r;s r!r4zDPRSpanPredictor.__init__sq    !&)) )DL$@!DDYt|'CQGG r NFr>r?rArBrCrDrE.c ||n|dd\}} |||||||} | d} || } | dd\} }| d} |d}|| dddddf}| || } ||| }||}|s| ||f| ddzSt| ||| j | j S)Nr )r?rArBrCrDrr)dim)r'r(r)rr) sizerYr[splitsqueeze contiguousr\viewr&rr)r<r>r?rArBrCrDrI n_passagessequence_lengthrJrKlogitsr'r(r)s r!rMzDPRSpanPredictor.forwards;D:Oinn&6&6&6UbUgUgUiUijlkljlUm# O,, )'/!5#   "!*11#)<<r<#:#: j#++B//::<< ''++6688 --oaaaAAAg.FGG$((_EE __ZAA +00<< N *.>?'!""+M M%!-!/)     r )NFFF) rrrrQrr4rrRr&rrMrUrVs@r!rXrXs!y(,"'%*!, , , , } ,  , # , ,  5- -, , , , , , , , r rXc"eZdZUdZeed<dZdS)DPRPretrainedContextEncoder An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r0 ctx_encoderNrrrrrrrQrr r!riris0 %r ric"eZdZUdZeed<dZdS)DPRPretrainedQuestionEncoderrjr0question_encoderNrlrr r!rnrns0 *r rnc"eZdZUdZeed<dZdS)DPRPretrainedReaderrjr0span_predictorNrlrr r!rqrqs0 (r rqzf The bare DPRContextEncoder transformer outputting pooler outputs as context representations. ceZdZdeffd Ze ddedzdedzdedzdedzdedzd edzd edzd ee ed fzfd Z xZ S)DPRContextEncoderr0ct|||_t||_|dSN)r3r4r0r.rkr:r;s r!r4zDPRContextEncoder.__init__ sH     %f-- r Nr>r?r@rArBrCrDrE.c ^||n |jj}||n |jj}||n |jj}||t d||} n.||dd} nt d||jn|j} |(|tj| | n||jj k}|!tj | tj | }| |||||||} |s | ddSt| j| j| j S) aS input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRContextEncoder, DPRContextEncoderTokenizer >>> tokenizer = DPRContextEncoderTokenizer.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> model = DPRContextEncoder.from_pretrained("facebook/dpr-ctx_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ```NDYou cannot specify both input_ids and inputs_embeds at the same timer^5You have to specify either input_ids or inputs_embedsdevicedtyper{rGrrrr)r0rBrCrDr6r`r{rones pad_token_idzeroslongrkrrrr r<r>r?r@rArBrCrDrI input_shaper{rJs r!rMzDPRContextEncoder.forwardsd2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "#..**KK  &',,..ss3KKTUU U%.%:!! @T  !$ ;v66664;#;;   !"[EJvVVVN""))'/!5##   122; &!/w?Tahas    r NNNNNNN) rrrrr4r rrRrrrMrUrVs@r!rtrts  y$((,(,'+)-,0#'Y Y D=Y  Y  Y } Y  $; Y #TkY D[Y  !5#5 5Y Y Y ^Y Y Y Y Y r rtzh The bare DPRQuestionEncoder transformer outputting pooler outputs as question representations. ceZdZdeffd Ze ddedzdedzdedzdedzdedzd edzd edzd ee ed fzfd Z xZ S)DPRQuestionEncoderr0ct|||_t||_|dSrv)r3r4r0r.ror:r;s r!r4zDPRQuestionEncoder.__init__vsH     *6 2 2 r Nr>r?r@rArBrCrDrE.c ||n |jj}||n |jj}||n |jj}||t d|+||||} n.||dd} nt d||jn|j} |(|tj | | n||jj k}|!tj | tj | }| |||||||} |s | ddSt| j| j| j S) aj input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. To match pretraining, DPR input sequence should be formatted with [CLS] and [SEP] tokens as follows: (a) For sequence pairs (for a pair title+text for example): ``` tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP] token_type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1 ``` (b) For single sequences (for a question for example): ``` tokens: [CLS] the dog is hairy . [SEP] token_type_ids: 0 0 0 0 0 0 0 ``` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import DPRQuestionEncoder, DPRQuestionEncoderTokenizer >>> tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> model = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base") >>> input_ids = tokenizer("Hello, is my dog cute ?", return_tensors="pt")["input_ids"] >>> embeddings = model(input_ids).pooler_output ``` Nrxr^ryrzr|rGrr~)r0rBrCrDr6%warn_if_padding_and_no_attention_maskr`r{rrrrrror$rrrrs r!rMzDPRQuestionEncoder.forward}sd2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !$ ;v66664;#;;   !"[EJvVVVN''))'/!5#(   122; '!/w?Tahas    r r) rrrrr4r rrRr$rrMrUrVs@r!rrps  y$((,(,'+)-,0#'Z Z D=Z  Z  Z } Z  $; Z #TkZ D[Z  "E&#+$6 6Z Z Z ^Z Z Z Z Z r rzE The bare DPRReader transformer outputting span predictions. ceZdZdeffd Ze d dedzdedzdedzdedzdedzd edzd ee ed fzfd Z xZ S) DPRReaderr0ct|||_t||_|dSrv)r3r4r0rXrrr:r;s r!r4zDPRReader.__init__sH     .v66 r Nr>r?rArBrCrDrE.c ||n |jj}||n |jj}||n |jj}||t d|+||||}n.||dd}nt d||jn|j} |tj || }| ||||||S)a input_ids (`tuple[torch.LongTensor]` of shapes `(n_passages, sequence_length)`): Indices of input sequence tokens in the vocabulary. It has to be a sequence triplet with 1) the question and 2) the passages titles and 3) the passages texts To match pretraining, DPR `input_ids` sequence should be formatted with [CLS] and [SEP] with the format: `[CLS] [SEP] [SEP] ` DPR is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than the left. Indices can be obtained using [`DPRReaderTokenizer`]. See this class documentation for more details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(n_passages, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Examples: ```python >>> from transformers import DPRReader, DPRReaderTokenizer >>> tokenizer = DPRReaderTokenizer.from_pretrained("facebook/dpr-reader-single-nq-base") >>> model = DPRReader.from_pretrained("facebook/dpr-reader-single-nq-base") >>> encoded_inputs = tokenizer( ... questions=["What is love ?"], ... titles=["Haddaway"], ... texts=["'What Is Love' is a song recorded by the artist Haddaway"], ... return_tensors="pt", ... ) >>> outputs = model(**encoded_inputs) >>> start_logits = outputs.start_logits >>> end_logits = outputs.end_logits >>> relevance_logits = outputs.relevance_logits ``` Nrxr^ryrz)rArBrCrD) r0rBrCrDr6rr`r{rrrr) r<r>r?rArBrCrDrIrr{s r!rMzDPRReader.forwards!b2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !"Z FCCCN""  '/!5# #   r )NNNNNN) rrrrr4r rrRr&rrMrUrVs@r!rrs y$((,'+)-,0#'L L D=L  L } L  $; L #Tk L D[L  5- -L L L ^L L L L L r r)rtrir+rnrqrr)"r dataclassesrrrrmodeling_outputsrmodeling_utilsrutilsr r r bert.modeling_bertr configuration_dprr get_loggerrloggerrr$r&r+r.rXrirnrqrtrr__all__rr r!rs<;!!!!!! ::::::------ +*****((((((  H % %   < < < <