5 r " S S\R> 5 r! " S S\RD 5 r# " S S\R> 5 r$ " S S\R> 5 r% " S S\R> 5 r& " S S\R> 5 r' " S S \R> 5 r( " S S \R> 5 r) " S S \R> 5 r* " S! S"\R> 5 r+ " S# S$\R> 5 r,\ " S% S&\5 5 r-\ " S' S(\-5 5 r.\ " S) S*\-5 5 r/\" S+S,9 " S- S.\-5 5 r0\ " S/ S0\-5 5 r1\ " S1 S2\-5 5 r2\ " S3 S4\-5 5 r3/ S5Qr4g)6zPyTorch SqueezeBert model. N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss )ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutput QuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)auto_docstringlogging )SqueezeBertConfigc 6 ^ \ rS rSrSrU 4S jrSS jrSrU =r$ )SqueezeBertEmbeddings- zGConstruct the embeddings from word, position and token_type embeddings.c v > [ TU ] 5 [ R " UR UR
UR S9U l [ R " UR UR
5 U l [ R " UR UR
5 U l [ R " UR UR S9U l [ R " UR 5 U l U R% S[& R( " UR 5 R+ S5 SS9 g )N) padding_idxeps position_ids)r F)
persistent)super__init__r Embedding
vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNorm hidden_sizelayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandselfconfig __class__s l/var/www/html/shao/venv/lib/python3.13/site-packages/transformers/models/squeezebert/modeling_squeezebert.pyr" SqueezeBertEmbeddings.__init__0 s
!||F,=,=v?T?Tbhbubu v #%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`" f&8&8f>S>STzz&"<"<=
ELL)G)G H O OPW Xej
c Ub UR 5 nOUR 5 S S nUS nUc U R S S 2S U24 nUc8 [ R " U[ R U R R
S9nUc U R
U5 nU R U5 nU R U5 nXG- U- n U R U 5 n U R U 5 n U $ )Nr r dtypedevice) sizer r3 zeroslongr@ r' r) r+ r, r1 )
r7 input_idstoken_type_idsr
inputs_embeds input_shape
seq_lengthr) r+
embeddingss
r: forward SqueezeBertEmbeddings.forward@ s #..*K',,.s3K ^
,,Q ^ J "8;PP
^^J/
\\*-
r< )r, r1 r) r+ r' )NNNN __name__
__module__ __qualname____firstlineno____doc__r" rJ __static_attributes__
__classcell__r9 s @r: r r - s Q
r< r c 2 ^ \ rS rSrSrU 4S jrS rSrU =r$ )
MatMulWrapperY z
Wrapper for torch.matmul(). This makes flop-counting easier to implement. Note that if you directly call
torch.matmul() in your code, the flop counter will typically ignore the flops of the matmul.
c " > [ TU ] 5 g N)r! r" )r7 r9 s r: r" MatMulWrapper.__init___ s
r< c . [ R " X5 $ )a
:param inputs: two torch tensors :return: matmul of these tensors
Here are the typical dimensions found in BERT (the B is optional) mat1.shape: [B, , M, K]
mat2.shape: [B, , K, N] output shape: [B, , M, N]
)r3 matmul)r7 mat1mat2s r: rJ MatMulWrapper.forwardb s ||D''r< rL rT s @r: rV rV Y s
( (r< rV c ( \ rS rSrSrSS jrS rSrg)SqueezeBertLayerNormm z
This is a nn.LayerNorm subclass that accepts NCW data layout and performs normalization in the C dimension.
N = batch C = channels W = sequence length
c @ [ R R XUS9 g )N)normalized_shaper )r r, r" )r7 r- r s r: r" SqueezeBertLayerNorm.__init__t s
dc Jr< c UR SSS5 n[ R R X5 nUR SSS5 $ )Nr r )permuter r, rJ )r7 xs r: rJ SqueezeBertLayerNorm.forwardw s;
IIaA LL )yyAq!!r< r` N)g-q=)rM rN rO rP rQ r" rJ rR r` r< r: rb rb m s K"r< rb c 2 ^ \ rS rSrSrU 4S jrS rSrU =r$ )ConvDropoutLayerNorm} z0
ConvDropoutLayerNorm: Conv, Dropout, LayerNorm
c > [ TU ] 5 [ R " XSUS9U l [ U5 U l [ R " U5 U l g Nr in_channels out_channels kernel_sizegroups) r! r" r Conv1dconv1drb layernormr/ r1 )r7 cincoutru dropout_probr9 s r: r" ConvDropoutLayerNorm.__init__ s@
iiCPQZ`a -d3zz,/ r< c t U R U5 nU R U5 nX2- nU R U5 nU$ rY rw r1 rx )r7
hidden_states input_tensorrj s r: rJ ConvDropoutLayerNorm.forward s8 KK
& LLO
NN1 r< r~ rL rT s @r: rm rm } s 0 r< rm c 2 ^ \ rS rSrSrU 4S jrS rSrU =r$ )ConvActivation z"
ConvActivation: Conv, Activation
c t > [ TU ] 5 [ R " XSUS9U l [
U U l g rp )r! r" r rv rw r
act)r7 ry rz ru r r9 s r: r" ConvActivation.__init__ s/
iiCPQZ`a #;r< c F U R U5 nU R U5 $ rY )rw r )r7 rj outputs r: rJ ConvActivation.forward s Q xx r< )r rw rL rT s @r: r r s
r< r c D ^ \ rS rSrSU 4S jjrS rS rS rS rSr U =r
$ ) SqueezeBertSelfAttention c n > [ TU ] 5 X!R - S:w a [ SU SUR S35 eUR U l [ X!R - 5 U l U R U R
- U l [ R " X"SUS9U l [ R " X"SUS9U l
[ R " X"SUS9U l [ R " UR 5 U l [ R " SS9U l [# 5 U l [# 5 U l g )
z
config = used for some things; ignored for others (work in progress...) cin = input channels = output channels
groups = number of groups to use in conv1d layers
r zcin (z6) is not a multiple of the number of attention heads ()r rq r dimN)r! r" num_attention_heads
ValueErrorintattention_head_size
all_head_sizer rv querykeyvaluer/ attention_probs_dropout_probr1 SoftmaxsoftmaxrV matmul_qk
matmul_qkv)r7 r8 ry q_groupsk_groupsv_groupsr9 s r: r" !SqueezeBertSelfAttention.__init__ s
++ +q 0 u RSYSmSmRnnop
$*#=#= #&s-G-G'G#H ! 5 58P8P PYY3aX`a
99AV^_YY3aX`a
zz&"E"EF zzb) & '/r< c UR 5 S U R U R UR 5 S 4nUR " U6 nUR SSSS5 $ )zg
- input: [N, C, W]
- output: [N, C1, W, C2] where C1 is the head index, and C2 is one head's contents
r r r r rh )rA r r viewri r7 rj new_x_shapes r: transpose_for_scores-SqueezeBertSelfAttention.transpose_for_scores s[
vvx{D$<$V>VXYX^X^X`acXde
FFK yyAq!$$r< c UR 5 S U R U R UR 5 S 4nUR " U6 nU$ )zg
- input: [N, C, W]
- output: [N, C1, C2, W] where C1 is the head index, and C2 is one head's contents
r r )rA r r r r s r: transpose_key_for_scores1SqueezeBertSelfAttention.transpose_key_for_scores sK
vvx{D$<$V>VXYX^X^X`acXde
FFK r< c UR SSSS5 R 5 nUR 5 S U R UR 5 S 4nUR " U6 nU$ )z-
- input: [N, C1, W, C2]
- output: [N, C, W]
r r r rh )ri
contiguousrA r r r s r: transpose_output)SqueezeBertSelfAttention.transpose_output sZ
IIaAq ! , , .vvx{D$6$6 D
FFK r< c U R U5 nU R U5 nU R U5 nU R U5 nU R U5 nU R U5 n U R Xx5 n
U
[ R " U R 5 - n
X- n
U R U
5 n U R U 5 n U R X5 n U R U 5 n SU 0n
U( a XS' U
$ )z
expects hidden_states in [N, C, W] data layout.
The attention_mask data layout is [N, W], and it does not need to be transposed.
context_layerattention_score)
r r r r r r mathsqrtr r r1 r r )r7 r attention_maskoutput_attentionsmixed_query_layermixed_key_layermixed_value_layer query_layer key_layer value_layerr attention_probsr results r: rJ SqueezeBertSelfAttention.forward s !JJ} 5 ((=1 JJ} 5//0AB 11/B //0AB ..@)DIId6N6N,OO): ,,7 ,,7 E
--m<
!=1 (7$ %
r< )
r r r1 r r r r r r r )r r r ) rM rN rO rP r" r r r rJ rR rS rT s @r: r r s! *0%! !r< r c . ^ \ rS rSrU 4S jrS rSrU =r$ )SqueezeBertModule c > [ TU ] 5 UR nUR nUR nUR n[ XUR
UR UR S9U l [ X#UR UR S9U l [ X4UR UR S9U l [ XEUR" UR S9U l g)aP
- hidden_size = input chans = output chans for Q, K, V (they are all the same ... for now) = output chans for
the module
- intermediate_size = output chans for intermediate layer
- groups = number of groups for all layers in the BertModule. (eventually we could change the interface to
allow different groups for different layers)
)r8 ry r r r )ry rz ru r{ )ry rz ru r N)r! r" r- intermediate_sizer r r r attentionrm post_attention_groupsr0 post_attentionr intermediate_groups
hidden_act intermediate
output_groupsr )r7 r8 c0c1c2c3r9 s r: r" SqueezeBertModule.__init__ s
%
%
1FOOfoo`f`o`o
3F$@$@vOiOi
+r6C]C]cictct u*F$8$8vGaGa
r< c U R XU5 nUS nU R XQ5 nU R U5 nU R Xv5 nSU0n U( a US U S' U $ )Nr feature_mapr )r r r r )
r7 r r r attattention_outputpost_attention_outputintermediate_output layer_output output_dicts
r: rJ SqueezeBertModule.forward su nn]^ [ TU ] 5 TR TR :X d S5 e[ R
" U4S j[
TR 5 5 5 U l g )NzIf you want embedding_size != intermediate hidden_size, please insert a Conv1d layer to adjust the number of channels before the first SqueezeBertModule.c 3 8 ># U H n[ T5 v M g 7frY )r ).0_r8 s r: .SqueezeBertEncoder.__init__... s #gGf!$5f$=$=Gfs ) r! r" r% r- r
ModuleListrangenum_hidden_layerslayersr6 s `r: r" SqueezeBertEncoder.__init__% sW
$$(:(::
2
: mm#guVMeMeGf#gg r< c Uc SnO#UR S 5 [ U5 :X a SnOSnUSL d S5 eUR SSS5 nU( a SOS nU( a SOS n U R H] n
U( a+ UR SSS5 nX4-
nUR SSS5 nU
R XU5 n U S nU( d MU XS 4-
n M_ UR SSS5 nU( a X4-
nU( d [ S
XU 4 5 5 $ [
XU S 9$ ) NTFzAhead_mask is not yet supported in the SqueezeBert implementation.r rh r r` r r c 3 , # U H oc M Uv M
g 7frY r` )r vs r: r -SqueezeBertEncoder.forward..[ s h$Vq$Vs )last_hidden_stater
attentions)countlenri r rJ tupler ) r7 r r head_maskr output_hidden_states return_dicthead_mask_is_all_noneall_hidden_statesall_attentionslayerr s r: rJ SqueezeBertEncoder.forward0 s6 $( !
__T
"c)n
4$( !$) !$,q.qq, &--aA6
"6BD 0d[[E# - 5 5aA >
!%55! - 5 5aA >
== HYZL ( 7M 0A#B"DD ! &--aA6
!1 1 h]~$Vhh h +Yg
r< )r )NNFFTr rT s @r: r r $ s$ h ".
.
r< r c . ^ \ rS rSrU 4S jrS rSrU =r$ )SqueezeBertPooleria c > [ TU ] 5 [ R " UR UR 5 U l [ R " 5 U l g rY )r! r" r Linearr- denseTanh
activationr6 s r: r" SqueezeBertPooler.__init__b s9
YYv 1 163E3EF
'')r< c \ US S 2S4 nU R U5 nU R U5 nU$ )Nr )r r )r7 r first_token_tensor
pooled_outputs r: rJ SqueezeBertPooler.forwardg s6 +1a4 0
#56
6
r< )r r r rT s @r: r r a s $
r< r c . ^ \ rS rSrU 4S jrS rSrU =r$ )"SqueezeBertPredictionHeadTransformip c p > [ TU ] 5 [ R " UR UR 5 U l [
UR [ 5 ( a [ UR U l
OUR U l
[ R " UR UR S9U l g )Nr )
r! r" r r r- r
isinstancer strr
transform_act_fnr, r. r6 s r: r" +SqueezeBertPredictionHeadTransform.__init__q s~
YYv 1 163E3EF
f'' - -$*6+<+<$=D !$*$5$5D ! f&8&8f>S>STr< c l U R U5 nU R U5 nU R U5 nU$ rY )r r
r, r7 r s r: rJ *SqueezeBertPredictionHeadTransform.forwardz s4
=1
--m<
}5
r< )r, r r
r rT s @r: r r p s U r< r c 8 ^ \ rS rSrU 4S jrSS jrS rSrU =r$ )SqueezeBertLMPredictionHeadi c H > [ TU ] 5 [ U5 U l [ R
" UR UR SS9U l [ R " [ R " UR 5 5 U l U R U R l g )NF)bias)
r! r" r transformr r r- r$ decoder Parameterr3 rB r r6 s r: r" $SqueezeBertLMPredictionHead.__init__ sm
;FC yy!3!3V5F5FUS LLV->->!?@ !II r< c : U R U R l g rY )r r r7 s r: _tie_weights(SqueezeBertLMPredictionHead._tie_weights s II r< c J U R U5 nU R U5 nU$ rY )r r r
s r: rJ #SqueezeBertLMPredictionHead.forward s$ }5
]3
r< )r r r )returnN) rM rN rO rP r" r rJ rR rS rT s @r: r r s && r< r c . ^ \ rS rSrU 4S jrS rSrU =r$ )SqueezeBertOnlyMLMHeadi c B > [ TU ] 5 [ U5 U l g rY )r! r" r predictionsr6 s r: r" SqueezeBertOnlyMLMHead.__init__ s
6v>r< c ( U R U5 nU$ rY r! )r7 sequence_outputprediction_scoress r: rJ SqueezeBertOnlyMLMHead.forward s , ,_ = r< r$ r rT s @r: r r s ?! !r< r c * \ rS rSr% \\S' SrS rSrg)SqueezeBertPreTrainedModeli r8 transformerc [ U[ R [ R 45 ( ak UR R
R
SU R R S9 UR b% UR R
R 5 gg[ U[ R 5 ( ax UR R
R
SU R R S9 UR b2 UR R
UR R 5 gg[ U[ R 5 ( aJ UR R
R 5 UR R
R S5 g[ U[ 5 ( a% UR R
R 5 gg)zInitialize the weightsg )meanstdNg ?)r r r rv weightdatanormal_r8 initializer_ranger zero_r# r r, fill_r )r7 modules r:
_init_weights(SqueezeBertPreTrainedModel._init_weights s< fryy")) 4 5 5
MM & &CT[[5R5R & S{{& &&( '
-
- MM & &CT[[5R5R & S!!-
""6#5#56<<> .
-
- KK " " $ MM $ $S )
;
<
< KK " " $ =r< r` N) rM rN rO rP r __annotations__base_model_prefixr5 rR r` r< r: r) r) s %%r< r) c D ^ \ rS rSrU 4S jrS rS rS r\ SS\ \
R S\ \
R S\ \
R S \ \
R S
\ \
R S \ \
R S \ \
S
\ \
S\ \
S\\\4 4S jj5 rSrU =r$ )SqueezeBertModeli c > [ TU ] U5 [ U5 U l [ U5 U l [
U5 U l U R 5 g rY ) r! r" r rI r encoderr pooler post_initr6 s r: r" SqueezeBertModel.__init__ s@
/7)&1 '/
r< c . U R R $ rY rI r' r s r: get_input_embeddings%SqueezeBertModel.get_input_embeddings s ...r< c $ XR l g rY rA r7 new_embeddingss r: set_input_embeddings%SqueezeBertModel.set_input_embeddings s *8'r< c UR 5 H7 u p#U R R U R R U5 M9 g)z
Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base
class PreTrainedModel
N)itemsr< r r prune_heads)r7 heads_to_pruner headss r: _prune_heads SqueezeBertModel._prune_heads s<
+ 0 0 2LE LL u % / / ; ;E B 3r< rD r rE r r rF r r r r c
H Ub UOU R R nUb UOU R R nU b U OU R R n Ub Ub [ S5 eUb" U R X5 UR
5 n
O"Ub UR
5 S S n
O [ S5 eUb UR O UR n Uc [ R " XS9nUc$ [ R " U
[ R U S9nU R X*5 n U R XPR R 5 nU R XX6S9n
U R! U
U UUUU S9nUS nU R# U5 nU ( d
UU4US S - $ [% UUUR& UR( S
9$ ) NzDYou cannot specify both input_ids and inputs_embeds at the same timer z5You have to specify either input_ids or inputs_embeds)r@ r> )rD r rE rF )r r r r r r r r )r
pooler_outputr r )r8 r r use_return_dictr %warn_if_padding_and_no_attention_maskrA r@ r3 onesrB rC get_extended_attention_mask
get_head_maskr rI r< r= r r r )r7 rD r rE r r rF r r r rG r@ extended_attention_maskembedding_outputencoder_outputsr% r s r: rJ SqueezeBertModel.forward s 2C1N -TXT_T_TqTq$8$D $++JjJj &1% cd d
" 6 6y Q#..*K
&',,.s3K TU U%.%:!!
@T@T ! "ZZ CN ! "[[EJJv VN"&"B"B>"_ &&y++2O2OP ?? > +
,,*2 /!5#
'
*!, O4
#]3oab6II I) -')77&11
r< )rI r< r= ) NNNNNNNNN)rM rN rO rP r" rB rG rN r r r3 Tensor FloatTensorboolr r r rJ rR rS rT s @r: r: r: s /9C -11515/3,059,0/3&*A
ELL)A
! .A
! . A
u||, A
ELL)
A
1 12A
$D>A
'tnA
d^A
u00 1A
A
r< r: c f ^ \ rS rSrSS/rU 4S jrS rS r\ SS\ \
R S\ \
R S \ \
R S
\ \
R S \ \
R S \ \
R S
\ \
R S\ \ S\ \ S\ \ S\
\\4 4S jj5 rSrU =r$ )SqueezeBertForMaskedLMi z cls.predictions.decoder.weightz cls.predictions.decoder.biasc > [ TU ] U5 [ U5 U l [ U5 U l U R
5 g rY )r! r" r: r* r clsr> r6 s r: r" SqueezeBertForMaskedLM.__init__ s5
+F3)&1
r< c B U R R R $ rY )ra r! r r s r: get_output_embeddings,SqueezeBertForMaskedLM.get_output_embeddings& s xx##+++r< c XR R l UR U R R l g rY )ra r! r r rE s r: set_output_embeddings,SqueezeBertForMaskedLM.set_output_embeddings) s* '5 $$2$7$7 !r< rD r rE r r rF labelsr r r r c U
b U
OU R R n
U R UUUUUUUU U
S9 n U S n U R U 5 n
SnUbF [ 5 nU" U
R SU R R 5 UR S5 5 nU
( d U
4U SS - nUb U4U- $ U$ [ UU
U R U R S9$ )az
labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`
Nr rE r r rF r r r r r rh losslogitsr r )
r8 rR r* ra r r r$ r
r r )r7 rD r rE r r rF ri r r r outputsr% r& masked_lm_lossloss_fctr s r: rJ SqueezeBertForMaskedLM.forward- s ( &1%2
'tn2
d^2
un$ %2
2
r< r_ z
SqueezeBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the
pooled output) e.g. for GLUE tasks.
) custom_introc R ^ \ rS rSrU 4S jr\ SS\\R S\\R S\\R S\\R S\\R S\\R S \\R S
\\ S \\ S \\ S
\
\ \ 4 4S jj5 r
SrU =r$ )$SqueezeBertForSequenceClassificationic c P > [ TU ] U5 UR U l Xl [ U5 U l [ R " UR 5 U l [ R " UR U R R 5 U l U R 5 g rY )r! r"
num_labelsr8 r: r* r r/ r0 r1 r r-
classifierr> r6 s r: r" -SqueezeBertForSequenceClassification.__init__j ss
++ +F3zz&"<"<= ))F$6$6 8N8NO
r< rD r rE r r rF ri r r r r c R U
b U
OU R R n
U R UUUUUUUU U
S9 n U S n U R U 5 n U R U 5 n
SnUGb U R R
c U R S:X a SU R l OoU R S: aN UR [ R :X d UR [ R :X a SU R l OSU R l U R R
S:X aI [ 5 nU R S:X a& U" U
R 5 UR 5 5 nOU" X5 nOU R R
S:X a= [ 5 nU" U
R SU R 5 UR S5 5 nO,U R R
S:X a [ 5 nU" X5 nU
( d U
4U SS - nUb U4U- $ U$ [! UU
U R" U R$ S 9$ )
ae
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,
config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If
`config.num_labels > 1` a classification loss is computed (Cross-Entropy).
Nrk r
regressionsingle_label_classificationmulti_label_classificationr rh rl )r8 rR r* r1 rz problem_typery r? r3 rC r r squeezer r r r r r )r7 rD r rE r r rF ri r r r ro r rn rm rq r s r: rJ ,SqueezeBertForSequenceClassification.forwardv s ( &1%F
'tnF
d^F
u.. /F
F
r< rw c R ^ \ rS rSrU 4S jr\ SS\\R S\\R S\\R S\\R S\\R S\\R S \\R S
\\ S \\ S \\ S
\
\ \ 4 4S jj5 r
SrU =r$ ) SqueezeBertForMultipleChoicei c > [ TU ] U5 [ U5 U l [ R
" UR 5 U l [ R " UR S5 U l
U R 5 g )Nr ) r! r" r: r* r r/ r0 r1 r r- rz r> r6 s r: r" %SqueezeBertForMultipleChoice.__init__ sW
+F3zz&"<"<= ))F$6$6:
r< rD r rE r r rF ri r r r r c Z U
b U
OU R R n
Ub UR S OUR S n Ub! UR SUR S5 5 OSnUb! UR SUR S5 5 OSnUb! UR SUR S5 5 OSnUb! UR SUR S5 5 OSnUb1 UR SUR S5 UR S5 5 OSnU R UUUUUUUU U
S9 n U S n
U R
U
5 n
U R U
5 nUR SU 5 nSnUb [ 5 nU" X5 nU
( d U4U SS - nUb U4U- $ U$ [ UUU R U R S9$ )a
input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`):
Indices of input sequence tokens in the vocabulary.
Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and
[`PreTrainedTokenizer.__call__`] for details.
[What are input IDs?](../glossary#input-ids)
token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*):
Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,
1]`:
- 0 corresponds to a *sentence A* token,
- 1 corresponds to a *sentence B* token.
[What are token type IDs?](../glossary#token-type-ids)
position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*):
Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0,
config.max_position_embeddings - 1]`.
[What are position IDs?](../glossary#position-ids)
inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*):
Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This
is useful if you want more control over how to convert `input_ids` indices into associated vectors than the
model's internal embedding lookup matrix.
labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
Labels for computing the multiple choice classification loss. Indices should be in `[0, ...,
num_choices-1]` where *num_choices* is the size of the second dimension of the input tensors. (see
*input_ids* above)
Nr r rk rh rl ) r8 rR shaper rA r* r1 rz r r r r )r7 rD r rE r r rF ri r r r num_choicesro r rn reshaped_logitsrm rq r s r: rJ $SqueezeBertForMultipleChoice.forward s X &1%G>SINN2y~~b'9:Y] M[Mg ,,R 1D1DR1HImqM[Mg ,,R 1D1DR1HImqGSG_|((\->->r-BCei (
r=#5#5b#9=;M;Mb;Q R "" ))% ' /!5# #
]3
/ ++b+6 ')HO4D%''!"+5F)-)9TGf$ Ev E("!// ))
r< )rz r1 r* rs )rM rN rO rP r" r r r3 r[ r] r r r rJ rR rS rT s @r: r r s -11515/3,004)-,0/3&*X
ELL)X
! .X
! . X
u||, X
ELL)
X
-X
&X
$D>X
'tnX
d^X
u// 0X
X
r< r c R ^ \ rS rSrU 4S jr\ SS\\R S\\R S\\R S\\R S\\R S\\R S \\R S
\\ S \\ S \\ S
\
\ \ 4 4S jj5 r
SrU =r$ )!SqueezeBertForTokenClassificationi( c 0 > [ TU ] U5 UR U l [ U5 U l [
R " UR 5 U l [
R " UR UR 5 U l U R 5 g rY )
r! r" ry r: r* r r/ r0 r1 r r- rz r> r6 s r: r" *SqueezeBertForTokenClassification.__init__* sj
+++F3zz&"<"<= ))F$6$68I8IJ
r< rD r rE r r rF ri r r r r c U
b U
OU R R n
U R UUUUUUUU U
S9 n U S n U R U 5 n U R U 5 n
SnUb< [ 5 nU" U
R
SU R 5 UR
S5 5 nU
( d U
4U SS - nUb U4U- $ U$ [ UU
U R U R S9$ )z
labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`.
Nrk r r rh rl ) r8 rR r* r1 rz r r ry r r r )r7 rD r rE r r rF ri r r r ro r% rn rm rq r s r: rJ )SqueezeBertForTokenClassification.forward5 s $ &1%2
'tn2
d^2
u++ ,2
2
r< r c r ^ \ rS rSrU 4S jr\ SS\\R S\\R S\\R S\\R S\\R S\\R S \\R S
\\R S \\ S \\ S
\\ S\
\ \ 4 4S jj5 r
SrU =r$ ) SqueezeBertForQuestionAnsweringik c > [ TU ] U5 UR U l [ U5 U l [
R " UR UR 5 U l U R 5 g rY )
r! r" ry r: r* r r r-
qa_outputsr> r6 s r: r" (SqueezeBertForQuestionAnswering.__init__m sT
+++F3 ))F$6$68I8IJ
r< rD r rE r r rF start_positions
end_positionsr r r r c $ U b U OU R R n U R UUUUUUU U
U S9 n U S n
U R U
5 nUR SSS9u nnUR S5 R
5 nUR S5 R
5 nS nUb Ub [ UR 5 5 S: a UR S5 n[ UR 5 5 S: a UR S5 nUR S5 nUR SU5 nUR SU5 n[ US9nU" X5 nU" UU5 nUU- S- nU ( d UU4U SS - nUb U4U- $ U$ [ UUUU R U R S9$ ) Nrk r r r r ) ignore_indexrh )rm start_logits
end_logitsr r )r8 rR r* r splitr r r rA clampr r r r )r7 rD r rE r r rF r r r r r ro r% rn r r
total_loss
ignored_indexrq
start_lossend_lossr s r: rJ 'SqueezeBertForQuestionAnswering.forwardw s &1%
ELL)>
! .>
! . >
u||, >
ELL)
>
->
"%,,/>
->
$D>>
'tn>
d^>
u22 3>
>
r< r )r_ r r rw r r: r r) )5rQ r typingr r r3 r torch.nnr r r activationsr
modeling_outputsr r r
r r r r modeling_utilsr utilsr r configuration_squeezebertr
get_loggerrM loggerModuler rV r, rb rm r r r r r r r r r) r: r_ rw r r r __all__r` r< r: r s ! " A A ! . 9
H %)BII )X(BII ((
"2<<
" 299 ( RYY Wryy Wt' 'T:
:
z " ")) .!RYY ! % % %. [
1 [
[
| F
7 F
F
R T
+E T
T
n d
#= d
d
N ?
(B ?
?
D J
&@ J
J
Z r<