# block_zoo package

## Subpackages

- block_zoo.attentions package
  - Submodules
  - block_zoo.attentions.Attention module
  - block_zoo.attentions.BiAttFlow module
  - block_zoo.attentions.BilinearAttention module
  - block_zoo.attentions.FullAttention module
  - block_zoo.attentions.LinearAttention module
  - block_zoo.attentions.MatchAttention module
  - block_zoo.attentions.Seq2SeqAttention module
  - Module contents
- block_zoo.embedding package
  - Submodules
  - block_zoo.embedding.CNNCharEmbedding module
  - Module contents
- block_zoo.encoder_decoder package
  - Submodules
  - block_zoo.encoder_decoder.SLUDecoder module
  - block_zoo.encoder_decoder.SLUEncoder module
  - Module contents
- block_zoo.math package
  - Submodules
  - block_zoo.math.Add2D module
  - block_zoo.math.Add3D module
  - block_zoo.math.ElementWisedMultiply2D module
  - block_zoo.math.ElementWisedMultiply3D module
  - block_zoo.math.MatrixMultiply module
  - block_zoo.math.Minus2D module
  - block_zoo.math.Minus3D module
  - Module contents
- block_zoo.normalizations package
  - Submodules
  - block_zoo.normalizations.LayerNorm module
  - Module contents
- block_zoo.op package
  - Submodules
  - block_zoo.op.Combination module
  - block_zoo.op.Concat2D module
  - block_zoo.op.Concat3D module
  - Module contents
- block_zoo.transformer package
  - Submodules
  - block_zoo.transformer.MLP module
  - block_zoo.transformer.MultiHeadAttention module
  - Module contents

## Submodules

## block_zoo.BaseLayer module

class `block_zoo.BaseLayer.``BaseConf`(**kwargs)

  Bases: `abc.ABC`

  Basic configuration

  Parameters: | input_dim (int) – the dimension of input. hidden_dim (int) – the dimension of hidden state. dropout (float) – dropout rate. (others).. –

  `add_attr_exist_assertion_for_dev`(attr)

    check if there are some attributes being forgot by developers

    Parameters: | attr (str) – the attribution name
    Returns: | None

  `add_attr_exist_assertion_for_user`(attr)

    check if there are some attributes being forgot by users

    Parameters: | attr (str) – the attribution name
    Returns: | None

  `add_attr_range_assertion`(attr, range, bounds_legal=(True, True))

    check if attribute falls into the legal range

    Parameters: | attr (str) – the attribution name range (tuple) – (num/float(‘-inf’)/None, num/float(‘inf’)/None), None means -inf or inf. bounds_legal (tuple) – (bool, bool), if the left/right bound is legal
    Returns: | None

  `add_attr_type_assertion`(attr, specified_type)

    check if the types of attributes are legal

    Parameters: | attr (str) – the attribution name specified_type (None/str/list) – one specified_type of a list of specified_type(including None)
    Returns: | None

  `add_attr_value_assertion`(attr, legal_values)

    check if attr equals to one of the legal values

    Parameters: | attr (str) – the attribution name legal_values (list) – include the legal value
    Returns: | None

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

  `verify_before_inference`()

    Some conditions must be fulfilled, otherwise there would be errors when calling inference()

    The difference between verify_before_inference() and verify() is that:
      verify_before_inference() is called before inference() while verify() is called after inference().

    Returns: | None

class `block_zoo.BaseLayer.``BaseLayer`(layer_conf)

  Bases: `torch.nn.modules.module.Module`

  The base class of layers

  Parameters: | layer_conf (BaseConf) – configuration of a layer

  `forward`(*args)

    Parameters: | *args (list) – a list of args in which arg should be a pair of (representation, length)
    Returns: | None

  `is_cuda`()

    To judge if the layer is on CUDA if there are parameters in this layer, judge according to the parameters; else: judge according to the self.layer_conf.use_gpu

    Returns: | whether to use gpu
    Return type: | bool

## block_zoo.BiGRU module

class `block_zoo.BiGRU.``BiGRU`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Bidirectional GRU

  Parameters: | layer_conf (BiGRUConf) – configuration of a layer

  `forward`(string, string_len)

    process inputs

    Parameters: | string (Tensor) – [batch_size, seq_len, dim] string_len (Tensor) – [batch_size]
    Returns: | [batch_size, seq_len, 2 * hidden_dim]
    Return type: | Tensor

class `block_zoo.BiGRU.``BiGRUConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of BiGRU

  Parameters: | hidden_dim (int) – dimension of hidden state dropout (float) – dropout rate

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

## block_zoo.BiGRULast module

class `block_zoo.BiGRULast.``BiGRULast`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Get the last hidden state of Bi GRU

  Parameters: | layer_conf (BiGRULastConf) – configuration of a layer

  `forward`(string, string_len)

    process inputs

    Parameters: | string (Tensor) – [batch_size, seq_len, dim] string_len (Tensor) – [batch_size]
    Returns: | [batch_size, 2 * hidden_dim]
    Return type: | Tensor

class `block_zoo.BiGRULast.``BiGRULastConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of the layer BiGRULast

  Parameters: | hidden_dim (int) – dimension of hidden state dropout (float) – dropout rate

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

  `verify_before_inference`()

    Some conditions must be fulfilled, otherwise there would be errors when calling inference()

    The difference between verify_before_inference() and verify() is that:
      verify_before_inference() is called before inference() while verify() is called after inference().

    Returns: | None

## block_zoo.BiLSTM module

class `block_zoo.BiLSTM.``BiLSTM`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Bidrectional LSTM

  Parameters: | layer_conf (BiLSTMConf) – configuration of a layer

  `forward`(string, string_len)

    process inputs

    Parameters: | string (Tensor) – [batch_size, seq_len, dim] string_len (Tensor) – [batch_size]
    Returns: | [batch_size, seq_len, 2 * hidden_dim]
    Return type: | Tensor

class `block_zoo.BiLSTM.``BiLSTMConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of BiLSTM

  Parameters: | hidden_dim (int) – dimension of hidden state dropout (float) – dropout rate num_layers (int) – number of BiLSTM layers

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

## block_zoo.BiLSTMAtt module

class `block_zoo.BiLSTMAtt.``BiLSTMAtt`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  BiLSTM with self attention

  Parameters: | layer_conf (BiLSTMAttConf) – configuration of a layer

  `forward`(string, string_len)

    process inputs

    Parameters: | string (Tensor) – [batch_size, seq_len, dim] string_len (Tensor) – [batch_size]
    Returns: | [batch_size, seq_len, 2 * hidden_dim]
    Return type: | Tensor

class `block_zoo.BiLSTMAtt.``BiLSTMAttConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of BiLSTMAtt layer

  Parameters: | hidden_dim (int) – dimension of hidden state dropout (float) – dropout rate num_layers (int) – number of BiLSTM layers

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

  `verify_before_inference`()

    Some conditions must be fulfilled, otherwise there would be errors when calling inference()

    The difference between verify_before_inference() and verify() is that:
      verify_before_inference() is called before inference() while verify() is called after inference().

    Returns: | None

## block_zoo.BiLSTMLast module

class `block_zoo.BiLSTMLast.``BiLSTMLast`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  get last hidden states of Bidrectional LSTM

  Parameters: | layer_conf (BiLSTMConf) – configuration of a layer

  `forward`(string, string_len)

    process inputs

    Parameters: | string (Tensor) – [batch_size, seq_len, dim] string_len (Tensor) – [batch_size]
    Returns: | [batch_size, 2 * hidden_dim]
    Return type: | Tensor

class `block_zoo.BiLSTMLast.``BiLSTMLastConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of BiLSTMLast

  Parameters: | hidden_dim (int) – dimension of hidden state dropout (float) – dropout rate num_layers (int) – number of BiLSTM layers

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

## block_zoo.BiQRNN module

class `block_zoo.BiQRNN.``BiQRNN`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Bidrectional QRNN

  Parameters: | layer_conf (BiQRNNConf) – configuration of a layer

  `forward`(string, string_len)

    process inputs

    Parameters: | string (Tensor) – [batch_size, seq_len, dim] string_len (Tensor) – [batch_size]
    Returns: | [batch_size, seq_len, hidden_dim]
    Return type: | Tensor

class `block_zoo.BiQRNN.``BiQRNNConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of BiQRNN

  Parameters: | hidden_dim (int) – dimension of hidden state window – the size of the convolutional window. Supports 1 and 2. Default: 1 zoneout – Whether to apply zoneout (failing to update elements in the hidden state). Default: 0 dropout (float) – dropout rate bewteen BiQRNN layers num_layers (int) – number of BiQRNN layers

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

class `block_zoo.BiQRNN.``ForgetMult`

  Bases: `torch.nn.modules.module.Module`

  ForgetMult computes a simple recurrent equation: h_t = f_t * x_t + (1 - f_t) * h_{t-1}

  This equation is equivalent to dynamic weighted averaging.

  Inputs: X, hidden
    - X (seq_len, batch, input_size): tensor containing the features of the input sequence.
    - F (seq_len, batch, input_size): tensor containing the forget gate values, assumed in range [0, 1].
    - hidden_init (batch, input_size): tensor containing the initial hidden state for the recurrence (h_{t-1}).
  `forward`(f, x, hidden_init=None)

    Defines the computation performed at every call.

    Should be overridden by all subclasses.

    Note

    Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class `block_zoo.BiQRNN.``QRNN`(input_size, hidden_size, num_layers=1, bias=True, batch_first=False, dropout=0.0, bidirectional=False, **kwargs)

  Bases: `torch.nn.modules.module.Module`

  Applies a multiple layer Quasi-Recurrent Neural Network (QRNN) to an input sequence.

  Parameters: | input_size – The number of expected features in the input x. hidden_size – The number of features in the hidden state h. If not specified, the input size is used. num_layers – The number of QRNN layers to produce. dropout – Whether to use dropout between QRNN layers. Default: 0. bidirectional – If True, becomes a bidirectional QRNN. Default: False. save_prev_x – Whether to store previous inputs for use in future convolutional windows (i.e. for a continuing sequence such as in language modeling). If true, you must call reset to remove cached previous values of x. Default: False. window – Defines the size of the convolutional window (how many previous tokens to look when computing the QRNN values). Supports 1 and 2. Default: 1. zoneout – Whether to apply zoneout (i.e. failing to update elements in the hidden state) to the hidden state updates. Default: 0. output_gate – If True, performs QRNN-fo (applying an output gate to the output). If False, performs QRNN-f. Default: True.

  Inputs: X, hidden
    - X (seq_len, batch, input_size): tensor containing the features of the input sequence.
    - hidden (num_layers * num_directions, batch, hidden_size): tensor containing the initial hidden state for the QRNN.
  Outputs: output, h_n
    - output (seq_len, batch, hidden_size * num_directions): tensor containing the output of the QRNN for each timestep.
    - h_n (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t=seq_len
  `forward`(input, hidden=None)

    Defines the computation performed at every call.

    Should be overridden by all subclasses.

    Note

    Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

  `reset`()

    If your convolutional window is greater than 1, you must reset at the beginning of each new sequence

  `tensor_reverse`(tensor)

class `block_zoo.BiQRNN.``QRNNLayer`(input_size, hidden_size=None, save_prev_x=False, zoneout=0, window=1, output_gate=True)

  Bases: `torch.nn.modules.module.Module`

  Applies a single layer Quasi-Recurrent Neural Network (QRNN) to an input sequence.

  Parameters: | input_size – The number of expected features in the input x. hidden_size – The number of features in the hidden state h. If not specified, the input size is used. save_prev_x – Whether to store previous inputs for use in future convolutional windows (i.e. for a continuing sequence such as in language modeling). If true, you must call reset to remove cached previous values of x. Default: False. window – Defines the size of the convolutional window (how many previous tokens to look when computing the QRNN values). Supports 1 and 2. Default: 1. zoneout – Whether to apply zoneout (i.e. failing to update elements in the hidden state) to the hidden state updates. Default: 0. output_gate – If True, performs QRNN-fo (applying an output gate to the output). If False, performs QRNN-f. Default: True.

  Inputs: X, hidden
    - X (seq_len, batch, input_size): tensor containing the features of the input sequence.
    - hidden (batch, hidden_size): tensor containing the initial hidden state for the QRNN.
  Outputs: output, h_n
    - output (seq_len, batch, hidden_size): tensor containing the output of the QRNN for each timestep.
    - h_n (1, batch, hidden_size): tensor containing the hidden state for t=seq_len
  `forward`(X, hidden=None)

    Defines the computation performed at every call.

    Should be overridden by all subclasses.

    Note

    Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

  `reset`()

## block_zoo.Conv module

class `block_zoo.Conv.``Conv`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Convolution along just 1 direction

  Parameters: | layer_conf (ConvConf) – configuration of a layer

  `forward`(string, string_len=None)

    process inputs

    Parameters: | string (Tensor) – tensor with shape: [batch_size, seq_len, feature_dim] string_len (Tensor) – [batch_size]
    Returns: | shape: [batch_size, (seq_len - conv_window_size) // stride + 1, output_channel_num]
    Return type: | Tensor

class `block_zoo.Conv.``ConvConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of Conv

  Parameters: | stride (int) – the stride of the convolving kernel. Can be a single number or a tuple (sH, sW). Default: 1 padding (int) – implicit zero paddings on both sides of the input. Can be a single number or a tuple (padH, padW). Default: 0 window_size (int) – actually, the window size is (window_size, feature_dim), because for NLP tasks, 1d convolution is more commonly used. input_channel_num (int) – for NLP tasks, input_channel_num would always be 1 output_channel_num (int) – number of feature maps batch_norm (bool) – If True, apply batch normalization before activation activation (string) – activation functions, e.g. ReLU

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

  `verify_before_inference`()

    Some conditions must be fulfilled, otherwise there would be errors when calling inference()

    The difference between verify_before_inference() and verify() is that:
      verify_before_inference() is called before inference() while verify() is called after inference().

    Returns: | None

## block_zoo.ConvPooling module

class `block_zoo.ConvPooling.``ConvPooling`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Convolution along just 1 direction

  Parameters: | layer_conf (ConvConf) – configuration of a layer

  `forward`(string, string_len=None)

    process inputs

    Parameters: | string (Tensor) – tensor with shape: [batch_size, seq_len, feature_dim] string_len (Tensor) – [batch_size]
    Returns: | shape: [batch_size, (seq_len - conv_window_size) // stride + 1, output_channel_num]
    Return type: | Tensor

class `block_zoo.ConvPooling.``ConvPoolingConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of Conv + Pooling architecture

  Parameters: | stride (int) – the stride of the convolving kernel. Can be a single number or a tuple (sH, sW). Default: 1 padding (int) – implicit zero paddings on both sides of the input. Can be a single number or a tuple (padH, padW). Default: 0 window_sizes (list) – for each window_size, the actual window size is (window_size, feature_dim), because for NLP tasks, 1d convolution is more commonly used. input_channel_num (int) – for NLP tasks, input_channel_num would always be 1 output_channel_num (int) – number of feature maps batch_norm (bool) – If True, apply batch normalization before activation activation (string) – activation functions, e.g. ReLU

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

  `verify_before_inference`()

    Some conditions must be fulfilled, otherwise there would be errors when calling inference()

    The difference between verify_before_inference() and verify() is that:
      verify_before_inference() is called before inference() while verify() is called after inference().

    Returns: | None

## block_zoo.Dropout module

class `block_zoo.Dropout.``Dropout`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Parameters: | layer_conf (DropoutConf) – configuration of a layer

  `forward`(string, string_len=None)

    process inputs

    Parameters: | string (Tensor) – any shape. string_len (Tensor) – [batch_size], default is None.
    Returns: | has the same shape as string.
    Return type: | Tensor

class `block_zoo.Dropout.``DropoutConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration for Dropout

  Parameters: | dropout (float) – dropout rate, probability of an element to be zeroed

  Returns:

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

## block_zoo.Embedding module

class `block_zoo.Embedding.``Embedding`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Embedding layer

  Parameters: | layer_conf (EmbeddingConf) – configuration of a layer

  `forward`(inputs, use_gpu=False)

    process inputs

    Parameters: | inputs (dict) – a dictionary to describe each transformer_model inputs. e.g.: char_emb’: [[char ids of word1], [char ids of word2], […], …], shape: [batch_size, seq_len, word character num] ’word’: word ids (Variable), shape:[batch_size, seq_len], ’postag’: postag ids (Variable), shape: [batch_size, seq_len], … use_gpu (bool) – put embedding matrix on GPU (True) or not (False)
    Returns: | the embedding representation with shape [batch_size, seq_len, emb_dim]
    Return type: | Variable

class `block_zoo.Embedding.``EmbeddingConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration for Embedding

  Parameters: | conf – a dictionary. The key is embedding type, such as word embedding, char embedding, Part-of-Speech embedding and so on.

  Example:

      "conf": {
        "word": {
          "cols": ["question_text", "answer_text"],
          "dim": 300,
          "fix_weight": true
        },
        "postag": {
          "cols": ["question_postag","answer_postag"],
          "dim": 20
        },
        "char": {
          "cols": ["question_char", "answer_char"],
          "type": "CNNCharEmbedding",
          "dropout": 0.2,
          "dim": 30,
          "embedding_matrix_dim": 8,
          "stride":1,
          "window_size": 5,
          "activation": null
        }
      }

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

  `verify_before_inference`()

    Some conditions must be fulfilled, otherwise there would be errors when calling inference()

    The difference between verify_before_inference() and verify() is that:
      verify_before_inference() is called before inference() while verify() is called after inference().

    Returns: | None

## block_zoo.EncoderDecoder module

class `block_zoo.EncoderDecoder.``EncoderDecoder`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  The encoder decoder framework

  Parameters: | layer_conf (EncoderDecoderConf) – configuration of a layer

  `forward`(string, string_len)

    process inputs with encoder & decoder

    Parameters: | string (Variable) – [batch_size, seq_len, dim] string_len (ndarray) – [batch_size]
    Returns: | decode scores with shape [batch_size, seq_len, decoder_vocab_size]
    Return type: | Variable

class `block_zoo.EncoderDecoder.``EncoderDecoderConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of Encoder-Decoder

  Parameters: | encoder (str) – encoder name encoder_conf (dict) – configurations of encoder decoder (str) – decoder name decoder_conf (dict) – configurations of decoder

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Dimension inference of encoder and decoder is conducted here, but not in the Model.

    Returns:

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

  `verify_before_inference`()

    Some conditions must be fulfilled, otherwise there would be errors when calling inference()

    The difference between verify_before_inference() and verify() is that:
      verify_before_inference() is called before inference() while verify() is called after inference().

    Returns: | None

## block_zoo.Flatten module

class `block_zoo.Flatten.``Flatten`(layer_conf)

  Bases: `torch.nn.modules.module.Module`

  Flatten layer to flatten the input from [bsatch_size, seq_len, dim] to [batch_size, seq_len*dim]

  Parameters: | layer_conf (FlattenConf) – configuration of a layer

  `forward`(string, string_len)

    process input

    Parameters: | *args – (Tensor): string,string_len e.g. string (Tensor): [batch_size, seq_len, dim], string_len (Tensor): [batch_size]
    Returns: | [batch_size, seq_len*dim], [batch_size]
    Return type: | Tensor

class `block_zoo.Flatten.``FlattenConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of Flatten layer

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

## block_zoo.HighwayLinear module

class `block_zoo.HighwayLinear.``HighwayLinear`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  A Highway layer does a gated combination of a linear transformation and a non-linear transformation of its input. \(y = g * x + (1 - g) * f(A(x))\), where \(A\) is a linear transformation, \(f\) is an element-wise non-linearity, and \(g\) is an element-wise gate, computed as \(sigmoid(B(x))\). This module will apply a fixed number of highway layers to its input, returning the final result.

  Parameters: | layer_conf (HighwayLinearConf) – configuration of a layer

  `forward`(string, string_len)

    process inputs

    Parameters: | string (Tensor) – [batch_size, seq_len, dim] string_len (Tensor) – [batch_size]
    Returns: | [batch_size, seq_len, 2 * hidden_dim]
    Return type: | Tensor

class `block_zoo.HighwayLinear.``HighwayLinearConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of BiLSTM

  Parameters: | hidden_dim (int) – dimension of hidden state dropout (float) – dropout rate num_layers (int) – number of BiLSTM layers

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

## block_zoo.Linear module

class `block_zoo.Linear.``Linear`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Linear layer

  Parameters: | layer_conf (LinearConf) – configuration of a layer

  `forward`(string, string_len=None)

    process inputs

    Parameters: | string (Tensor) – any shape. string_len (Tensor) – [batch_size], default is None.
    Returns: | has the same shape as string.
    Return type: | Tensor

class `block_zoo.Linear.``LinearConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration for Linear layer

  Parameters: | hidden_dim (int or list) – if is int, it means there is one linear layer and the hidden_dim is the dimension of that layer. if is list of int, it means there is multiple linear layer and hidden_dim are the dimensions of these layers. activation (str) – Name of activation function. All the non-linear activations in http://pytorch.org/docs/0.3.1/nn.html#non-linear-activations are supported, such as ‘Tanh’, ‘ReLU’, ‘PReLU’, ‘ReLU6’ and ‘LeakyReLU’. Default is None. last_hidden_activation (bool) – [Optional], whether to add nonlinearity to the last linear layer’s output. Default is True. last_hidden_softmax (bool) – [Optional], whether to add softmax to the last linear layer’s output. Default is False.

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

  `verify_before_inference`()

    Some conditions must be fulfilled, otherwise there would be errors when calling inference()

    The difference between verify_before_inference() and verify() is that:
      verify_before_inference() is called before inference() while verify() is called after inference().

    Returns: | None

## block_zoo.Pooling module

class `block_zoo.Pooling.``Pooling`(layer_conf)

  Bases: `block_zoo.BaseLayer.BaseLayer`

  Pooling layer

  Parameters: | layer_conf (PoolingConf) – configuration of a layer

  `forward`(string, string_len=None)

    process inputs

    Parameters: | string (Tensor) – any shape. string_len (Tensor) – [batch_size], default is None.
    Returns: | Pooling result of string
    Return type: | Tensor

class `block_zoo.Pooling.``PoolingConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Parameters: | pool_type (str) – ‘max’ or ‘mean’, default is ‘max’. pool_axis (int) – which axis to conduct pooling, default is 1.

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

## block_zoo.Transformer module

class `block_zoo.Transformer.``Transformer`(layer_conf)

  Bases: `torch.nn.modules.module.Module`

  Transformer layer

  Parameters: | layer_conf (TransformerConf) – configuration of a layer

  `forward`(string, string_len)

    process input

    Parameters: | string (Tensor) – [batch_size, seq_len, dim] string_len (Tensor) – [batch_size]
    Returns: | [batch_size, seq_len, output_dim], [batch_size]
    Return type: | Tensor

class `block_zoo.Transformer.``TransformerConf`(**kwargs)

  Bases: `block_zoo.BaseLayer.BaseConf`

  Configuration of Transformer

  Parameters: | attention (str) – attention name attention_conf (dict) – configurations of attention layernorm1 (str) – layernorm1 name layernorm1_conf (dict) – configurations of layernorm1 mlp (str) – mlp name mlp_conf (dict) – configuration of mlp layernorm2 (str) – layernorm2 name layernorm2_conf (dict) – configurations of layernorm2 n_layer (int) –

  `declare`()

    Define things like “input_ranks” and “num_of_inputs”, which are certain with regard to your layer

    > num_of_input is N(N>0) means this layer accepts N inputs;

    > num_of_input is -1 means this layer accepts any number of inputs;

    > The rank here is not the same as matrix rank:

    > > For a scalar, its rank is 0;

    > > For a vector, its rank is 1;

    > > For a matrix, its rank is 2;

    > > For a cube of numbers, its rank is 3.

    > … For instance, the rank of (batch size, sequence length, hidden_dim) is 3.

    > if num_of_input > 0:

    > > len(input_ranks) should be equal to num_of_input

    > elif num_of_input == -1:

    > > input_ranks should be a list with only one element and the rank of all the inputs should be equal to that element.

    > NOTE: when we build the model, if num_of_input is -1, we would replace it with the real number of inputs and replace input_ranks with a list of real input_ranks.

    Returns: | None

  `default`()

    Define the default hyper parameters here. You can define these hyper parameters in your configuration file as well.

    Returns: | None

  `inference`()

    Inference things like output_dim, which may relies on defined hyper parameter such as hidden dim and input_dim

    Returns: | None

  `verify`()

    Define some necessary varification for your layer when we define the model.

    If you define your own layer and rewrite this funciton, please add “super(YourLayerConf, self).verify()” at the beginning

    Returns: | None

## Module contents