sinusoidalPositionEncodingLayer
Description
A sinusoidal position encoding layer maps position indices to vectors using sinusoidal operations. Use this layer in transformer neural networks to provide information about the position of the data in a sequence or image.
Creation
Syntax
Description
creates a sinusoidal position encoding layer and sets the layer
= sinusoidalPositionEncodingLayer(outputSize
)OutputSize
property.
Properties
Sinusoidal Position Encoding
OutputSize
— Number of channels in layer output
even positive integer
This property is read-only.
Number of channels in the layer output, specified as an even positive integer.
Data Types: single
| double
| int8
| int16
| int32
| int64
| uint8
| uint16
| uint32
| uint64
Positions
— Positions in input
"auto"
(default) | "temporal-indicies"
| "spatial-indicies"
| "data-values"
This property is read-only.
Positions in the input, specified as one of these values:
"auto"
— For sequence or spatial-temporal input, use the temporal indices as positions, which is equivalent to using"temporal-indices"
. For one-dimensional image input, use the spatial indices as positions, which is equivalent to using"spatial-indices"
. For other input, use the input values as positions, which is equivalent to using"data-values"
."temporal-indices"
— Use the temporal indices of the input as positions."spatial-indices"
— Use the spatial indices of the input as positions."data-values"
— Use the values in the input as positions.
Layer
Name
— Layer name
""
(default) | character vector | string scalar
NumInputs
— Number of inputs
1
(default)
This property is read-only.
Number of inputs to the layer, returned as 1
. This layer accepts a
single input only.
Data Types: double
InputNames
— Input names
{'in'}
(default)
This property is read-only.
Input names, returned as {'in'}
. This layer accepts a single input
only.
Data Types: cell
NumOutputs
— Number of outputs
1
(default)
This property is read-only.
Number of outputs from the layer, returned as 1
. This layer has a
single output only.
Data Types: double
OutputNames
— Output names
{'out'}
(default)
This property is read-only.
Output names, returned as {'out'}
. This layer has a single output
only.
Data Types: cell
Examples
Create Sinusoidal Position Encoding Layer
Create a sinusoidal position encoding layer with an output size of 300.
layer = sinusoidalPositionEncodingLayer(300)
layer = SinusoidalPositionEncodingLayer with properties: Name: '' OutputSize: 300 Positions: 'auto' Learnable Parameters No properties. State Parameters No properties. Use properties method to see a list of all properties.
Create a neural network containing a sinusoidal position encoding layer.
net = dlnetwork; numChannels = 1; embeddingOutputSize = 64; numWords = 128; maxPosition = 128; numHeads = 4; numKeyChannels = 4*embeddingOutputSize; layers = [ sequenceInputLayer(numChannels,Name="input") wordEmbeddingLayer(embeddingOutputSize,numWords,Name="word-emb") sinusoidalPositionEncodingLayer(embeddingOutputSize,Name="pos-enc"); additionLayer(2,Name="add") selfAttentionLayer(numHeads,numKeyChannels,AttentionMask="causal") fullyConnectedLayer(numWords) softmaxLayer]; net = addLayers(net,layers); net = connectLayers(net,"word-emb","add/in2");
View the neural network architecture.
plot(net) axis off box off
Algorithms
Sinusoidal Position Encoding Layer
A sinusoidal position encoding layer maps position indices to vectors using sinusoidal operations. The layer encodes position information of data for transformer neural networks.
The output of the layer has the same number of dimensions as the input. In the output, each vector in position p over the channel dimension is given by:
where p is the position, d is the encoding output
size given by OutputSize
and is the wavelength given by:
for .
When Positions
is "auto"
, the layout of the output
depends on the type of data:
For sequence data
X
represented by anumChannels
-by-numObservations
-by-numTimeSteps
array, wherenumChannels
,numObservations
, andnumTimeSteps
are the numbers of channels, observations, and time steps of the input, respectively, the output is anOutputSize
-by-numObservations
-by-numTimeSteps
arrayY
, where each vector inY(:,:,t)
over the channel dimension is .For 1-D image data
X
represented by aheight
-by-numChannels
-by-numObservations
array, whereheight
,numChannels
, andnumObservations
are the height, number of channels, and number of observations of the input images, respectively, the output is aheight
-by-OutputSize
-by-numObservations
arrayY
, where each vector inY(i,:,:)
over the channel dimension is .For 2-D image sequence data
X
represented by aheight
-by-width
-by-numChannels
-by-numObservations
-by-numTimeSteps
array, whereheight
andwidth
are the height and width of the input image sequences, respectively, andnumChannels
,numObservations
, andnumTimeSteps
are the numbers of channels, observations, and time steps of the input image sequences, respectively, the output is aheight
-by-width
-by-OutputSize
-by-numObservations
-by-numTimeSteps
arrayY
, where each vector inY(:,:,:,:,t)
over the channel dimension is .
Layer Input and Output Formats
Layers in a layer array or layer graph pass data to subsequent layers as formatted dlarray
objects.
The format of a dlarray
object is a string of characters in which each
character describes the corresponding dimension of the data. The formats consist of one or
more of these characters:
"S"
— Spatial"C"
— Channel"B"
— Batch"T"
— Time"U"
— Unspecified
For example, you can describe 2-D image data that is represented as a 4-D array, where the
first two dimensions correspond to the spatial dimensions of the images, the third
dimension corresponds to the channels of the images, and the fourth dimension
corresponds to the batch dimension, as having the format "SSCB"
(spatial, spatial, channel, batch).
You can interact with these dlarray
objects in automatic differentiation
workflows, such as those for developing a custom layer, using a functionLayer
object, or using the forward
and predict
functions with
dlnetwork
objects.
This table shows the supported input formats of SinusoidalPositionEncodingLayer
objects and the
corresponding output format. If the software passes the output of the layer to a custom
layer that does not inherit from the nnet.layer.Formattable
class, or a
FunctionLayer
object with the Formattable
property
set to 0
(false
), then the layer receives an
unformatted dlarray
object with dimensions ordered according to the formats
in this table. The formats listed here are only a subset. The layer may support additional
formats such as formats with additional "S"
(spatial) or
"U"
(unspecified) dimensions.
Input Format | Positions | Output Format |
---|---|---|
"CB" (channel, batch) |
| "CB" (channel, batch) |
"SCB" (spatial, channel, batch) |
| "SCB" (spatial, channel, batch) |
"SSCB" (spatial, spatial, channel, batch) | "data-values" | "SSCB" (spatial, spatial, channel, batch) |
"SSSCB" (spatial, spatial, spatial, channel,
batch) | "data-values" | "SSSCB" (spatial, spatial, spatial, channel,
batch) |
"CBT" (channel, batch, time) |
| "CBT" (channel, batch, time) |
"SCBT" (spatial, channel, batch, time) |
| "SCBT" (spatial, channel, batch, time) |
"SSCBT" (spatial, spatial, channel, batch, time) |
| "SSCBT" (spatial, spatial, channel, batch, time) |
"SSSCBT" (spatial, spatial, spatial, channel, batch,
time) |
| "SSSCBT" (spatial, spatial, spatial, channel, batch,
time) |
"SC" (spatial, channel) |
| "SC" (spatial, channel) |
"SSC" (spatial, spatial, channel) | "data-values" | "SSC" (spatial, spatial, channel) |
"SSSC" (spatial, spatial, spatial, channel) | "data-values" | "SSSC" (spatial, spatial, spatial, channel) |
"SB" (spatial, batch) |
| "SCB" (spatial, channel, batch) |
"SSB" (spatial, spatial, batch) | "data-values" | "SSCB" (spatial, spatial, channel, batch) |
"SSSB" (spatial, spatial, spatial, batch) | "data-values" | "SSSCB" (spatial, spatial, spatial, channel,
batch) |
"SS" (spatial, spatial) | "data-values" | "SSC" (spatial, spatial, channel) |
"SSS" (spatial, spatial, spatial) | "data-values" | "SSSC" (spatial, spatial, spatial, channel) |
"SU" (spatial, unspecified) |
| "SCU" (spatial, channel, unspecified) |
"BU" (batch, unspecified) |
| "CBU" (channel, batch, unspecified) |
"UU" (unspecified, unspecified) |
| "CUU" (channel, unspecified, unspecified) |
"UUU" (unspecified, unspecified, unspecified) |
| "CUUU" (channel, unspecified, unspecified,
unspecified) |
"UUUU" (unspecified, unspecified, unspecified,
unspecified) |
| "CUUUU" (channel, unspecified, unspecified, unspecified,
unspecified) |
"UUUUU" (unspecified, unspecified, unspecified,
unspecified, unspecified) |
| "CUUUUU" (channel, unspecified, unspecified, unspecified,
unspecified, unspecified) |
In dlnetwork
objects, SinusoidalPositionEncodingLayer
objects also support
these input and output format combinations.
Input Format | Positions | Output Format |
---|---|---|
"CT" (channel, time) |
| "CT" (channel, time) |
"SCT" (spatial, channel, time) |
| "SCT" (spatial, channel, time) |
"SSCT" (spatial, spatial, channel, time) |
| "SSCT" (spatial, spatial, channel, time) |
"SSSCT" (spatial, spatial, spatial, channel, time) |
| "SSSCT" (spatial, spatial, spatial, channel, time) |
"BT" (batch, time) |
| "CBT" (channel, batch, time) |
"SBT" (spatial, batch, time) |
| "SCBT" (spatial, channel, batch, time) |
"SSBT" (spatial, spatial, batch, time) |
| "SSCBT" (spatial, spatial, channel, batch, time) |
"SSSBT" (spatial, spatial, spatial, batch, time) |
| "SSSCBT" (spatial, spatial, spatial, channel, batch,
time) |
"ST" (spatial, time) |
| "SCT" (spatial, channel, time) |
"SST" (spatial, spatial, time) |
| "SSCT" (spatial, spatial, channel, time) |
"SSST" (spatial, spatial, spatial, time) |
| "SSSCT" (spatial, spatial, spatial, channel, time) |
"TU" (time, unspecified) |
| "CTU" (channel, time, unspecified) |
References
[1] Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc., 2017. https://papers.nips.cc/paper/7181-attention-is-all-you-need.
Version History
Introduced in R2023b
MATLAB-Befehl
Sie haben auf einen Link geklickt, der diesem MATLAB-Befehl entspricht:
Führen Sie den Befehl durch Eingabe in das MATLAB-Befehlsfenster aus. Webbrowser unterstützen keine MATLAB-Befehle.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)