pix2pixHDGlobalGenerator

Create pix2pixHD global generator network

Since R2021a

Syntax

net = pix2pixHDGlobalGenerator(inputSize)

net = pix2pixHDGlobalGenerator(inputSize,Name=Value)

Description

net = pix2pixHDGlobalGenerator(inputSize) creates a pix2pixHD generator network for input of size inputSize. For more information about the network architecture, see pix2pixHD Generator Network.

This function requires Deep Learning Toolbox™.

net = pix2pixHDGlobalGenerator(inputSize,Name=Value) modifies properties of the pix2pixHD network using name-value arguments.

Examples

Create Pix2PixHD Generator

Specify the network input size for 32-channel data of size 512-by-1024 pixels.

inputSize = [512 1024 32];

Create a pix2pixHD global generator network.

net = pix2pixHDGlobalGenerator(inputSize)

net = 
  dlnetwork with properties:

         Layers: [84×1 nnet.cnn.layer.Layer]
    Connections: [92×2 table]
     Learnables: [110×3 table]
          State: [0×3 table]
     InputNames: {'GlobalGenerator_inputLayer'}
    OutputNames: {'GlobalGenerator_fActivation'}
    Initialized: 1

  View summary with summary.

Display the network.

analyzeNetwork(net)

Create Pix2PixHD Generator with Batch Normalization

Specify the network input size for 32-channel data of size 512-by-1024 pixels.

inputSize = [512 1024 32]; 

Create a pix2pixHD generator network that performs batch normalization after each convolution.

net = pix2pixHDGlobalGenerator(inputSize,"Normalization","batch")

net = 
  dlnetwork with properties:

         Layers: [84×1 nnet.cnn.layer.Layer]
    Connections: [92×2 table]
     Learnables: [110×3 table]
          State: [54×3 table]
     InputNames: {'GlobalGenerator_inputLayer'}
    OutputNames: {'GlobalGenerator_fActivation'}
    Initialized: 1

  View summary with summary.

Display the network.

analyzeNetwork(net)

Input Arguments

inputSize — Network input size

3-element vector of positive integers

Network input size, specified as a 3-element vector of positive integers. inputSize has the form [H W C], where H is the height, W is the width, and C is the number of channels.

Example: [28 28 3] specifies an input size of 28-by-28 pixels for a 3-channel image.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: net = pix2pixHDGlobalGenerator(inputSize,NumFiltersInFirstBlock=32) creates a network with 32 filters in the first convolution layer.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: net = pix2pixHDGlobalGenerator(inputSize,"NumFiltersInFirstBlock",32) creates a network with 32 filters in the first convolution layer.

NumDownsamplingBlocks — Number of downsampling blocks

4 (default) | positive integer

Number of downsampling blocks in the network encoder module, specified as a positive integer. In total, the network downsamples the input by a factor of 2^NumDownsamplingBlocks. The decoder module consists of the same number of upsampling blocks.

NumFiltersInFirstBlock — Number of filters in first convolution layer

64 (default) | positive even integer

Number of filters in the first convolution layer, specified as a positive even integer.

NumOutputChannels — Number of output channels

3 (default) | positive integer

Number of output channels, specified as a positive integer.

FilterSizeInFirstAndLastBlocks — Filter size in first and last convolution layers

7 (default) | positive odd integer | 2-element vector of positive odd integers

Filter size in the first and last convolution layers of the network, specified as a positive odd integer or 2-element vector of positive odd integers of the form [height width]. When you specify the filter size as a scalar, the filter has equal height and width.

FilterSizeInIntermediateBlocks — Filter size in intermediate convolution layers

3 (default) | 2-element vector of positive odd integers | positive odd integer

Filter size in intermediate convolution layers, specified as a positive odd integer or 2-element vector of positive odd integers of the form [height width]. The intermediate convolution layers are the convolution layers excluding the first and last convolution layer. When you specify the filter size as a scalar, the filter has identical height and width. Typical values are between 3 and 7.

NumResidualBlocks — Number of residual blocks

9 (default) | positive integer

Number of residual blocks, specified as a positive integer.

ConvolutionPaddingValue — Style of padding

"symmetric-exclude-edge" (default) | "symmetric-include-edge" | "replicate" | numeric scalar

Style of padding used in the network, specified as one of these values.

PaddingValue	Description	Example
Numeric scalar	Pad with the specified numeric value	$$\left[\begin{array}{ccc}3& 1& 4\\ 1& 5& 9\\ 2& 6& 5\end{array}\right]\to \left[\begin{array}{ccccccc}2& 2& 2& 2& 2& 2& 2\\ 2& 2& 2& 2& 2& 2& 2\\ 2& 2& 3& 1& 4& 2& 2\\ 2& 2& 1& 5& 9& 2& 2\\ 2& 2& 2& 6& 5& 2& 2\\ 2& 2& 2& 2& 2& 2& 2\\ 2& 2& 2& 2& 2& 2& 2\end{array}\right]$$
"symmetric-include-edge"	Pad using mirrored values of the input, including the edge values	$$\left[\begin{array}{ccc}3& 1& 4\\ 1& 5& 9\\ 2& 6& 5\end{array}\right]\to \left[\begin{array}{ccccccc}5& 1& 1& 5& 9& 9& 5\\ 1& 3& 3& 1& 4& 4& 1\\ 1& 3& 3& 1& 4& 4& 1\\ 5& 1& 1& 5& 9& 9& 5\\ 6& 2& 2& 6& 5& 5& 6\\ 6& 2& 2& 6& 5& 5& 6\\ 5& 1& 1& 5& 9& 9& 5\end{array}\right]$$
"symmetric-exclude-edge"	Pad using mirrored values of the input, excluding the edge values	$$\left[\begin{array}{ccc}3& 1& 4\\ 1& 5& 9\\ 2& 6& 5\end{array}\right]\to \left[\begin{array}{ccccccc}5& 6& 2& 6& 5& 6& 2\\ 9& 5& 1& 5& 9& 5& 1\\ 4& 1& 3& 1& 4& 1& 3\\ 9& 5& 1& 5& 9& 5& 1\\ 5& 6& 2& 6& 5& 6& 2\\ 9& 5& 1& 5& 9& 5& 1\\ 4& 1& 3& 1& 4& 1& 3\end{array}\right]$$
"replicate"	Pad using repeated border elements of the input	$$\left[\begin{array}{ccc}3& 1& 4\\ 1& 5& 9\\ 2& 6& 5\end{array}\right]\to \left[\begin{array}{ccccccc}3& 3& 3& 1& 4& 4& 4\\ 3& 3& 3& 1& 4& 4& 4\\ 3& 3& 3& 1& 4& 4& 4\\ 1& 1& 1& 5& 9& 9& 9\\ 2& 2& 2& 6& 5& 5& 5\\ 2& 2& 2& 6& 5& 5& 5\\ 2& 2& 2& 6& 5& 5& 5\end{array}\right]$$

UpsampleMethod — Method used to upsample activations

"transposedConv" (default) | "bilinearResize" | "pixelShuffle"

Method used to upsample activations, specified as one of these values:

• "transposedConv" — Use a transposedConv2dLayer (Deep Learning Toolbox) with a stride of [2 2]

• "bilinearResize" — Use a convolution2dLayer (Deep Learning Toolbox) with a stride of [1 1] followed by a resize2dLayer with a scale of [2 2]

• "pixelShuffle" — Use a convolution2dLayer (Deep Learning Toolbox) with a stride of [1 1] followed by a depthToSpace2dLayer with a block size of [2 2]

Data Types: char | string

ConvolutionWeightsInitializer — Weight initialization used in convolution layers

"narrow-normal" (default) | "glorot" | "he" | function

Weight initialization used in convolution layers, specified as "glorot", "he", "narrow-normal", or a function handle. For more information, see Specify Custom Weight Initialization Function (Deep Learning Toolbox).

ActivationLayer — Activation function

"relu" (default) | "leakyRelu" | "elu" | layer object

Activation function to use in the network, specified as one of these values. For more information and a list of available layers, see Activation Layers (Deep Learning Toolbox).

• "relu" — Use a reluLayer (Deep Learning Toolbox)

• "leakyRelu" — Use a leakyReluLayer (Deep Learning Toolbox) with a scale factor of 0.2

• "elu" — Use an eluLayer (Deep Learning Toolbox)

• A layer object

FinalActivationLayer — Activation function after final convolution

"tanh" (default) | "sigmoid" | "softmax" | "none" | layer object

Activation function after the final convolution layer, specified as one of these values. For more information and a list of available layers, see Activation Layers (Deep Learning Toolbox).

• "tanh" — Use a tanhLayer (Deep Learning Toolbox)

• "sigmoid" — Use a sigmoidLayer (Deep Learning Toolbox)

• "softmax" — Use a softmaxLayer (Deep Learning Toolbox)

• "none" — Do not use a final activation layer

• A layer object

NormalizationLayer — Normalization operation

"instance" (default) | "none" | "batch" | layer object

Normalization operation to use after each convolution, specified as one of these values. For more information and a list of available layers, see Normalization Layers (Deep Learning Toolbox).

• "instance" — Use an instanceNormalizationLayer (Deep Learning Toolbox)

• "batch" — Use a batchNormalizationLayer (Deep Learning Toolbox)

• "none" — Do not use a normalization layer

• A layer object

Dropout — Probability of dropout

0 (default) | number in the range [0, 1]

Probability of dropout, specified as a number in the range [0, 1]. If you specify a value of 0, then the network does not include dropout layers. If you specify a value greater than 0, then the network includes a dropoutLayer (Deep Learning Toolbox) in each residual block.

NamePrefix — Prefix to all layer names

"GlobalGenerator_" (default) | string | character vector

Prefix to all layer names in the network, specified as a string or character vector.

Data Types: char | string

Output Arguments

net — pix2pixHD generator network

dlnetwork object

Pix2pixHD generator network, returned as a dlnetwork (Deep Learning Toolbox) object.

More About

pix2pixHD Generator Network

A pix2pixHD generator network consists of an encoder module followed by a decoder module. The default network follows the architecture proposed by Wang et. al. [1].

The encoder module downsamples the input by a factor of 2^NumDownsamplingBlocks. The encoder module consists of an initial block of layers, NumDownsamplingBlocks downsampling blocks, and NumResidualBlocks residual blocks. The decoder module upsamples the input by a factor of 2^NumDownsamplingBlocks. The decoder module consists of NumDownsamplingBlocks upsampling blocks and a final block.

The table describes the blocks of layers that comprise the encoder and decoder modules.

Block Type	Layers	Diagram of Default Block
Initial block	An imageInputLayer (Deep Learning Toolbox) A convolution2dLayer (Deep Learning Toolbox) with a stride of [1 1] and a filter size of FilterSizeInFirstAndLastBlocks An optional normalization layer, specified by the NormalizationLayer name-value argument. An activation layer specified by the ActivationLayer name-value argument.
Downsampling block	A convolution2dLayer (Deep Learning Toolbox) with a stride of [2 2] to perform downsampling. The convolution layer has a filter size of FilterSizeInIntermediateBlocks. An optional normalization layer, specified by the NormalizationLayer name-value argument. An activation layer specified by the ActivationLayer name-value argument.
Residual block	A convolution2dLayer (Deep Learning Toolbox) with a stride of [1 1] and a filter size of FilterSizeInIntermediateBlocks. An optional normalization layer, specified by the NormalizationLayer name-value argument. An activation layer specified by the ActivationLayer name-value argument. An optional dropoutLayer (Deep Learning Toolbox). By default, residual blocks omit a dropout layer. Include a dropout layer by specifying the Dropout name-value argument as a value in the range (0, 1]. A second convolution2dLayer (Deep Learning Toolbox). An optional second normalization layer. An additionLayer (Deep Learning Toolbox) that provides a skip connection between every block.
Upsampling block	An upsampling layer that upsamples by a factor of 2 according to the UpsampleMethod name-value argument. The convolution layer has a filter size of FilterSizeInIntermediateBlocks. An optional normalization layer, specified by the NormalizationLayer name-value argument. An activation layer specified by the ActivationLayer name-value argument.
Final block	A convolution2dLayer (Deep Learning Toolbox) with a stride of [1 1] and a filter size of FilterSizeInFirstAndLastBlocks. An optional activation layer specified by the FinalActivationLayer name-value argument.

Tips

• You can create the discriminator network for pix2pixHD by using the patchGANDiscriminator function.

• Train the pix2pixHD GAN network using a custom training loop.

References

[1] Wang, Ting-Chun, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. "High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs." In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8798–8807. Salt Lake City, UT, USA: IEEE, 2018. https://doi.org/10.1109/CVPR.2018.00917.

Version History

Introduced in R2021a