Run Sequence-to-Sequence Classification on FPGAs by Using Deep Learning HDL Toolbox
This example shows how to create, compile, and deploy a long short-term memory (LSTM) network trained on accelerometer data from human movement by using the Deep Learning HDL Toolbox™ Support Package for Xilinx FPGA and SoC. Use the deployed network to classify human activity based on sequence input data. Use MATLAB® to retrieve the prediction results from the target device.
The network attached to this example was trained using the Sequence-to-Sequence Classification Using Deep Learning. This example uses sensor data obtained from a smartphone worn on the body. This example deploys an LSTM network trained to recognize the activity of the wearer given time series data that represents accelerometer readings in three different directions. The graphs below show the raw data for these accelerometer readings over time and the resulting classifications. The training data contains time series data for seven people. Each sequence has three features and varies in length. The data set contains six training observations and one test observation.
Prerequisites
Xilinx® Zynq® Ultrascale+™ ZCU102 SoC development kit
Deep Learning HDL Toolbox™ Support Package for Xilinx FPGA and SoC
Deep Learning Toolbox™
Deep Learning HDL Toolbox™
Load the Pretrained Network
To load the pretrained human body movement network, enter:
load SequenceToSequenceClassification
View the layers of the network by using the analyzeNetwork
function. The function returns a graphical representation of the network and detailed parameter settings of the layers in the network.
analyzeNetwork(net)
Define FPGA Board Interface
Define the target FPGA board programming interface by using the dlhdl.Target
object. Specify that the interface is for a Xilinx board with an Ethernet interface.
To create the target object, enter:
hTarget = dlhdl.Target('Xilinx','Interface','Ethernet');
To use the JTAG interface, install Xilinx™ Vivado™ Design Suite 2020.2. To set the Xilinx Vivado tool path, enter:
hdlsetuptoolpath('ToolName', 'Xilinx Vivado', 'ToolPath', 'C:\Xilinx\Vivado\2020.2\bin\vivado.bat');
Prepare Network for Deployment
Prepare the network for deployment by creating a dlhdl.Workflow
object. Specify the network and bitstream name. Ensure that the bitstream name matches the data type and FPGA board. In this example the target FPGA board is the Xilinx ZCU102 SOC board. The bitstream uses a single data type.
hW = dlhdl.Workflow('network', net, 'Bitstream', 'zcu102_lstm_single','Target',hTarget);
To run the example in a Xilinx ZC706 board, enter:
hW = dlhdl.Workflow('Network', snet, 'Bitstream', 'zc706_lstm_single','Target',hTarget);
Compile Network
Run the compile
method of the dlhdl.Workflow
object to compile the network and generate the instructions, weights, and biases for deployment. The total number of frames exceeds the default value of 30. Set the InputFrameNumberLimit
name-value argument to 10000
to run predictions in chunks of 10,000 frames to prevent timeouts.
dn = compile(hW,'InputFrameNumberLimit',10000)
### Compiling network for Deep Learning FPGA prototyping ... ### Targeting FPGA bitstream zcu102_lstm_single. ### The network includes the following layers: 1 'sequenceinput' Sequence Input Sequence input with 3 dimensions (SW Layer) 2 'lstm' LSTM LSTM with 200 hidden units (HW Layer) 3 'fc' Fully Connected 5 fully connected layer (HW Layer) 4 'softmax' Softmax softmax (SW Layer) 5 'classoutput' Classification Output crossentropyex with 'Dancing' and 4 other classes (SW Layer) ### Notice: The layer 'sequenceinput' with type 'nnet.cnn.layer.ImageInputLayer' is implemented in software. ### Notice: The layer 'softmax' with type 'nnet.cnn.layer.SoftmaxLayer' is implemented in software. ### Notice: The layer 'classoutput' with type 'nnet.cnn.layer.ClassificationOutputLayer' is implemented in software. ### Compiling layer group: lstm.wi ... ### Compiling layer group: lstm.wi ... complete. ### Compiling layer group: lstm.wo ... ### Compiling layer group: lstm.wo ... complete. ### Compiling layer group: lstm.wg ... ### Compiling layer group: lstm.wg ... complete. ### Compiling layer group: lstm.wf ... ### Compiling layer group: lstm.wf ... complete. ### Compiling layer group: fc ... ### Compiling layer group: fc ... complete. ### Allocating external memory buffers: offset_name offset_address allocated_space _______________________ ______________ ________________ "InputDataOffset" "0x00000000" "4.0 MB" "OutputResultOffset" "0x00400000" "4.0 MB" "SchedulerDataOffset" "0x00800000" "4.0 MB" "SystemBufferOffset" "0x00c00000" "20.0 MB" "InstructionDataOffset" "0x02000000" "4.0 MB" "FCWeightDataOffset" "0x02400000" "4.0 MB" "EndOffset" "0x02800000" "Total: 40.0 MB" ### Network compilation complete.
dn = struct with fields:
weights: [1×1 struct]
instructions: [1×1 struct]
registers: [1×1 struct]
syncInstructions: [1×1 struct]
constantData: {}
ddrInfo: [1×1 struct]
Program Bitstream onto FPGA and Download Network Weights
To deploy the network on the Xilinx ZCU102 SoC hardware, run the deploy
method of the dlhdl.Workflow
object. This function uses the output of the compile
function to program the FPGA board and download the network weights and biases. The deploy
function starts programming the FPGA device and displays progress messages, and the required time to deploy the network.
deploy(hW)
### Programming FPGA Bitstream using Ethernet... ### Attempting to connect to the hardware board at 192.168.1.101... ### Connection successful ### Programming FPGA device on Xilinx SoC hardware board at 192.168.1.101... ### Copying FPGA programming files to SD card... ### Setting FPGA bitstream and devicetree for boot... # Copying Bitstream zcu102_lstm_single.bit to /mnt/hdlcoder_rd # Set Bitstream to hdlcoder_rd/zcu102_lstm_single.bit # Copying Devicetree devicetree_dlhdl.dtb to /mnt/hdlcoder_rd # Set Devicetree to hdlcoder_rd/devicetree_dlhdl.dtb # Set up boot for Reference Design: 'AXI-Stream DDR Memory Access : 3-AXIM' ### Rebooting Xilinx SoC at 192.168.1.101... ### Reboot may take several seconds... ### Attempting to connect to the hardware board at 192.168.1.101... ### Connection successful ### Programming the FPGA bitstream has been completed successfully. ### Resetting network state. ### Loading weights to FC Processor. ### FC Weights loaded. Current time is 09-May-2023 10:17:00
Load Human Activity Test Data
Load the test data and classify the activity at each time step. Each sequence has three features and varies in length. The three features correspond to the accelerometer readings in three different directions.
Load the human activity test data. XTest
contains a single sequence of dimension 3. YTest
contains a sequence of categorical labels that correspond to the activity at each time step.
load HumanActivityTest numFeatures = 3; figure plot(XTest{1}') xlabel("Time Step") legend("Feature " + (1:numFeatures)) title("Test Data")
Run the Prediction
Classify the test data by using the classify
function.
YPred = classify(hW.Network, XTest{1});
Calculate the accuracy of the prediction.
acc = sum(YPred == YTest{1})./numel(YTest{1})
acc = 0.9995
Compare the predictions with the test data by using a plot.
figure plot(YPred,'.-') hold on plot(YTest{1}) hold off xlabel("Time Step") ylabel("Activity") title("Predicted Activities") legend(["Predicted" "Test Data"])
Compare this graph to the output of the predict
method.
Run the predict
method of the dlhdl.Workflow
object, to retrieve the hardware prediction results.
predictions = hW.predict(XTest{1}(:,1:10000),Profile='on');
### Resetting network state. ### Finished writing input activations. ### Running a sequence of length 10000. Deep Learning Processor Profiler Performance Results LastFrameLatency(cycles) LastFrameLatency(seconds) FramesNum Total Latency Frames/s ------------- ------------- --------- --------- --------- Network 76164 0.00035 10000 763547289 2881.3 memSeparator_0 88 0.00000 lstm.wi 17896 0.00008 lstm.wo 18007 0.00008 lstm.wg 17996 0.00008 lstm.wf 18027 0.00008 lstm.sigmoid_1 285 0.00000 lstm.sigmoid_3 267 0.00000 lstm.tanh_1 287 0.00000 lstm.sigmoid_2 277 0.00000 lstm.multiplication_2 427 0.00000 lstm.multiplication_1 427 0.00000 lstm.c_add 411 0.00000 lstm.tanh_2 301 0.00000 lstm.multiplication_3 411 0.00000 fc 1057 0.00000 * The clock frequency of the DL processor is: 220MHz
predictions = horzcat(predictions, hW.predict(XTest{1}(:,10001:20000),Profile='on'));
### Resetting network state. ### Finished writing input activations. ### Running a sequence of length 10000.
predictions = horzcat(predictions, hW.predict(XTest{1}(:,20001:30000),Profile='on')); predictions = horzcat(predictions, hW.predict(XTest{1}(:,30001:40000),Profile='on')); predictions = horzcat(predictions, hW.predict(XTest{1}(:,40001:50000),Profile='on')); predictions = horzcat(predictions, hW.predict(XTest{1}(:,50001:end),Profile='on')); save("hardwarepredictions.mat","predictions") indices = []; actions = []; for x = 1:length(YPred) [r,i] = max(predictions(:,x)); indices = [indices i]; switch i case 1 actions = [actions categorical("Dancing")]; case 2 actions = [actions categorical("Running")]; case 5 actions = [actions categorical("Walking")]; case 4 actions = [actions categorical("Standing")]; case 3 actions = [actions categorical("Sitting")]; end end
Plot the comparison between the FPGA board predictions and test data.
figure plot(actions,'.-') hold on plot(YTest{1}) hold off xlabel("Time Step") ylabel("Activity") title("Predicted Activities") legend(["Predicted" "Test Data"])
The hardware-predicted activities are similar to the activities classified by the classify
function.
See Also
dlhdl.Workflow
| dlhdl.Target
| compile
| deploy
| predict
| predictAndUpdateState
| resetState