Dynapcnn Visualizer#
Introduction#
Currently the development kits that SynSense provides are mainly used for benchmarking purposes. However, through our backend library samna
we support processing of events and interpretation of the output in a streaming fashion. samna
API however is mainly designed and developed for low-level communication with the chips. This sometimes makes it tricky to work with for higher-level functions. samna
also has a package called samnagui
with which we can do some visualization. Internally, we use it for testing of our models in real-time and real-life conditions.
In order to give users an easy access without having to deal with a lot of boilerplate code and burdensome logic, we implemented a visualizer class.
Available plots in samnagui
#
There are 4 different plots in samnagui
that are available for use.
Activity Plot: A plot that visualizes the events produced by the on-chip sensor.
Line Plot: A plot that allows visualization of the events produced by the model running on the chip. Additionally this is used when we display power consumption measurements.
Image Plot: A plot that allows you to display an image. Internally we have used this plot to display an image denoting a predicted output class.
Documentation is available under the following link: samna.ui documentation
Useful samna
nodes for visualizing and Just-In-Time (JIT) compiled nodes.#
Nodes#
samna
also comes with implementation of several nodes that are useful for communicating between a samnagui
visualizer and the chip.
Events that are related to DVS events from the sensor are:
DvsEventDecimate
: EliminatingL
out ofM
events.
set_decimation_fraction(M: int, L: int)
DvsEventRescale
: Rescale events.x / width
andy / height
.
set_rescaling_coefficients(width_coeff: int, height_coeff: int)
DvsToVizConverter
: Converts events from sensor to visualization events.
The ones that are connected to the chip output spikes:SpikeCollectionNode
: Picks output spikes at intervals (ms) and makes events that can be used out of them.
set_interval_milli_sec(interval: int)
SpikeCountNode
: Counts how many events from each output received amongfeature_count
events and outputs a visualizer event.
set_feature_count(feature_count: int)
MajorityReadoutNode
: Among the events produced bySpikeCollectionNode
selects the most active output channel. Used alongside theImagePlot
.
These nodes are available for different chips. Further documentation can be found in the following link: samna-available-filters
Just-In-Time compiled nodes#
The nodes mentioned above are also available for any devboard under samna.graph.Jit{nameOfNode}
.
Dynapcnn Visualizer#
DynapcnnVisualizer
class uses the nodes and plots mentioned above in order to use it and provides an intuitive interface. An example code-snippet can be found below:
Setup#
import os
current_folder_path = str(os.path.join(os.getcwd()))
file_tokens = current_folder_path.split("/")[:-3]
params_path = os.path.join( os.path.join("/", *file_tokens), "examples/dvs_gesture_params.pt")
icons_folder_path = os.path.join( os.path.join("/", *file_tokens), "examples/icons/")
Import requirements#
import torch
import torch.nn as nn
from sinabs.from_torch import from_model
from sinabs.backend.dynapcnn import DynapcnnNetwork
Define model#
ann = nn.Sequential(
nn.Conv2d(2, 16, kernel_size=2, stride=2, bias=False),
nn.ReLU(),
# core 1
nn.Conv2d(16, 16, kernel_size=3, padding=1, bias=False),
nn.ReLU(),
nn.AvgPool2d(kernel_size=2, stride=2),
# core 2
nn.Conv2d(16, 32, kernel_size=3, padding=1, bias=False),
nn.ReLU(),
nn.AvgPool2d(kernel_size=2, stride=2),
# core 7
nn.Conv2d(32, 32, kernel_size=3, padding=1, bias=False),
nn.ReLU(),
nn.AvgPool2d(kernel_size=2, stride=2),
# core 4
nn.Conv2d(32, 64, kernel_size=3, padding=1, bias=False),
nn.ReLU(),
nn.AvgPool2d(kernel_size=2, stride=2),
# core 5
nn.Conv2d(64, 64, kernel_size=3, padding=1, bias=False),
nn.ReLU(),
nn.AvgPool2d(kernel_size=2, stride=2),
# core 6
nn.Dropout2d(0.5),
nn.Conv2d(64, 256, kernel_size=2, bias=False),
nn.ReLU(),
# core 3
nn.Dropout2d(0.5),
nn.Flatten(),
nn.Linear(256, 128, bias=False),
nn.ReLU(),
# core 8
nn.Linear(128, 11, bias=False),
)
Load model weights from the example folder#
load_result = ann.load_state_dict(torch.load(params_path), strict=False)
Convert to SNN#
When converting to a DynapcnnNetwork
using sinabs
and sinabs-dynapcnn
, please use the following flags:
dvs_input
=True
has to be set so that the model can receive input from the on-board sensor or an external DVS sensor.discretize
=True
has to be set so that the model can be ported to the chip.
sinabs_model = from_model(ann, add_spiking_output=True, batch_size=1)
input_shape = (2, 128, 128)
hardware_compatible_model = DynapcnnNetwork(
sinabs_model.spiking_model.cpu(),
dvs_input=True,
discretize=True,
input_shape=input_shape
)
Port the model to the chip#
Port the model on the chip. Important!
monitor_layers
=["dvs", -1]
Other layers can also be monitored for other purposes, butmonitor_layers
should contain at leastdvs
and-1
so that the visualizer has access to thedvs
layer and-1
for the output layer of the network.
hardware_compatible_model.to(
device="speck2edevkit", # speck2edevkit
monitor_layers=["dvs", -1], # Last layer
chip_layers_ordering="auto"
)
Use DynapcnnVisualizer#
In order to visualize the class outputs as images, we need to get the images. The images should be passed in the same order as the output layer of the network. Important!
If you want to visualize power measurements during streaming inference, set
add_power_monitor_plot
=True
.If you want to visualize readout images as class predictions during streaming you need to pass
add_readout_plot
=True
.If you don’t want to visualize spike counts of output classes as line graphs over time during streaming you need to pass
add_spike_count_plot
=False
.In order to show a prediction for each
N
milliseconds, set the parameterspike_collection_interval
=N
.In order to show the images, the paths of these images should be passed to
readout_images
parameter.In order to show a prediction only if there are more than a
threshold
number of events from that output, set thereadout_prediction_threshold
=threshold
.In order to default to a certain class (in
DVSGesture
example, we use the lastother
class) setreadout_default_return_value
=class_idx
(int).In order to limit the prediction between some thresholds (i.e. it is meaningless to make a prediction with too low and too high values) set
readout_default_threshold_low
andreadout_default_threshold_high
parameters.In all the chips except for
DynapcnnDevkit
you can monitor the power consumption in 5 different rows. These rows areio
,logic
,ram
,pixel-digital
andpixel-analog
.io
: Power consumption of IO unit on the chip.logic
: Power consumption caused by operations in the chip and communication with spikes.ram
: Power consumption ofmemory
used to storestates
andweights
.pixel-digital
: Power consumption of the digital part (i.e. communication via AER) of pixels.pixel-analog
: Power consumtpion of the analog part. (i.e. pixels. ).
DynapcnnDevkit
does not come with an on-board Dvs sensor, thus does not support the last two. If you want only the first 3 columns, passpower_monitor_number_of_items
=3
. If you want columns pass5
.If you follow the naming conventions for readout images (i.e.
{labelidx}_{labelname}.png
) the feature names are going to be parsed from the image names. However, you can also pass labels manually usingfeature_names
=["class01", "class02", ...]
. That can be used also while labelingSpikeCountPlot
legend, whenReadoutLayer
is not preferred.The last layer feature count is going to be automatically extracted from the model. However you can pass it manually
feature_count
=count
. This can be useful for plotting purposes, when there are other classes that you did not take into account in the model you trained.extra_arguments
: You can pass the function names and variables for the individual plots forspike_count
,readout
andpower_measurement
plots. Available function names and argument types can be found in the following link: here
from sinabs.backend.dynapcnn.dynapcnn_visualizer import DynapcnnVisualizer
visualizer = DynapcnnVisualizer(
window_scale=(4, 8),
dvs_shape=(128, 128),
add_power_monitor_plot=True,
add_readout_plot=True,
spike_collection_interval=500,
readout_images=sorted([os.path.join(icons_folder_path, f) for f in os.listdir(icons_folder_path)])
)
Finally connect your model to the visualizer#
Please note that DynapcnnVisualizer
class is powered by Just-In-Time (JIT) compilation in C++. If you are running this on a computer, which does not have a powerful CPU this may take a while. You will see the window spawning, but you will not see anything displayed on it until the compilation is complete.
visualizer.connect(hardware_compatible_model)
Try it yourself#
The example script that runs the visualizer can be found under /examples/visualizer/gesture_viz.py
.
MacOS users#
Due to the difference in the behaviour of python’s multiprocessing library on MacOS, you should run the examples/visualizer/gesture_viz.py
script with -i
flag. python -i /examples/visualizer/gesture_viz.py
.