The Basics#

Deploying a SNN on devices based on DYNAP-CNN technology involves several steps such as model architecture conversion, placement and parameter quantization. This package automates this process for the end-user and enables quick deployment and testing of your models on to the dev-kits.

TLDR;#

A short (and perhaps the quickest path) to deploying your model on one of our dev-kits is shown in the example below.

import torch
import torch.nn as nn
from typing import List
from sinabs.from_torch import from_model
from sinabs.backend.dynapcnn import DynapcnnNetwork

ann = nn.Sequential(
    nn.Conv2d(1, 20, 5, 1, bias=False),
    nn.ReLU(),
    nn.AvgPool2d(2,2),
    nn.Conv2d(20, 32, 5, 1, bias=False),
    nn.ReLU(),
    nn.AvgPool2d(2,2),
    nn.Conv2d(32, 128, 3, 1, bias=False),
    nn.ReLU(),
    nn.AvgPool2d(2,2),
    nn.Flatten(),
    nn.Linear(128, 500, bias=False),
    nn.ReLU(),
    nn.Linear(500, 10, bias=False),
)

# Load your weights or train the model
ann.load_state_dict(torch.load("model_params.pt"), map_location="cpu")

# Convert your model to SNN
sinabs_model = from_model(ann, add_spiking_output=True)  # Your sinabs SNN model

# Convert your SNN to `DynapcnnNetwork`
hw_model = DynapcnnNetwork(
    sinabs_model.spiking_model,
    discretize=True,
    input_shape=(1, 28, 28)
)

# Deploy model to a dev-kit
hw_model.to(device="dynapcnndevkit:0")

# Send events to chip
events_in: List["Spike"] =  ... # Load your events
events_out = hw_model(events_in)

# Analyze your output events
...

Model conversion to DYNAP-CNN core structure#

DYNAP-CNN based chips like DYNAP-CNN DevKit or Speck series comprise several cores or layers. Each of these layers comprises three functionalities:

2D Convolution
Integrate and Fire Neurons
Sum Pooling

Accordingly, the DynapcnnLayer class is a sequential model with three layers:

conv_layer
spk_layer
pool_layer

In order to deploy a model onto these chips, the network structure needs to be converted into a sequence of DynapcnnLayers. The DynapcnnNetwork class automates this model conversion from a sequential sinabs spiking neural network into a sequence of DynapcnnLayers. In addition, it also descretizes/quantizes the parameters to 8 bits (according to the chip specifications).

Layer conversion#

Often, the network architectures comprise of layers such as AvgPool2d, Flatten or Linear. The chips do not support these layers in their original form and require some transformation. For instance, while AvgPool2d works in simulations, spikes cannot really be averaged. Instead SumPool2d is a better fit for spiking networks. Similarly, a Linear layer can be replaced with Conv2d with a kernel size 1x1 such that it is compatible with DynapcnnLayer. Instantiating DynapcnnNetwork takes care of all such conversions.

Parameter quantization#

The hardware suppports fixed point weights (8 bits for weights and 16 bits for membrane potentials for instance). The models trained in pytorch typically use floating point representation of weights. Setting discretize=True converts the model parameters from floating point to fixed point representation while preserving the highest possible precision.

Device selection#

The device naming is inspired by pytorch device naming convention ie DEVICE_TYPE:INDEX. dynapcnndevkit:0 refers to the first Dynapcnn DevKit available. If there are multiple devices of the same kind connected to the PC, then they are referred by higher incremental indices.

To see all the recognized devices, please have a look at the sinabs.backend.dynapcnn.io.device_types

from sinabs.backend.dynapcnn import io
print(io.device_types)

List of devices currently recognized by samna#

.. note:: Not all of these are supported by this plugin and not all of these are compatible with DYNAP-CNN

# A map of all device types and their corresponding samna `device_name`
device_types = {
    "speck": "speck",
    "dynapse2": "DYNAP-SE2 DevBoard",
    "dynapse2_stack": "DYNAP-SE2 Stack",
    "speck2devkit": "Speck2DevKit",
    "dynapse1devkit": "Dynapse1DevKit",
    "davis346": "Davis 346",
    "davis240": "Davis 240",
    "dvxplorer": "DVXplorer",
    "pollendevkit": "PollenDevKit",
    "dynapcnndevkit": "DynapcnnDevKit",
}

You can also get a list of all supported devices currently connected/available by running the following lines:

from sinabs.backend.dynapcnn import io
io.get_all_samna_devices()

Finally to get a list of all the supported devices that this plugin supports and allows you to port your models, you can inspect ChipFactory.supported_devices

from sinabs.backend.dynapcnn.chip_factory import ChipFactory
ChipFactory.supported_devices

Placement of layers on device cores#

A sequence of DynapcnnLayers (i.e. a model that has been converted to DynapcnnNetwork) is ready to be mapped onto the chip cores/layers. This is done by placing each layer of the model onto a layer on the chip. The exact placement is specified by the parameter chip_layers_ordering.

This is an important parameter because each layer of the model has a certain memory requirement for kernel parameters and neurons. The chip layers are not homogenous and have a limited amount of memory allocated to each. Consequently, not all layers on the chip will be compatible with each layer in the model.

If the chip_layers_ordering is set to "auto", the network is going to be mapped onto the chip based on a placement algorithm. If the algorithm is unable to place the model onto the chip, it will throw an error message.

Some methods helpful for debugging if you run into problems are ConfigBuilder.get_valid_mapping() and the object ConfigBuilder.get_constraints().

After successfully mapping a model, the chip_layers_ordering can be inspected by executing DynapcnnNetwork.chip_layers_ordering.

Porting model to device#

DynapcnnNetwork class has an API similar to that of native pytorch and its .to method. Similar to porting a model to cpu with model.to("cpu") and GPU with model.to("cuda:0") you can also port your DynapcnnCompatibleModel to a chip with model.to("dynapcnndevkit:0").

You can also specify a few additional parameters as shown below.

hw_model.to(
    device="dynapcnndevkit:0",
    chip_layers_ordering="auto",  # default value is "auto"
    monitor_layers=[-1],
    config_modifier=config_modifier,
)

As shown in the above example, you can specify which layers are to be monitored. Note here that the layer indices are that of the model. For instance -1 refers to the last layer of the model. In addition, for advanced users, a config_modifier can be passed. This is a callable or a function that takes a config object and does any custom setting changes before writing this on the chip.

See the __doc__ string for further details on each of these parameters.

Sending and receiving spikes#

You can send a pre-defined sequence of events to the chip using the model’s forward method as you do with standard pytorch models.

events_out = hw_model(events_in)

The events_in has to be a list of Spike objects corresponding to the chip in use. Similarly events_out will also be a list of Spike objects corresponding to the events generated from the monitored layers. Each Spike event has attributes layer, x, y, feature and timestamp.

Monitoring layer activity#

In order to monitor the spiking activity of a given layer, the corresponding layer has to be specified in the monitor_layers parameter. In most use cases, you will want to monitor the activity of the last layer of the model and so this parameter will be set to [-1].

Once enabled, all the corresponding spikes will be found in the sequence of returned events from the chip. The samna_output_buffer accumulates all the events sent out by the chip, including those from the monitored layers. The events from this buffer are then read out and returned to the user on calling the forward method.

The Basics

Contents