# Available Network Architecture ## Brief Introduction of the DynapCNN Core Our devkit has 9 `DynapCNN Core`s, each core can executes `Asynchronous Convolution`. Each core also have the following features: 1. Each core has an unique index number, the first core's index starts from 0. So the index is in range [0, 8). 2. Each core can **only** define **2 destination cores** as its output destination core(layer). See [detail.](https://synsense-sys-int.gitlab.io/samna/0.46.1/reference/speck2f/configuration/index.html#samna.speck2f.configuration.CnnLayerDestination) 3. You can define multiple cores to have one **same destination core**. Technically,you can use this feature to achieve a `short-cut`(residual connection) like ResNet does. See [detail.](https://synsense-sys-int.gitlab.io/samna/0.46.1/reference/speck2f/configuration/index.html#samna.speck2f.configuration.CnnLayerDestination.layer) 4. Each core can optionally apply a `sum-pooling` operation before feeding the output events into the destination core. See [detail.](https://synsense-sys-int.gitlab.io/samna/0.46.1/reference/speck2f/configuration/index.html#samna.speck2f.configuration.CnnLayerDestination.pooling) 5. Each core can optionally apply a `channel-shift` operation before feeding the output events into the destination core. Technically, you can use this feature to `concatenate` two cores' output events among the channel axis. See [detail.](https://synsense-sys-int.gitlab.io/samna/0.46.1/reference/speck2f/configuration/index.html#samna.speck2f.configuration.CnnLayerDestination.feature_shift) 6. You can define the order of cores/layers by defining the `destination` index of each core, i.e. the order of the 9 layers on the chip can be customized by yourself. We already have this feature in sinabs-dynapcnn. When you deploy an SNN to the devkit, you can do: ```python from sinabs.backend.dynapcnn import DynapcnnNetwork # suppose snn is a 4-layers network dynapcnn = DynapcnnNetwork(snn=snn, discretize=True, dvs_input=False, input_shape=input_shape) # deploy the SNN dynapcnn.to(devcie="your device", chip_layers_ordering="auto") # or you can do dynapcnn.to(devcie="your device", chip_layers_ordering=[0, 1, 2, 3]) # or dynapcnn.to(devcie="your device", chip_layers_ordering=[2, 5, 7, 1]) ``` ## What network structure can I define? `Sinabs` can parse a `torch.nn.Sequential` like architecture, so it is recommended to use a `Sequential` like network. As of `v3.1.0`, we released a network graph extraction feature that helps users deploy their networks with more complex architectures into the devkit. Our `Speck` chip, in fact, supports branched architectures. With the graph extraction feature, we support a range of network structures, as shown below: A network with a merge and a split: ![A network with a merge and a split](imgs/network-with-merge-and-split.png) Two networks with merging outputs: ![Two networks with merging outputs](imgs/two-networks-merging-output.png) A network with residual connections: ![A network with residual connections](imgs/network-with-residual-connection.png) A more complex network: ![A more complex network](imgs/complex-network.png) Note: with the graph extracture feature it is possible to implement recurrent neural networks. However, this is not recommended or supported as it can result in deadlock on the chip. Note2: the use of two parallel network although supported by our chip was not fully considered in our sinabs implementation. ## How to make use of the graph extraction feature? For general architectures, users need to define their classes, by defining at least the `__init__` method with all the layers, as well as an appropriate `forward` method. Here is an example to define a network with a merge and a split: ```python import torch import torch.nn as nn from sinabs.activation.surrogate_gradient_fn import PeriodicExponential from sinabs.layers import IAFSqueeze, Merge, SumPool2d class SNN(nn.Module): def __init__(self, batch_size) -> None: super().__init__() # -- graph node A -- self.conv_A = nn.Conv2d(2, 4, 2, 1, bias=False) self.iaf_A = IAFSqueeze( batch_size=batch_size, min_v_mem=-1.0, spike_threshold=1.0, surrogate_grad_fn=PeriodicExponential(), ) # -- graph node B -- self.conv_B = nn.Conv2d(4, 4, 2, 1, bias=False) self.iaf2_B = IAFSqueeze( batch_size=batch_size, min_v_mem=-1.0, spike_threshold=1.0, surrogate_grad_fn=PeriodicExponential(), ) self.pool_B = SumPool2d(2, 2) # -- graph node C -- self.conv_C = nn.Conv2d(4, 4, 2, 1, bias=False) self.iaf_C = IAFSqueeze( batch_size=batch_size, min_v_mem=-1.0, spike_threshold=1.0, surrogate_grad_fn=PeriodicExponential(), ) self.pool_C = SumPool2d(2, 2) # -- graph node D -- self.conv_D = nn.Conv2d(4, 4, 2, 1, bias=False) self.iaf_D = IAFSqueeze( batch_size=batch_size, min_v_mem=-1.0, spike_threshold=1.0, surrogate_grad_fn=PeriodicExponential(), ) # -- graph node E -- self.conv_E = nn.Conv2d(4, 4, 2, 1, bias=False) self.iaf3_E = IAFSqueeze( batch_size=batch_size, min_v_mem=-1.0, spike_threshold=1.0, surrogate_grad_fn=PeriodicExponential(), ) self.pool_E = SumPool2d(2, 2) # -- graph node F -- self.conv_F = nn.Conv2d(4, 4, 2, 1, bias=False) self.iaf_F = IAFSqueeze( batch_size=batch_size, min_v_mem=-1.0, spike_threshold=1.0, surrogate_grad_fn=PeriodicExponential(), ) # -- graph node G -- self.fc3 = nn.Linear(144, 10, bias=False) self.iaf3_fc = IAFSqueeze( batch_size=batch_size, min_v_mem=-1.0, spike_threshold=1.0, surrogate_grad_fn=PeriodicExponential(), ) # -- merges -- self.merge1 = Merge() # -- falts -- self.flat_D = nn.Flatten() self.flat_F = nn.Flatten() def forward(self, x): # conv 1 - A/0 convA_out = self.conv_A(x) iaf_A_out = self.iaf_A(convA_out) # conv 2 - B/1 conv_B_out = self.conv_B(iaf_A_out) iaf_B_out = self.iaf2_B(conv_B_out) pool_B_out = self.pool_B(iaf_B_out) # conv 3 - C/2 conv_C_out = self.conv_C(pool_B_out) iaf_C_out = self.iaf_C(conv_C_out) pool_C_out = self.pool_C(iaf_C_out) # conv 4 - D/4 conv_D_out = self.conv_D(pool_C_out) iaf_D_out = self.iaf_D(conv_D_out) # fc 1 - E/3 conv_E_out = self.conv_E(pool_B_out) iaf3_E_out = self.iaf3_E(conv_E_out) pool_E_out = self.pool_E(iaf3_E_out) # fc 2 - F/6 conv_F_out = self.conv_F(pool_E_out) iaf_F_out = self.iaf_F(conv_F_out) # fc 2 - G/5 flat_D_out = self.flat_D(iaf_D_out) flat_F_out = self.flat_F(iaf_F_out) merge1_out = self.merge1(flat_D_out, flat_F_out) fc3_out = self.fc3(merge1_out) iaf3_fc_out = self.iaf3_fc(fc3_out) return iaf3_fc_out ``` ## Can I achieve a "Residual Connection" like ResNet does? Like mentioned above, "Yes, we can define a residual short-cut on the devkit". You can manually change the `samna.speck2f.configuration.CNNLayerDestination.layer` to achieve this, if you are very familiar with the `samna-configuration`. You can also make use of our network graph extraction feature, to implement residual networks. ## How can I define "Residual Connection" manually? You can also achieve "Residual Connection" by manually modify the `samna-configuration`. Let's say you want an architecture like below: ```python from torch import nn from sinabs.layers import IAFSqueeze class ResidualBlock(nn.Module): def __init__(self): super(ResidualBlock, self).__init__() self.conv1 = nn.Conv2d(1, 2, kernel_size=(1, 1), bias=False) self.iaf1 = IAFSqueeze(batch_size=1, min_v_mem=-1.0) self.conv2 = nn.Conv2d(2, 2, kernel_size=(1, 1), bias=False) self.iaf2 = IAFSqueeze(batch_size=1, min_v_mem=-1.0) self.conv3 = nn.Conv2d(2, 4, kernel_size=(1, 1), bias=False) self.iaf3 = IAFSqueeze(batch_size=1, min_v_mem=-1.0) def forward(self, x): tmp = self.conv1(x) tmp = self.iaf1(tmp) out = self.conv2(tmp) out = self.iaf2(tmp) # residual connection out += tmp out = self.conv3(out) out = self.iaf3(out) return out ``` You can write it like: ```python # define a Sequential first SNN = nn.Sequential( nn.Conv2d(1, 2, kernel_size=(1, 1), bias=False), IAFSqueeze(batch_size=1, min_v_mem=-1.0), nn.Conv2d(2, 2, kernel_size=(1, 1), bias=False), IAFSqueeze(batch_size=1, min_v_mem=-1.0), nn.Conv2d(2, 4, kernel_size=(1, 1), bias=False), IAFSqueeze(batch_size=1, min_v_mem=-1.0), ) # make samna configuration dynapcnn = DynapcnnNetwork(snn=SNN, input_shape=(1, 16, 16), dvs_input=False) samna_cfg = dynapcnn.make_config(device="speck2fmodule") # samna_cfg.cnn_layers[layer].destinations[0] stores each core's first destination layers configuration # check the default layer ordering for layer in [0, 1, 2]: print(f"Is layer {layer} output turned on: {samna_cfg.cnn_layers[layer].destinations[0].enable}") print(f"The destination layer of layer {layer} is layer {samna_cfg.cnn_layers[layer].destinations[0].layer}") # manually modify the samna config # since 1 DYNAP-CNN core can have 2 destination layer # we need to enable the core#0's 2nd output destination and target it to core#2 # so we need to modify samna_cfg.cnn_layers[0].destinations[1] samna_cfg.cnn_layers[0].destinations[1].enable = True samna_cfg.cnn_layers[0].destinations[1].layer = 2 # by applying the modification above, we not only send the output of core#0 to core#1 but also to core#2. # which means we achieve the residual block! # finally we just need to apply the samna configuration to the devkit, we finish the deployment. devkit = samna.device.open_device("Speck2fModuleDevKit") devkit.get_model().apply_configuration(samna_cfg) ``` It is a lot of manual work but it will let you have your Residual Block. We recommend to use our network graph extraction feature for residual connections. ## What execution order should I be aware of when I am implementing a sequential structure? You should be aware with the internal layer order. DYNAP-CNN techonology defines serveral layers that can be communicates each other. In a layer, the Convolution and Neuron activation must be implemented with an order like: **Conv--> IAF -->pool(optional)** The cascaded convolution and neuron activation in a DYNAPCNN layer is not allowed. ![dataflow](/_static/Overview/dataflow_layers.png) ### Ex1. Bad Case: Cascaded convolution ``` network = nn.sequential([ nn.conv2d(), nn.conv2d(), IAFsqueeze(), ]) ``` ### Ex2. Bad Case: None sequential ``` class Network: def __init__(self): self.conv1 = nn.conv2d() self.iaf = IAFsqueeze() def forward(self, x): out = self.conv1(x) out = self.iaf(out) return out ``` ### Ex3. Bad Case: Use unsupport operation ``` network = nn.sequential([ nn.conv2d(), nn.BatchNorm2d(), # unsupport in speck IAFsqueeze(), ]) ``` ### Ex3. Good Case: Use unsupport operation ``` network = nn.sequential([ nn.conv2d(), IAFsqueeze(), nn.pool(), # up to here is using 1 dynapcnn layer nn.conv2d(), IAFsqueeze(), nn.Flatten(), nn.Linear(), IAFsqueeze(), # up to here is using 2 dynapcnn layer ]) ``` ## Memory Constrains Each core has a different "neuron memory" and "weight memory" constraints in the design. Please be careful about the memory limitations when you design your SNN. See detail in the [overview of devkit.](../overview.md) ## Feature Map Size Constrains The maximum output feature map size supported by each core is 64 x 64, while our maximum input shape is 128 x 128. So you need to at least down-sample the input into 64 x 64 by pooling or stride-convolution in the first layer of your model. ## Limitation of Using ReadoutLayer If you are using readout layer, the number of output class should < **15**. See detail in the [readout layer introduction.](../notebooks/using_readout_layer.ipynb)