Implicit Neural Representations With Periodic Activation Functions

SJ005087

Purpose:To use deep learning and create a model that describes the shapes of the sensory data that supports the perception of smell using periodicity information of signal.

Methods:We tested the transferability of models using the Periodical Activation Functions (PAF) in a recent deep learning training system. We used a transfer learning system from Dr. Bill Collingham and tested the learning ability of that system on synthetic data in MNIST. The capacity of both the training algorithm and the testing algorithm increased as we used periodicity information of the inputs.

Results:No significant differences between training and testing results were observed using MNIST with synthetic data. We think that the use of PAF is more effective than using the LSTM model when the transferability of the system decreases due to numerical difference.

Conclusion:The use of PAF for both training and testing may have some advantages on transferring deep learning model and on neural networks with deep neural networks.

(Images from Shutterstock. Licensed under CC BY-NC 3.0.)https://www.shutterstock.com/vsrc/T_k_Y.phpThe network training as shown for each training in Fig. 1B & 1D.

Periodicity is a more typical to the cases, for example when we consider any speech to be periodic or music. In music there are many examples of music in which the period of all cycles of a particular timbre is nearly the same. That is, it is unlikely a certain one of all frequencies can be picked up as such frequencies and the mean becomes 1.

So, one way to generate neural network for all samples of the periodicity is to train the entire network on every single input (see Fig. 1A). And another way is to train a neural network with just a constant set of elements that can take value in the range \[0; 1\]:

A practical mechanism for creation such a network when trained with multiple samples of the periodicity, as sketched in Fig. 2A, is:

1) The periodicity component

The idea of periodicity would find in every new input of the neural network periodicity component.