\ignore{
\documentstyle[11pt]{report}
\textwidth 13.7cm
\textheight 21.5cm
\newcommand{\myimp}{\verb+ :- +}
\newcommand{\ignore}[1]{}
\def\definitionname{Definition}

\makeindex
\begin{document}
}

\chapter{\label{chapter:nn}The \texttt{nn} (Neural Networks) Module}

The \texttt{nn} module provides a high-level interface between Picat and the FANN\footnote{\url{http://leenissen.dk/fann/wp/}} neural networks library, which implements feedforward neural networks.\footnote{The Picat-FANN interface was implemented by Sanders Hernandez.}  A feedforward neural network consists of neurons organized in layers from an input layer to an output layer, possibly with a number of hidden layers. Neurons in a layer (except for the input layer) are connected neurons in the previous layer. The connections have weights. A feedforward network represents a function from input to output. The neurons in the input layers receive the input. The information is propagated forward through the layers until it reaches the output layers, where the output is returned. The information that a neuron receives is determined by the connected predecessor neurons, the weights of the connections, and an activation function. The connection weights of a neural network are normally adjusted through training on a given set of input-output pairs, called training data. Once a neural network is trained, it can be used to predict the output for a given input.

The following gives an example program which creates a neural network for the \texttt{xor} function,  and trains it on a set of data stored in a file:
\begin{verbatim}
import nn.

main =>
    NN = new_nn({2,3,1}),
    nn_train(NN,"xor.data"),
    nn_save(NN,"xor.net"),
    nn_destroy_all.
\end{verbatim}
The function \texttt{new\_nn(\{2,3,1\})} returns a neural network with three layers, where the input layer has 2 neurons, the hidden layer has 3 neurons, and the output layer has 1 neuron. The program does not specify any activation functions used between layers, entailing that the default activation function, which is \texttt{sym\_sigmoid}, will be used. The predicate \texttt{nn\_train(NN,"xor.data")} trains the neural network with the training data stored in the file \texttt{"xor.data"}. The user is able to specify an algorithm to be used in the training and several parameters that affect the behavior of the algorithm, such as the maximum number of iterations (called \textit{epoches}), the learning rate, and the error function. This example does not specify a training algorithm or any of the training parameters, entailing that the default algorithm, which is \texttt{rprop}, is used with the default setting. The predicate \texttt{nn\_save} saves the trained neural network into a file named \texttt{"xor.net"}. The \texttt{nn\_destroy\_all}clears up the neural network and the internal data used during training.

The text file \texttt{"xor.data"} contains the following training data:
\begin{verbatim}
4 2 1
-1 -1
-1
-1 1
1
1 -1
1
1 1
-1
\end{verbatim}
The three integers in the first line states, respectively, that the number of input-output pairs is 4, the number of input values is 2, and the number of output values is 1. The remaining lines give the input-output pairs. 

The following program performs the same task as the above program, except that it trains the neural network with internal data:
\begin{verbatim}
import nn.

main =>
    NN = new_nn({2,3,1}),
    nn_train(NN,[({-1,-1}, -1), 
                 ({-1,1}, 1), 
                 ({1,-1}, 1), 
                 ({1,1}, -1)]),
    nn_save(NN,"xor.net"),
    nn_destroy_all.
\end{verbatim}
The predicate \texttt{nn\_train} is overloaded. When the second argument is a file name, Picat reads training data from the file. Otherwise, Picat expects a list of input-output pairs.

The following example program illustrate how to use a trained network:
\begin{verbatim}
import nn.

main =>
    NN = nn_load("xor.net"),
    printf("xor(-1,1) = %w\n",nn_run(NN,{-1,1})),
    nn_destroy_all.
\end{verbatim}
The function \texttt{nn\_load} loads a neural network. The function \texttt{nn\_run} uses a network to predict the output for an input.

\section{Create and Destroy Neural Networks}
\begin{itemize}
\item \texttt{new\_nn($Layers$) = $NN$}\index{\texttt{new\_nn/1}}: This function creates a fully connected neural network of the structure $Layers$, which is a list or an array of positive integers. For example, the function \texttt{new\_nn(\{2,3,1\})} returns a neural network that has an input layer of 2 neurons, a hidden layer of 3 neurons, and an output layer of 1 neuron.

\item \texttt{new\_standard\_nn($Layers$) = $NN$}\index{\texttt{new\_standard\_nn/1}}: This function is the same as \texttt{new\_nn($Layers$)}.

\item \texttt{new\_sparse\_nn($Layers$,$Rate$) = $NN$}\index{\texttt{new\_sparse\_nn/2}}: This function creates a sparse neural network that has the structure $Layers$ and the connection rate $Rate$. The connection rate determines the sparseness of the network, with 1 indicating that the network is fully connected, and 0 indicating that the network is not connected at all.

\item \texttt{new\_shortcut\_nn($Layers$) = $NN$}\index{\texttt{new\_shortcut\_nn/1}}: This function creates a fully connected neural network of the structure $Layers$ that has shortcut connections between neurons in each layer and all neurons in later layers.

\item \texttt{nn\_destroy($NN$)}\index{\texttt{nn\_destroy/1}}: This predicate destroys the neural network $NN$.

\item \texttt{nn\_destroy\_all}\index{\texttt{nn\_destroy\_all/0}}: This predicate destroys all the neural networks.
\end{itemize}

\section{Activation Functions}
An activation function for a neuron determines how information is propagated to it from its predecessor neurons. When a new neural network is created, it uses the default activation function \texttt{sym\_sigmoid} for all of its non-input neurons. The following predicates can be utilized to set activation functions.

\begin{itemize}
\item \texttt{nn\_set\_activation\_function\_layer($NN$, $Func$, $Layer$)}\index{\texttt{nn\_set\_activation\_function\_layer}}: This predicate sets the activation function to $Func$ for all the neurons in layer $Layer$ in the neural network $NN$. Let the number of layers in $NN$ is $n$. Since the first layer is numbered 1, $Layer$ must satisfy $2\ \le\ Layer\ \le n$. The following activation functions are available:
\begin{itemize}
\item \texttt{linear}
\item \texttt{threshold}
\item \texttt{sym\_threshold}: symetric threshold.
\item \texttt{sigmoid}
\item \texttt{step\_sigmoid}: stepped sigmoid
\item \texttt{sym\_sigmoid}: symetric threshold
\item \texttt{gaussian}
\item \texttt{sym\_gaussian}: symetric gaussian
\item \texttt{elliot}: an alternative for sigmoid
\item \texttt{sym\_elliot}: symetric elliot
\item \texttt{linear\_piece}
\item \texttt{sym\_linear\_piece}: symetric linear piece
\item \texttt{sin}
\item \texttt{sym\_sin}: symetric sin
\item \texttt{cos}
\item \texttt{sym\_cos}: symetric cos
\end{itemize}
\end{itemize}
The detault activation function is \texttt{sym\_sigmoid}. Each of these functions has a correspondidng name in FANN. Please refer to FANN documentation for a more detailed description of these functions.

\item \texttt{nn\_set\_activation\_function\_hidden($NN$, $Func$)}\index{\texttt{nn\_set\_activation\_function\_hidden}}: This predicate sets the activation function to $Func$ for all of the hidden layers in the neural network $NN$.

\item \texttt{nn\_set\_activation\_function\_output($NN$, $Func$)}\index{\texttt{nn\_set\_activation\_function\_output}}: This predicate sets the activation function to $Func$ for the output layer in the neural network $NN$.

\item \texttt{nn\_set\_activation\_steepness\_layer($NN$, $Steepness$, $Layer$)}\index{\texttt{nn\_set\_activation\_steepness\_layer}}: This predicate sets the activation steepness for all of the neurons in layer number layer, where $-1\ \le\ Steepness\ \le 1$ and $2 \le Layer n$ (n is the number of layers in $NN$).  A high steepness value means a more aggressive training. The default steepness is 0.5.

\item \texttt{nn\_set\_activation\_steepness\_hidden($NN$, $Steepness$)}\index{\texttt{nn\_set\_activation\_steepness\_hidden}}: This predicate sets the activation steepness to $Steepness$ for all of the hidden layers in the neural network $NN$.

\item \texttt{nn\_set\_activation\_steepness\_output($NN$, $Steepness$)}\index{\texttt{nn\_set\_activation\_steepness\_output}}: This predicate sets the activation steepness to $Steepness$ for the output layer in the neural network $NN$.

\section{Training Data}
A training dataset can be supplied to FANN either through a text file or a Picat collection. A training dataset file must have the following format:

\begin{verbatim}
num_train_data num_input num_output
inputdata separated by space
outputdata separated by space

.
.
.

inputdata separated by space
outputdata separated by space
\end{verbatim}
A training dataset stored in a Picat collection must be either a list or an array of input-output pairs. An input-output pair has the form \texttt{($Input$,$Output$)}, where $Input$ is an array of numbers or a single number, and so is $Output$.

\begin{itemize}
\item \texttt{nn\_train\_data\_size($Data$) = $Size$}\index{\texttt{nn\_train\_data\_sizen/1}}: This function returns the number of input-output pairs in the dataset $Data$, which is either a file name or a Picat collection.

\item \texttt{nn\_train\_data\_get($Data$, $I$) = $Pair$}\index{\texttt{nn\_train\_data\_get/2}}: This function returns the $I$th pair in the dataset $Data$. Notice that while this function takes linear time when $Data$ is a list, it takes constant time when $Data$ is a file name or an array.\footnote{The data is loaded into an internal array when the dataset if accessed the first time.}
\end{itemize}
\end{itemize}

\section{Train Neural Networks}
\begin{itemize}
\item \texttt{nn\_train($NN$,$Data$,$Ops$)}\index{\texttt{nn\_train/3}}: This predicate trains the neural network $NN$ using the dataset in $Data$, which is either a text file name or a Picat collection, and under the control of training options $Ops$. The following traning options are supported:
\begin{itemize}
\item \texttt{maxep($X$)}: $X$ is the maximum epochs for training.
\item \texttt{report($X$)}:	reporting every $X$ number or epochs. 
\item \texttt{derror($X$)}:	$X$ is the desired error in training.
\item \texttt{train\_func($X$)}: $X$ is the training algorithm, which can be one of the following: \texttt{batch}, \texttt{inc}, \texttt{qprop} (quick prop), \texttt{rprop} (, and \texttt{sprop}.
\item \texttt{lrate($X$)}: $X$ is the learning rate.
\item \texttt{momentum($X$)}: $X$ is the learning momentum.
\item \texttt{effor\_func($X$)}: $X$ is the error function, which can be either \texttt{linear} or \texttt{tanh}.
\item \texttt{stop\_func($X$)}:	$X$ is the stop function, which is either \texttt{bit} or \texttt{mse}.
\item \texttt{bfl($X$)}:  $X$ is the bit fail limit.
\item \texttt{qp\_decay($X$)}:	 $X$ is the \texttt{qprop} decay.
\item \texttt{qp\_mu($X$)} : $X$ is \texttt{qprop}  mu factor.
\item \texttt{rp\_increase($X$)}:	$X$ is the \texttt{rprop} increase factor.
\item \texttt{rp\_decrease($X$)}: $X$ is the \texttt{rprop} decrease factor.
\item \texttt{rp\_deltamin($X$)}: $X$ is the \texttt{rprop} delta min.
\item \texttt{rp\_deltama($X$)}: $X$ is the \texttt{rprop} delta max.
\item \texttt{rp\_deltazero($X$)}: $X$ is the  \texttt{rprop} delta zero.
\item \texttt{sp\_weight($X$)}:	$X$	 \texttt{sarprop} weight decay.
\item \texttt{sp\_thresh($X$)}: $X$	\texttt{sarprop} step error threshold factor.
\item \texttt{sp\_shift($X$)}: $X$	\texttt{sarprop} step error shift.
\item \texttt{sp\_temp($X$)}: $X$	\texttt{sarprop} temperature.
\item \texttt{scale(In\_min, In\_max, Out\_min, Out\_max)}: use training data scaling.
\item \texttt{inscale(in\_min, in\_max)}:  use input data scaling.
\item \texttt{outscale(out\_min, out\_max)}: use output data scaling.
\end{itemize}

\item \texttt{nn\_train($NN$,$Data$)}\index{\texttt{nn\_train/2}}: This predicate is the same as \texttt{nn\_train($NN$,$Data$,[])}.
\end{itemize}

\section{Save and Load Neural Networks}
\begin{itemize}
\item \texttt{nn\_save($NN$,$File$)}\index{\texttt{nn\_save/2}}: This predicate saves the neural network $NN$ to a file named $File$.

\item \texttt{nn\_load($File$) = $NN$}\index{\texttt{nn\_load/1}}: This function creates a neural network from a FANN neural network file named $File$.
\end{itemize}

\section{Run Neural Networks}
\begin{itemize}
\item \texttt{nn\_run($NN$, $Input$, $Opts$) = $Output$}\index{\texttt{nn\_run/3}}: This function predicts the output of a given input using the trained neural network $NN$, where $Input$ is an array of numbers or a single number, and $Opts$ is a list of testing options. The supported testing options include:
\begin{itemize}
\item \texttt{scaleIn($X$)}: $X$ indicates whether or not the input is scaled, with -1 meaning descale, 1 meaning scale, and 0 meaning nothing.
\item \texttt{scaleOut($X$)}: $X$ indicates whether or not the output is scaled,  with -1 meaning descale, 1 meaning scale, and 0 meaning nothing.	
\item \texttt{resetMSE}: resets the current Mean Squared error of the network.
\end{itemize}

\item \texttt{nn\_run($NN$, $Input$) = $Output$}\index{\texttt{nn\_run/3}}: This function is the same as \texttt{nn\_run($NN$, $Input$, []).
\end{itemize}
\ignore{
\end{document}
}
