Neural Network Library

Bare bones neural network/machine learning library implementation, made for (somewhat) educational purposes. It features simple neuron model with layers and Kohonen Networks.

In my attempt to understand neural networks and machine learning (ML) better, I decided to write a simple ML library entirely in C from scratch. With this library you can write simple prediction models using feed-forward, fully connected networks (no CNNs) and perform ML based dimensionality reduction.

Below are some image prediction results from a neural network trained by the library (mix of MNIST database and handwritten, all 28x28). These results show predictions after training for \(\approx 30'000\) iterations, where error for the test set ended up being 'small' enough \(\approx 0.000932\).

These images are from a set that the neural network has never 'seen' before (neither in training nor the test set). It should be noted that, by today's standars, the neural network's architecture is pretty bad (since I didn't implement CNNs); the resulting neural network has 784 input neurons (28 times 28 because of the images' resolution we use) and the datasets are pretty small (10 items per). Despite that the convergence and resulting output are really good for such a simple predictor. As a side note, it's interesting to see that the percentages don't really add up to 100, but the neural network is still close enough and understands the pattern well.

It is a little trickier to sensibly visualise Kohonen Networks inner machinations, but there's a simple test one can perform to see if their implementation is correct and the network actually converges. Making a colour map, that is, initialising all the neurons to a random vector \(\vec{v} \in \mathbb{R}^3\) and running the training for some number of iterations. Results for such a test are shown below.

Presented test was run multiple times with random weights, that's why you can see the colours changing place between images. There's also a nice circular 'residue', that comes from the 'neighbourhood selection'. None of that matters though, as this test was only supposed to answer 'do we converge at all?' and the answer is a resounding yes. All this means that the implementation is correct.

These are the two primary neural network structures you can train and play around with using this library (fully connected feed-forward with backpropagation, or a Kohonen Network). There's one small thing that this library also implements and that's something I always, either carry around with me into all my projects, or implement (copy-paste) a simpler version of it: memory arena. Way back in the day I stumpled upon this article by Ryan Fleury and it completely transformed the way I think about allocations in my programs. Current implementation of my ML library uses arenas extensively when allocating matrices, gradient for backpropagation etc. It is an incredibly helpful tool and so in the spirit of keeping this somewhat educational, here's a simple structure and some functions that you can try and implement yourself (read Ryan's article first).

typedef struct Bump_Allocator Bump_Allocator;
struct Bump_Allocator
{
  u64 alignment;
  
  u64 commit_pos;
  
  u64 chunk_pos;
  u64 chunk_cap;
};

internal void *winmem_memory_reserve(u64 size);
internal b32 winmem_memory_commit(void *ptr, u64 size);
internal Bump_Allocator *winmem_bump_make(void);
internal void *winmem_bump_push(Bump_Allocator *allocator, u64 size);

To make this even more helpful think of introducing a 'temporary' allocator, something that allocates for a lifetime of a function call.

Looking back at this project, I'd say the hardest part to work on was backwards propagation. It wasn't hard to understand per-se, it was hard to implement because of so many moving pieces; keeping track of which layer I am on, what needs to be fed backwards, which derivative should I take now etc. was a small challenge, but seeing the final effect was worth it. You could say 'but backwards propagation is just applied chain rule' and you'd be correct, but saying that grossly oversimplifies things and undermines years of research that went into this to make it a cornerstone of ML. At its heart, yes, backwards propagation is applied chain rule, but that's not the full, clear picture and implementing/researching it helped me gain that clarity.