Building a Neural Network from Scratch in C

For the past few months, I’ve been completely hooked on the "how" of AI. It’s one thing to use a chatbot, but it’s another thing entirely to understand the tiny gears turning under the hood. Eventually, I decided I had to build a neuron myself.

The neuron is the smallest computational unit of a neural network—the DNA of every AI system you’ve ever used. But building a full-scale network is like trying to fight Goliath before you’ve even picked up a stone. To keep things manageable, I started with the simplest possible system: a single neuron capable of detecting patterns in numbers, specifically Arithmetic Progressions (AP), Geometric Progressions (GP), and the Fibonacci sequence.

It was a deeply educational experience. What fascinated me most was realizing that, at its core, "learning" is just a set of numbers adjusting themselves to be slightly less wrong. Here is how it works.

What is a Neuron?

To understand a neural network, you have to start with the "cell." In simple terms, a neuron is just a function. It takes inputs, multiplies them by "weights" (to decide how important they are), adds a "bias" (a constant nudge), and produces an output.

The math looks like this:

In my implementation, the neuron tries to predict the next number in a sequence by looking at the current number and the previous one.

The Structure in C

I defined the neuron as a simple struct. I initialized the weights at 0.5, though in "real" machine learning, you’d usually start with random numbers.

typedef struct {
    double w1;    // Weight for the current number
    double w2;    // Weight for the previous number
    double bias;  // The constant adjustment
} Neuron;

In this setup, w1 is the importance of the most recent number, w2 is for the one before it, and the bias shifts the whole prediction up or down.

Step 1: Making a Prediction (The Forward Pass)

Once we have our neuron, it’s time to put it to work. It takes the current and previous numbers and makes its best "guess" at the next value.

The formula is straightforward:

Think of this as the neuron’s internal logic. At this stage—the Forward Pass—the model is just calculating. It isn't changing its mind yet. The learning only happens once the neuron realizes just how wrong that guess was.

Step 2: How the Neuron Knows It’s Wrong

After the guess is made, we have to check the "accuracy." The simplest way is to find the raw error:

Why We Use Squared Error

In practice, machine learning doesn't just look at the raw error. We usually square it:

This does two clever things:

1. It treats positive and negative errors the same. (Distance is distance, regardless of direction).
2. It penalizes big mistakes. Because we’re squaring the number, a small error stays small, but a large error becomes massive, forcing the neuron to pay more attention.

Sum of Squared Errors (SSE)

When we train on a whole set of data, we want to know the total error across every example. We call this the Sum of Squared Errors (SSE):

The ultimate goal: Minimize this loss until it's as close to zero as possible.

Step 3: Learning via Gradient Descent

Now for the "Aha!" moment. Once the neuron knows its error, it uses Gradient Descent to update its parameters. Gradient descent is essentially a way to "walk down" the hill of error until you reach the valley of accuracy.

The program runs a loop that updates w1, w2, and bias using these rules:

• w₁ += learning_rate × error × current
• w₂ += learning_rate × error × previous
• bias += learning_rate × error

The learning rate is our "step size." If it’s too big, the neuron overreacts; if it’s too small, it takes forever to learn. Every time the neuron makes a mistake, these formulas nudge the weights in the direction that would have made the prediction more accurate.

Step 4: Training the Neuron

To actually "teach" it, we feed it the training data repeatedly. Each pass through the data is like a study session. The neuron guesses, checks the answer, and nudges its weights.

Here is a simplified look at that training loop:

for (int i = 0; i < iterations; i++) {
    for (int j = 0; j < size; j++) {
        // The Forward Pass
        double prediction = model->w1 * curr[j] + model->w2 * prev[j] + model->bias;

        // Calculate the difference
        double error = target[j] - prediction;

        // Gradient Descent: The "Nudge"
        model->w1 += learning_rate * error * curr[j];
        model->w2 += learning_rate * error * prev[j];
        model->bias += learning_rate * error;
    }
}

After enough iterations, the weights stop jumping around and settle (converge) on the pattern.

Mathematical Sequences: The Results

The most satisfying part was seeing the neuron "discover" the rules of math. It didn't memorize the numbers; it found the relationship between them.

1. The Fibonacci Sequence

The rule is F(n) = F(n-1) + F(n-2).
After training, the neuron’s weights naturally settled near:

• w₁ ≈ 1
• w₂ ≈ 1
• bias ≈ 0

It essentially "wrote" its own Fibonacci code. If you give it 3 and 5, it calculates (1 × 5) + (1 × 3) + 0 = 8.

2. Arithmetic Progressions (AP)

For a sequence that adds a constant (like 2, 4, 6... where d=2):

• w₁ ≈ 1
• w₂ ≈ 0
• bias ≈ d (e.g., 2)

3. Geometric Progressions (GP)

For a sequence that multiplies by a ratio (like 3, 9, 27... where r=3):

• w₁ ≈ r (e.g., 3)
• w₂ ≈ 0
• bias ≈ 0

Mapping Code to Machine Learning Concepts

Even this tiny program is a mirror of what the big AI models are doing.

Limitations and Final Thoughts

I’ll be the first to admit: this neuron is a simple creature.

1. It’s Linear: It can only learn patterns that fit a straight line. It can't handle complex, non-linear curves.
2. It’s Lonely: It only has two inputs. Modern networks have thousands.
3. It’s Shallow: Real AI uses "layers" of neurons to process information in stages.

Despite that, building this was a revelation. It stripped away the hype of "Artificial Intelligence" and replaced it with something much cooler: elegant, self-correcting math.

You can find the full source code on GitHub: pie-314/neural-network-from-scratch-c.