C#: logistic regression for classification

Before you start

C# Math functions and the gradient descent idea.

Predict a probability with the sigmoid function and train the weights by gradient descent.
Predict the sample output and explain the state changes.
Complete the practice task and check the boundary cases.

The concept

Logistic regression is a classifier. It computes a weighted sum z = w·x + b, passes it through the sigmoid 1 / (1 + e^−z) to get a probability between 0 and 1, and predicts class 1 when the probability is at least 0.5. Unlike a squared-error line fit, it is trained to maximize the likelihood of the labels.

For the cross-entropy loss, the gradient with respect to w is simply the average of (prediction − label) × x, and with respect to b the average of (prediction − label). The data below are hours studied and whether the student passed. They overlap slightly, so the probability rises smoothly instead of jumping from 0 to 1.

Code example

using System;
using System.Globalization;

class Program {
  static double Sigmoid(double z) {
    return 1.0 / (1.0 + Math.Exp(-z));
  }

  static string F(double v) {
    return v.ToString("F2", CultureInfo.InvariantCulture);
  }

  static void Main() {
    double[] hours = { 1, 2, 3, 4, 5, 6 };
    int[] passed = { 0, 0, 1, 0, 1, 1 };
    double w = 0, b = 0, rate = 0.1;

    for (int epoch = 0; epoch < 5000; epoch++) {
      double gw = 0, gb = 0;
      for (int i = 0; i < hours.Length; i++) {
        double error = Sigmoid(w * hours[i] + b) - passed[i];
        gw += error * hours[i] / hours.Length;
        gb += error / hours.Length;
      }
      w -= rate * gw;
      b -= rate * gb;
    }

    Console.WriteLine("w=" + F(w) + " b=" + F(b));
    Console.WriteLine("P(pass | 1h) = " + F(Sigmoid(w * 1 + b)));
    Console.WriteLine("P(pass | 6h) = " + F(Sigmoid(w * 6 + b)));
    int correct = 0;
    for (int i = 0; i < hours.Length; i++) {
      int predicted = Sigmoid(w * hours[i] + b) >= 0.5 ? 1 : 0;
      if (predicted == passed[i]) correct++;
    }
    Console.WriteLine("accuracy: " + correct + "/" + hours.Length);
  }
}

Output

w=1.21 b=-4.25
P(pass | 1h) = 0.05
P(pass | 6h) = 0.95
accuracy: 4/6

Example details

Follow the code

Open the walkthrough to inspect the example one step at a time.

Show guided walkthrough

Guided walkthroughReady

1using System;

2using System.Globalization;

4class Program {

5 static double Sigmoid(double z) {

6 return 1.0 / (1.0 + Math.Exp(-z));

7 }

9 static string F(double v) {

10 return v.ToString("F2", CultureInfo.InvariantCulture);

11 }

13 static void Main() {

14 double[] hours = { 1, 2, 3, 4, 5, 6 };

15 int[] passed = { 0, 0, 1, 0, 1, 1 };

16 double w = 0, b = 0, rate = 0.1;

18 for (int epoch = 0; epoch < 5000; epoch++) {

19 double gw = 0, gb = 0;

20 for (int i = 0; i < hours.Length; i++) {

21 double error = Sigmoid(w * hours[i] + b) - passed[i];

22 gw += error * hours[i] / hours.Length;

23 gb += error / hours.Length;

24 }

25 w -= rate * gw;

26 b -= rate * gb;

27 }

29 Console.WriteLine("w=" + F(w) + " b=" + F(b));

30 Console.WriteLine("P(pass | 1h) = " + F(Sigmoid(w * 1 + b)));

31 Console.WriteLine("P(pass | 6h) = " + F(Sigmoid(w * 6 + b)));

32 int correct = 0;

33 for (int i = 0; i < hours.Length; i++) {

34 int predicted = Sigmoid(w * hours[i] + b) >= 0.5 ? 1 : 0;

35 if (predicted == passed[i]) correct++;

36 }

37 Console.WriteLine("accuracy: " + correct + "/" + hours.Length);

38 }

39}

Follow the example

Start to inspect the source, values, and output. These prepared steps explain the published sample; grouped loop steps describe multiple iterations.

View the complete dry-run table

Prepared trace of the published program. Some rows group repeated iterations.
What happens	State after step	Output so far
The sigmoid squashes any real number into the range (0, 1).	`—`	—
Prediction minus label. This simple form comes from the cross-entropy loss.	`—`	—
Move the weights downhill on the loss.	`w and b learn from the data`	—
The learned parameters: the decision boundary w·x + b = 0 is at x ≈ 3.5 hours.	`w and b learn from the data`	w=1.21 b=-4.25
One hour of study gives about 5% and six hours about 95%.	`w and b learn from the data`	w=1.21 b=-4.25 P(pass \| 1h) = 0.05 P(pass \| 6h) = 0.95
The two overlapping points (3 hours passed, 4 hours failed) are misclassified, so 4 of 6 are correct.	`w and b learn from the data`	w=1.21 b=-4.25 P(pass \| 1h) = 0.05 P(pass \| 6h) = 0.95 accuracy: 4/6

Behind the scenes

How C# processes this code

C# compiles to code for the .NET runtime, which invokes the program entry point.

Cost and common mistakes

O(epochs · n · d) time for n examples with d features.

On perfectly separable data the weights grow without limit as training continues. Add regularization to keep them finite.
Always evaluate on data not used for training; accuracy on the training set is optimistic.
The 0.5 threshold is a choice. Moving it trades precision against recall.

Try it yourself

What does a probability of exactly 0.5 mean?

Reveal answer and reasoning

z = w·x + b equals 0, so the point lies on the decision boundary.