Neural Networks Complete Notes

Download Unit 2 PDF Notes

Download the complete PDF notes for CS601 Machine Learning Unit 2. These notes are prepared for RGPV students who want fast revision, PYQ-based preparation and 7-mark/14-mark answer writing.

Unit 2 Official Syllabus

Linearity vs non-linearity, activation functions like sigmoid, ReLU, weights and bias, loss function, gradient descent, multilayer network, backpropagation, weight initialization, training and testing, unstable gradient problem, autoencoders, batch normalization, dropout, L1 and L2 regularization, momentum and hyperparameter tuning.

Why Unit 2 is Important?

Unit 2 is one of the most important units of CS601 Machine Learning because it explains the foundation of neural networks and deep learning. In RGPV exams, questions from gradient descent, backpropagation, activation functions, autoencoders, dropout and regularization appear repeatedly. If a student prepares this unit properly, they can easily attempt long-answer questions from neural network training and optimization.

This unit also connects directly with Unit 3 CNN and Unit 4 RNN. Concepts like activation functions, loss function, gradient descent and backpropagation are used in almost every deep learning model.

1. Linearity vs Non-Linearity

Linearity means output changes in direct proportion to input. Non-linearity means output does not change in a simple straight-line manner. Real-world data is mostly non-linear, so neural networks need non-linear functions to learn complex patterns.

Linear:
Input → Straight Relationship → Output

Non-Linear:
Input → Complex Relationship → Output

Exam Definition

Linearity refers to a straight-line relationship between input and output variables. Non-linearity refers to a complex relationship where output does not change proportionally with input.

Difference Table

2. Activation Functions

Activation function is a mathematical function used inside a neuron to decide whether the neuron should be activated or not. It introduces non-linearity into neural networks, allowing them to learn complex patterns.

Input
↓
Weighted Sum
↓
Activation Function
↓
Output

Need of Activation Function

• Introduces non-linearity
• Helps neural networks learn complex data
• Controls neuron output
• Improves prediction capability

Important Activation Functions

• Sigmoid
• ReLU
• Tanh
• Leaky ReLU
• Softmax

3. Sigmoid Function

Sigmoid is an activation function that converts any input value into a range between 0 and 1. It is mainly used when output is required in probability form.

σ(x) = 1 / (1 + e^-x)

Output Range: 0 to 1

Working

• If input is very negative, output is close to 0.
• If input is 0, output is 0.5.
• If input is very positive, output is close to 1.

Advantages

• Useful for probability output
• Smooth curve
• Easy to understand

Disadvantages

• Causes vanishing gradient problem
• Training can become slow

4. ReLU Function

ReLU stands for Rectified Linear Unit. It is one of the most commonly used activation functions in deep learning.

f(x) = max(0, x)

If x < 0 → output = 0
If x > 0 → output = x

Advantages

• Fast training
• Simple calculation
• Reduces vanishing gradient issue
• Widely used in CNN and deep learning

Disadvantage

ReLU may suffer from dying ReLU problem where some neurons stop learning permanently.

Sigmoid vs ReLU

5. Weights and Bias

Weights and bias are the learnable parameters of a neural network. Weight decides the importance of input, while bias adjusts the output.

Output = (Weight × Input) + Bias

Weight

Weight controls how much importance should be given to a particular input feature.

Bias

Bias shifts the activation function and helps the model make better predictions even when input is zero.

Example

If input = 5, weight = 3 and bias = 2, then output = 3 × 5 + 2 = 17.

6. Loss Function

Loss function measures the difference between actual output and predicted output. It tells the model how wrong its prediction is.

Actual Output
↓
Compare with Predicted Output
↓
Loss Function
↓
Error Value

Types of Loss Function

• Mean Squared Error: Used for regression problems.
• Cross Entropy Loss: Used for classification problems.

Importance

• Measures prediction error
• Guides gradient descent
• Helps improve model accuracy

7. Gradient Descent

Gradient Descent is an optimization algorithm used to minimize the loss function by updating weights and bias. It moves in the direction where loss decreases.

New Weight = Old Weight - Learning Rate × Gradient

Working Steps

1. Initialize weights randomly
2. Calculate prediction
3. Calculate loss
4. Find gradient
5. Update weights
6. Repeat until loss becomes minimum

Types of Gradient Descent

8. Multilayer Neural Network

A multilayer neural network contains input layer, one or more hidden layers and output layer. It is used to solve complex non-linear problems.

Input Layer
↓
Hidden Layer
↓
Hidden Layer
↓
Output Layer

Components

• Input Layer: Receives input data.
• Hidden Layer: Learns patterns.
• Output Layer: Produces final result.

Advantages

• Learns complex patterns
• Supports deep learning
• Useful in image, speech and NLP tasks

9. Backpropagation

Backpropagation is the most important algorithm for training neural networks. It calculates error at the output and propagates it backward to update weights.

Forward Pass:
Input → Hidden Layer → Output → Loss

Backward Pass:
Loss → Output Layer → Hidden Layer → Weight Update

Working Steps

1. Perform forward pass
2. Calculate output
3. Calculate loss
4. Propagate error backward
5. Update weights using gradient descent
6. Repeat until error reduces

Advantages

• Improves model accuracy
• Reduces loss
• Essential for deep learning

10. Weight Initialization

Weight initialization is the process of assigning initial values to neural network weights before training starts.

Why Important?

Poor initialization can cause slow training, unstable gradients and poor accuracy.

Types

• Random Initialization
• Xavier Initialization
• He Initialization

11. Training and Testing

Training is the process of teaching the model using data. Testing is the process of checking model performance on unseen data.

Dataset
↓
Training Data → Model Learning
↓
Testing Data → Performance Evaluation

Difference Table

12. Unstable Gradient Problem

Unstable gradient problem occurs when gradients become too small or too large during neural network training. This makes learning difficult.

Types

• Vanishing Gradient Problem
• Exploding Gradient Problem

Solutions

• Use ReLU activation
• Use batch normalization
• Use proper weight initialization
• Use gradient clipping

13. Vanishing Gradient Problem

Vanishing gradient occurs when gradients become extremely small during backpropagation. As a result, earlier layers learn very slowly or stop learning.

Gradient: 0.8 → 0.4 → 0.1 → 0.01 → 0.0001
Learning becomes very slow

Causes

• Deep networks
• Sigmoid activation
• Poor initialization

Solutions

• ReLU
• Batch normalization
• Residual connections
• Proper initialization

14. Exploding Gradient Problem

Exploding gradient occurs when gradients become extremely large during training. It causes unstable updates and may crash training.

Gradient: 1 → 10 → 100 → 1000 → Training unstable

Solutions

• Gradient clipping
• Lower learning rate
• Batch normalization
• Proper weight initialization

15. Autoencoders

Autoencoder is an unsupervised neural network used for feature extraction, data compression and noise removal.

Input
↓
Encoder
↓
Bottleneck Layer
↓
Decoder
↓
Reconstructed Output

Parts of Autoencoder

• Encoder: Compresses input data.
• Bottleneck: Stores important features.
• Decoder: Reconstructs original data.

Applications

• Image compression
• Noise removal
• Fraud detection
• Dimensionality reduction

16. Batch Normalization

Batch Normalization normalizes layer inputs during training. It improves training speed, stability and performance.

Input
↓
Calculate Mean and Variance
↓
Normalize
↓
Scale and Shift
↓
Output

Advantages

• Faster training
• Stable learning
• Reduces internal covariate shift
• Improves accuracy

17. Dropout

Dropout is a regularization technique used to prevent overfitting. During training, some neurons are randomly disabled.

Without Dropout:
O O O O

With Dropout:
O X O X

X = Disabled neuron

Advantages

• Prevents overfitting
• Improves generalization
• Makes network robust

18. L1 and L2 Regularization

Regularization is used to prevent overfitting by adding a penalty to the loss function.

L1 Regularization

Loss = Original Loss + λΣ|w|

L1 can make some weights zero and helps in feature selection.

L2 Regularization

Loss = Original Loss + λΣw²

L2 reduces large weights but usually does not make them zero.

19. Momentum

Momentum is an optimization technique that improves gradient descent by using previous updates. It helps the model move faster and reduces zig-zag movement.

Without Momentum: /\/\/\/\/\
With Momentum:    _________

Advantages

• Faster convergence
• Reduced oscillation
• Improved optimization

20. Hyperparameter Tuning

Hyperparameter tuning is the process of selecting the best values of hyperparameters before training a machine learning model.

Common Hyperparameters

• Learning rate
• Batch size
• Epochs
• Number of hidden layers
• Dropout rate

Methods

• Grid Search
• Random Search
• Bayesian Optimization

Choose Parameters
↓
Train Model
↓
Evaluate
↓
Tune Again
↓
Best Model

Unit 2 PYQ Trend Analysis

Top 20 Important Questions for Unit 2

1. Explain activation functions. Why are they required?
2. Explain sigmoid and ReLU activation functions.
3. Differentiate sigmoid and ReLU.
4. Explain weights and bias in neural networks.
5. Explain loss function with examples.
6. Explain gradient descent algorithm.
7. Explain types of gradient descent.
8. Explain multilayer neural network with diagram.
9. Explain backpropagation algorithm in detail.
10. Explain weight initialization and its importance.
11. Differentiate training and testing.
12. Explain unstable gradient problem.
13. Explain vanishing gradient problem.
14. Explain exploding gradient problem.
15. Explain autoencoder with architecture.
16. Explain batch normalization.
17. Explain dropout technique.
18. Differentiate L1 and L2 regularization.
19. Explain momentum in gradient descent.
20. Explain hyperparameter tuning.

Most Expected Questions 2026

• 95%: Explain Backpropagation Algorithm with neat diagram.
• 95%: Explain Gradient Descent and its types.
• 90%: Differentiate L1 and L2 Regularization.
• 90%: Explain Autoencoder architecture.
• 90%: Explain Batch Normalization and Dropout.
• 85%: Explain Sigmoid and ReLU activation functions.

24-Hour Revision Box

Activation Function → Adds non-linearity
Sigmoid → Output between 0 and 1
ReLU → Fast training
Weights → Importance
Bias → Adjustment
Loss Function → Error measurement
Gradient Descent → Minimizes loss
Backpropagation → Updates weights
Vanishing Gradient → Learning stops
Exploding Gradient → Training unstable
Autoencoder → Compression
Batch Normalization → Stable training
Dropout → Prevents overfitting
L1 → Feature selection
L2 → Weight reduction
Momentum → Faster learning
Hyperparameter Tuning → Best settings

Related Machine Learning Units

FAQs

Which is the most important topic in Unit 2?

Backpropagation and Gradient Descent are the most important topics for RGPV exams.

Is Unit 2 important for long-answer questions?

Yes, Unit 2 is very important for 7-mark and 14-mark questions.

What should I study first?

Start with activation functions, then gradient descent, backpropagation, autoencoders and regularization.

Which diagram should I remember?

Remember neural network architecture, backpropagation flow and autoencoder architecture.