RNN, LSTM, Attention & Reinforcement Learning Notes

Download Unit 4 PDF Notes

Download the complete PDF notes for CS601 Machine Learning Unit 4. These notes are useful for quick revision, PYQ-based preparation, 7-mark answers and 14-mark long answers.

Unit 4 Official Syllabus

Recurrent Neural Network, Long Short-Term Memory, Gated Recurrent Unit, Translation, Beam Search and Width, BLEU Score, Attention Model, Reinforcement Learning, RL Framework, MDP, Bellman Equations, Value Iteration and Policy Iteration, Actor-Critic Model, Q-Learning and SARSA.

Why Unit 4 is Important?

Unit 4 is one of the most important units of CS601 Machine Learning because it connects two powerful branches of modern AI: sequential learning and reinforcement learning. RNN, LSTM and GRU are used to handle sequence data such as text, speech and time-series data. Reinforcement learning topics such as MDP, Bellman Equation, Q-Learning and SARSA are repeatedly asked in RGPV Machine Learning papers.

If a student prepares LSTM, Reinforcement Learning Framework, MDP, Bellman Equation, Q-Learning, SARSA and Attention Model properly, they can confidently attempt most of the Unit 4 theory questions in the exam.

Chapter 1: Recurrent Neural Network (RNN)

A Recurrent Neural Network is a neural network designed to process sequential data. Unlike traditional neural networks, RNN uses previous information through hidden states. This memory makes RNN useful for tasks where order of data matters.

Exam Definition

RNN is a type of neural network in which previous outputs are fed back into the network to process sequential information and remember past context.

x1 → h1 → y1
      ↓
x2 → h2 → y2
      ↓
x3 → h3 → y3

Important Terms

• Sequential Data: Data where order matters, such as sentences and time-series.
• Hidden State: Memory of previous inputs.
• Temporal Dependency: Relationship between current and previous data.

Advantages

• Handles sequence data.
• Stores previous information.
• Useful in NLP and speech recognition.

Limitations

• Vanishing gradient problem.
• Difficulty in learning long-term dependencies.
• Slow training for long sequences.

Chapter 2: LSTM and GRU

RNN suffers from the vanishing gradient problem and cannot remember long sequences effectively. LSTM and GRU were developed to solve this limitation.

Long Short-Term Memory (LSTM)

LSTM is a special type of RNN that uses memory cells and gates to remember important information for long periods.

Input
↓
Forget Gate
↓
Input Gate
↓
Memory Cell
↓
Output Gate
↓
Output

LSTM Gates

• Forget Gate: Decides what information should be removed.
• Input Gate: Decides what new information should be stored.
• Output Gate: Decides what information should be sent as output.
• Memory Cell: Stores long-term information.

Gated Recurrent Unit (GRU)

GRU is a simplified version of LSTM. It uses reset gate and update gate. GRU is faster and requires fewer parameters.

Input
↓
Update Gate
↓
Reset Gate
↓
Output

LSTM vs GRU

Chapter 3: Machine Translation, Beam Search, BLEU Score and Attention

Machine Translation

Machine Translation converts text from one language into another using machine learning and deep learning models.

Source Language
↓
Encoder
↓
Context Vector
↓
Decoder
↓
Target Language

Beam Search

Beam Search is a search technique that keeps multiple best candidate sequences during decoding and selects the most probable final sentence.

Beam Width

Beam Width is the number of candidate sequences retained at every step during Beam Search.

BLEU Score

BLEU Score is an evaluation metric used to measure the quality of machine translation by comparing machine-generated translation with human reference translation.

Attention Model

Attention Model allows a neural network to focus on important parts of the input sequence while generating output. It improves performance for long sequences and machine translation.

Input Sentence
↓
Encoder
↓
Attention Layer
↓
Decoder
↓
Output Sentence

Chapter 4: Reinforcement Learning and MDP

Reinforcement Learning is a machine learning technique where an agent learns by interacting with an environment and receiving rewards or penalties.

Agent
↓ Action
Environment
↓ Reward + Next State
Agent

RL Components

• Agent: Learner or decision maker.
• Environment: World where agent acts.
• State: Current situation.
• Action: Decision taken by agent.
• Reward: Feedback received after action.
• Policy: Strategy used to select actions.

Markov Decision Process (MDP)

MDP is a mathematical framework used to represent decision-making problems in reinforcement learning.

State
↓
Action
↓
Next State
↓
Reward

MDP Components

• States
• Actions
• Rewards
• Transition Probability
• Policy

Markov Property

Markov Property states that the future depends only on the present state and not on the complete past history.

Chapter 5: Bellman Equation, Value Iteration and Policy Iteration

Bellman Equation

Bellman Equation calculates the value of a state by combining immediate reward and future reward. It is the foundation of many reinforcement learning algorithms.

V(s) = R + γV(s')

Here, V(s) is current state value, R is immediate reward, γ is discount factor and V(s') is future state value.

Discount Factor

Discount Factor decides the importance of future rewards. Its value lies between 0 and 1.

Value Iteration

Value Iteration repeatedly updates state values using Bellman Equation until optimal values are obtained.

Initialize Values
↓
Apply Bellman Equation
↓
Update Values
↓
Repeat
↓
Optimal Values

Policy Iteration

Policy Iteration repeatedly evaluates and improves a policy until the optimal policy is found.

Initial Policy
↓
Policy Evaluation
↓
Policy Improvement
↓
Optimal Policy

Value Iteration vs Policy Iteration

Chapter 6: Actor-Critic Model, Q-Learning and SARSA

Actor-Critic Model

Actor-Critic is a reinforcement learning architecture that combines policy-based and value-based learning. Actor selects actions, while Critic evaluates those actions.

State
↓
Actor selects Action
↓
Environment gives Reward
↓
Critic evaluates Action
↓
Actor improves Policy

Q-Learning

Q-Learning is a model-free reinforcement learning algorithm that learns the quality of actions using Q-values.

Q(S,A) = Q(S,A) + α[R + γ maxQ(S',A') - Q(S,A)]

Q-Table

Q-Table stores Q-values for different state-action pairs. The agent selects the action with the highest Q-value.

SARSA

SARSA stands for State Action Reward State Action. It is an on-policy reinforcement learning algorithm that updates values based on the action actually taken.

Q(S,A) = Q(S,A) + α[R + γQ(S',A') - Q(S,A)]

Q-Learning vs SARSA

Unit 4 PYQ Trend Analysis

Top 25 Important Questions for Unit 4

1. Explain Recurrent Neural Network with architecture.
2. Explain hidden state in RNN.
3. Explain vanishing gradient problem in RNN.
4. Explain LSTM architecture with gates.
5. Differentiate RNN and LSTM.
6. Explain GRU architecture.
7. Differentiate LSTM and GRU.
8. Explain Machine Translation using Encoder-Decoder model.
9. Explain Beam Search and Beam Width.
10. Explain BLEU Score.
11. Explain Attention Model with diagram.
12. Explain Reinforcement Learning Framework.
13. Explain Agent, Environment, State, Action and Reward.
14. Explain Markov Decision Process.
15. Explain Markov Property.
16. Explain Bellman Equation.
17. Explain Discount Factor.
18. Explain Value Iteration.
19. Explain Policy Iteration.
20. Differentiate Value Iteration and Policy Iteration.
21. Explain Actor-Critic Model.
22. Explain Q-Learning Algorithm.
23. Explain Q-Table.
24. Explain SARSA Algorithm.
25. Differentiate Q-Learning and SARSA.

Most Expected Questions 2026

• 95%: Explain Reinforcement Learning Framework with diagram.
• 95%: Explain LSTM architecture with neat diagram.
• 95%: Explain Bellman Equation and its significance.
• 95%: Explain Q-Learning and SARSA.
• 90%: Explain Markov Decision Process.
• 90%: Explain Attention Mechanism.
• 85%: Explain Beam Search and BLEU Score.

24-Hour Revision Box

RNN
↓
Sequence Data + Hidden State

LSTM
↓
Forget Gate + Input Gate + Output Gate

GRU
↓
Reset Gate + Update Gate

Machine Translation
↓
Encoder + Decoder

Beam Search
↓
Multiple Candidate Paths

BLEU Score
↓
Translation Quality

Attention
↓
Focus Important Words

Reinforcement Learning
↓
Agent + Environment + Reward

MDP
↓
State + Action + Reward + Transition

Bellman Equation
↓
V(s)=R+γV(s')

Value Iteration
↓
Update Values

Policy Iteration
↓
Evaluate and Improve Policy

Q-Learning
↓
Best Future Action

SARSA
↓
Actual Future Action

How to Write 7-Mark Answer?

For a 7-mark Unit 4 answer, write definition, diagram, working and applications. If the question is from RL, always draw agent-environment diagram. If the question is from LSTM, draw gates.

How to Write 14-Mark Answer?

For a 14-mark answer, follow this structure: introduction, definition, architecture diagram, detailed working, advantages, disadvantages, applications, comparison table and conclusion. For Unit 4, diagrams are very important because RNN, LSTM, RL, MDP and Q-Learning questions are diagram-friendly topics.

Related Machine Learning Units

FAQs

Which is the most important topic in CS601 Machine Learning Unit 4?

Reinforcement Learning Framework, LSTM, Bellman Equation, Q-Learning and SARSA are the most important topics in Unit 4.

Is RNN important for RGPV exam?

Yes, RNN and LSTM are highly important because they are frequently asked in long-answer questions.

What is the difference between Q-Learning and SARSA?

Q-Learning is off-policy and learns from the best future action, while SARSA is on-policy and learns from the actual future action.

What should I study first in Unit 4?

Start with Reinforcement Learning Framework, then MDP, Bellman Equation, Q-Learning, SARSA, LSTM and Attention Model.