Architecture

1️⃣ What Is Deep Learning?

Deep learning is a subset of machine learning that uses neural networks inspired by the human brain.

A neural network:

• Takes input data
• Passes it through layers
• Learns patterns
• Produces output

2️⃣ Basic Components of a Neural Network

🔹 1. Input Layer

This is where data enters the model.

Example:

• Image → pixels
• Text → word embeddings
• Numbers → feature vectors

🔹 2. Hidden Layers

These layers process the data.

Each layer:

• Receives input
• Multiplies by weights
• Adds bias
• Applies activation function
• Sends output to next layer

The more hidden layers → the “deeper” the network.

🔹 3. Output Layer

Produces the final result.

Examples:

• 1 neuron → regression
• N neurons → classification (one per class)

3️⃣ The Mathematics (Simple Version)

Each neuron does:

output = activation( (inputs × weights) + bias )

Or mathematically:

y = f(Wx + b)

Where:

• W = weights
• x = input
• b = bias
• f = activation function

4️⃣ Activation Functions

They introduce non-linearity (so the model can learn complex patterns).

Here are common ones:

ReLU(x) = max(0, x)
Sigmoid(x) = 1 / (1 + e^-x)
Tanh(x) = (e^x - e^-x) / (e^x + e^-x)

5️⃣ Types of Deep Learning Architectures

Below is a beginner-friendly overview.

| Architecture | Used For | Key Idea |
|-------------|----------|----------|
| Feedforward Neural Network (FNN) | Basic classification & regression | Data moves forward only |
| Convolutional Neural Network (CNN) | Image processing | Uses convolution filters |
| Recurrent Neural Network (RNN) | Sequence data | Has memory of previous inputs |
| LSTM | Long sequences | Solves vanishing gradient problem |
| GRU | Sequence modeling | Simpler version of LSTM |
| Transformer | NLP, LLMs | Uses self-attention mechanism |
| Autoencoder | Dimensionality reduction | Learns compressed representation |
| GAN | Image generation | Two networks compete |

6️⃣ Feedforward Neural Network (FNN)

Simplest architecture.

Structure:

Input → Hidden Layer(s) → Output

No loops.

Data flows one direction only.

7️⃣ Convolutional Neural Networks (CNN)

Used in:

• Image recognition
• Object detection
• Medical imaging

Famous Example:

• OpenAI uses CNN components in some vision systems.

Key idea:

• Instead of looking at the whole image, it scans small regions (filters).

Layers in CNN:

• Convolution
• ReLU
• Pooling
• Fully Connected

8️⃣ Recurrent Neural Networks (RNN)

Used for:

• Time series
• Speech
• Text

They have loops:

Output(t) depends on Input(t) AND Hidden(t-1)

Problem:

• Vanishing gradients

Solution:

• LSTM
• GRU

9️⃣ Transformer Architecture

Revolutionized NLP.

Used in:

• OpenAI models
• Google language models

Core concept:

• Self-attention
• No recurrence
• Fully parallel

Paper:

• Attention Is All You Need (technically a research paper)

🔟 Example: Simple Neural Network in Python

Using PyTorch:

import torch
import torch.nn as nn

class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(4, 8)   # input layer
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(8, 3)   # output layer
        
    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

model = SimpleNN()
print(model)

What this means:

• 4 inputs
• 1 hidden layer (8 neurons)
• 3 output neurons

1️⃣1️⃣ Example: CNN in PyTorch

import torch
import torch.nn as nn

class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 16, 3)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(16 * 13 * 13, 10)

    def forward(self, x):
        x = self.pool(self.relu(self.conv1(x)))
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        return x

model = SimpleCNN()
print(model)

1️⃣2️⃣ How Architecture Affects Performance

Architecture determines:

• Model accuracy
• Training speed
• Memory usage
• Generalization ability

Too small → underfitting

Too large → overfitting

Compilation of All Code Blocks (Combined)

import torch
import torch.nn as nn

# --------------------------
# Simple Feedforward Network
# --------------------------

class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(4, 8)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(8, 3)
        
    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        return x

model_fnn = SimpleNN()
print("Feedforward Network:")
print(model_fnn)

# --------------------------
# Simple CNN
# --------------------------

class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 16, 3)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(16 * 13 * 13, 10)

    def forward(self, x):
        x = self.pool(self.relu(self.conv1(x)))
        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        return x

model_cnn = SimpleCNN()
print("\nCNN Network:")
print(model_cnn)