Data Preprocessing and Knowledge Discovery

1. Data Preprocessing 14 Marks

Why Data Preprocessing is Required

• Real-world data is often incomplete.
• Different sources may use different formats.
• Records may contain errors and duplicate values.
• Large data may increase processing time.
• Irrelevant attributes can reduce model accuracy.
• Mining results depend strongly on data quality.

Main Steps

Objectives

• Improve data accuracy and consistency.
• Reduce the size of data.
• Improve algorithm performance.
• Produce meaningful and reliable patterns.

2. Data Quality Problems 14 Marks

Incomplete Data

Some attribute values or complete records may be missing.

Noisy Data

Data may contain random errors, abnormal values or measurement errors.

Inconsistent Data

The same fact may be represented differently in different sources.

Duplicate Data

The same entity may appear multiple times because of repeated entry or source integration.

Outdated Data

Old values may no longer represent the current real-world situation.

Data Quality Dimensions

• Accuracy
• Completeness
• Consistency
• Timeliness
• Validity
• Uniqueness

3. Data Cleaning 14 Marks

Major Tasks

• Fill missing values.
• Smooth noisy data.
• Identify and remove outliers.
• Resolve inconsistencies.
• Remove duplicate records.
• Validate formats and ranges.

General Cleaning Process

Benefits

• Improves model accuracy.
• Reduces misleading patterns.
• Improves integration.
• Reduces unnecessary processing.

4. Handling Missing Values 14 Marks

Methods

1. Ignore the Tuple

Delete the record containing the missing value. This is suitable when only a few records are affected.

2. Fill Manually

A user or expert enters the correct value. This is accurate but expensive for large datasets.

3. Use a Global Constant

Replace missing values with a value such as Unknown or Not Available.

4. Use Mean or Median

For numerical attributes, replace missing values with mean or median.

5. Use Class-Specific Mean

Calculate the mean only within the same class or group.

6. Predict the Most Probable Value

Use regression, decision tree or another model to estimate the missing value.

5. Handling Noisy Data 14 Marks

Binning

Sorted data is divided into bins and values are smoothed using bin mean, median or boundaries.

Regression

A function is fitted to data and noisy values are adjusted according to the fitted relationship.

Clustering

Values outside normal clusters may be identified as outliers or errors.

Computer and Human Inspection

Automated detection is combined with expert review for important records.

6. Data Integration 14 Marks

Major Issues

Schema Integration

Different database schemas must be matched and combined.

Entity Identification

The same entity may have different names or identifiers in different sources.

Redundancy

The same attribute or derived information may appear in multiple sources.

Data Value Conflict

Sources may use different units, codes or date formats.

Correlation Analysis

Correlation can help identify redundant numerical attributes.

Advantages

• Provides a unified data view.
• Improves consistency.
• Supports enterprise-wide analysis.

7. Data Transformation 14 Marks

Methods

Smoothing

Removes noise using techniques such as binning and regression.

Aggregation

Combines detailed values into summaries.

Generalization

Replaces low-level data with higher-level concepts.

Attribute Construction

Creates new attributes from existing attributes.

Normalization

Scales numerical attributes into a selected range.

8. Normalization Methods 14 Marks

Min-Max Normalization

Transforms a value into a specified range, commonly 0 to 1.

Z-Score Normalization

Uses the mean and standard deviation.

Decimal Scaling

Moves the decimal point according to the maximum absolute value.

Why Normalize?

• Prevents large-scale attributes from dominating.
• Improves distance-based algorithms.
• Speeds up model training.

9. Data Reduction 14 Marks

Need

• Reduce storage requirements.
• Reduce mining time.
• Improve visualization.
• Remove irrelevant or redundant information.

Main Strategies

• Data cube aggregation
• Dimensionality reduction
• Numerosity reduction
• Data compression
• Discretization and concept hierarchy

10. Dimensionality Reduction 14 Marks

Attribute Subset Selection

Selects only relevant attributes and removes redundant or irrelevant features.

Feature Extraction

Creates a smaller set of new features from the original attributes.

Principal Component Analysis

PCA transforms correlated variables into a smaller set of uncorrelated principal components.

Wavelet Transform

Transforms data into a compressed representation using wavelet coefficients.

Benefits

• Reduces computation.
• Reduces overfitting.
• Improves visualization.
• May improve model performance.

Curse of Dimensionality

As dimensions increase, data becomes sparse and distance measures may become less meaningful.

11. Numerosity Reduction 14 Marks

Parametric Methods

• Regression
• Log-linear models

Non-Parametric Methods

• Histograms
• Clustering
• Sampling
• Data cube aggregation

Sampling Types

• Simple random sampling
• Sampling without replacement
• Sampling with replacement
• Stratified sampling

12. Discretization 14 Marks

Example

Methods

Equal-Width Partitioning

The value range is divided into intervals of equal width.

Equal-Frequency Partitioning

Each interval contains approximately the same number of data values.

Histogram Analysis

Uses the distribution of values to define intervals.

Entropy-Based Discretization

Uses class information and information gain to choose split points.

Cluster-Based Discretization

Uses clusters to create natural value groups.

Advantages

• Simplifies data.
• Improves interpretability.
• Supports concept hierarchy.
• Can reduce noise.

13. Concept Hierarchy Generation 14 Marks

Example Hierarchies

Generation Methods

• Explicitly specified by users or experts.
• Specified using schema relationships.
• Automatically generated using the number of distinct values.
• Generated through discretization of numerical attributes.

Uses

• Data generalization
• OLAP roll-up
• Multilevel mining
• Improved report readability

14. Basics of Data Mining 14 Marks

Major Functionalities

• Concept description
• Association analysis
• Classification
• Prediction
• Clustering
• Outlier analysis
• Evolution and trend analysis

Data Mining System Architecture

Pattern Evaluation

Interestingness measures are used to identify patterns that are useful, novel and understandable.

15. Data Mining Techniques 14 Marks

Classification

Assigns data objects to predefined classes.

Clustering

Groups similar objects without predefined class labels.

Association Rule Mining

Discovers relationships among items occurring together.

Regression and Prediction

Predicts continuous values such as sales, price or demand.

Outlier Detection

Finds objects that differ significantly from normal behavior.

Sequential and Time-Series Mining

Discovers trends and patterns ordered by time.

Text and Web Mining

Extracts useful patterns from documents, web content, links and user behavior.

16. Knowledge Discovery Process (KDD) 14 Marks

KDD Steps

1. Data Cleaning

Removes noise, errors and inconsistencies.

2. Data Integration

Combines multiple data sources.

3. Data Selection

Selects data relevant to the analysis task.

4. Data Transformation

Converts data into appropriate mining form.

5. Data Mining

Applies intelligent algorithms to discover patterns.

6. Pattern Evaluation

Identifies useful and interesting patterns.

7. Knowledge Presentation

Presents results through reports, charts, rules and visualizations.

17. Applications of Data Mining 14 Marks

Retail and Marketing

• Market basket analysis
• Customer segmentation
• Sales forecasting
• Recommendation systems

Banking and Finance

• Credit-risk analysis
• Fraud detection
• Customer profitability analysis

Healthcare

• Disease prediction
• Patient-risk analysis
• Treatment effectiveness analysis

Telecommunication

• Customer churn prediction
• Network fault detection
• Usage pattern analysis

Education

• Student performance prediction
• Dropout detection
• Personalized learning

Cybersecurity

• Intrusion detection
• Malware behavior analysis
• Abnormal activity detection

18. Challenges of Data Mining 14 Marks

Data Quality

Incomplete and noisy data reduces the reliability of discovered patterns.

Scalability

Algorithms must process very large and rapidly growing datasets.

High Dimensionality

Datasets may contain thousands of attributes.

Heterogeneous Data

Data may be relational, spatial, temporal, textual, multimedia or web-based.

Privacy and Security

Personal and confidential data must be protected.

Dynamic and Streaming Data

Patterns may need to be discovered continuously as data arrives.

Pattern Evaluation

A large number of patterns may be generated, but only a few may be useful.

User Interaction

Mining systems should include domain knowledge and produce understandable results.

Ethical Issues

Biased or unfair patterns can lead to harmful decisions.

19. Important Comparisons 14 Marks

Data Cleaning vs Data Transformation

Data Integration vs Data Reduction

KDD vs Data Mining

Classification vs Clustering

Unit 2 Quick Revision

• Preprocessing improves data quality before mining.
• Cleaning handles missing, noisy and inconsistent data.
• Integration combines multiple sources.
• Transformation includes aggregation, generalization and normalization.
• Reduction decreases size while preserving useful information.
• Dimensionality reduction decreases the number of features.
• Numerosity reduction replaces data with models or summaries.
• Discretization converts continuous values into intervals.
• Concept hierarchy creates multiple abstraction levels.
• Data mining discovers useful patterns.
• KDD includes cleaning, selection, transformation, mining and presentation.
• Privacy, scalability and data quality are major challenges.

Important RGPV Exam Questions

Long Answer Questions

1. Define data preprocessing and explain its major steps.
2. Explain different data-quality problems.
3. Explain methods of data cleaning with examples.
4. Discuss different techniques for handling missing values.
5. Explain noisy-data smoothing techniques.
6. What is data integration? Explain its major issues.
7. Explain data transformation and normalization methods.
8. Define data reduction and explain its strategies.
9. Explain dimensionality reduction and numerosity reduction.
10. What is discretization? Explain its methods.
11. Explain concept hierarchy generation with examples.
12. Define data mining and explain its functionalities.
13. Explain major data-mining techniques.
14. Draw and explain the complete KDD process.
15. Explain applications and challenges of data mining.

Short Answer Questions

1. What is data preprocessing?
2. Define noisy data.
3. What is data integration?
4. Define normalization.
5. What is dimensionality reduction?
6. Define discretization.
7. What is a concept hierarchy?
8. Define KDD.
9. What is classification?
10. What is outlier detection?

Download Study Resources

Frequently Asked Questions

Related Study Pages