Courses / HEC NSCT – National Skill Competency Test: Complete Preparation for Computing Graduates
8.2 Data, Preprocessing and Exploratory Data Analysis
Area 8 – AI, Machine Learning & Data Analytics (10%) · 45 min read · 16 MCQs to check yourself
The ML workflow
Define problem → collect data → clean/preprocess → EDA → feature engineering → train → evaluate → tune → deploy → monitor.
Data types
Structured (tables) vs unstructured (text, images, audio). Numerical (continuous/discrete) vs categorical (nominal: colour; ordinal: low/medium/high).
Preprocessing
| Problem | Fix |
|---|---|
| Missing values | Drop rows/columns, or impute with mean/median/mode |
| Outliers | Detect with box plot / IQR / z-score; cap, remove or keep if genuine |
| Different scales | Normalization (min–max to 0–1) or standardization (z-score: mean 0, std 1) |
| Categorical text | One-hot encoding (nominal) or label/ordinal encoding |
| Duplicates, wrong types | Remove/convert |
| Imbalanced classes | Oversample (SMOTE), undersample, class weights |
Exploratory Data Analysis (EDA)
- Summary statistics: mean, median (robust to outliers), mode, standard deviation, quartiles.
- Histogram (distribution of one variable), box plot (median, IQR, outliers), scatter plot (relationship of two variables), heatmap of the correlation matrix, bar chart (categories).
- Correlation r ranges from −1 to +1; it measures linear relationship, not causation.
- Feature engineering: create useful features (age from date of birth), select features, reduce dimensions with PCA.
Data preprocessing and data cleaning (Gate Smashers)
This lesson ends with a quiz of 16 MCQs. Open the lesson on PrepOasis to take it and save your progress.
Previous: 8.1 AI Foundations and Search · Next: 8.3 Supervised Learning and Model Evaluation