# Data preparation

## Data cleaning

- [Data cleaning](https://elitedatascience.com/data-cleaning)
- [Spend Less Time Cleaning Data with Machine Learning](https://www.dataversity.net/spend-less-time-cleaning-data-with-machine-learning/#)
- [Helpful Python Code Snippets for Data Exploration in Pandas](https://medium.com/@msalmon00/helpful-python-code-snippets-for-data-exploration-in-pandas-b7c5aed5ecb9) - lots of python snippets to select / clean / prepare
- [Journal of Statistical Software - TidyData](https://www.jstatsoft.org/article/view/v059i10/)
- [5 Steps to correctly prepare your data for your machine learning model](https://towardsdatascience.com/5-steps-to-correctly-prep-your-data-for-your-machine-learning-model-c06c24762b73?gi=6b4a6895ab1)
- [Introduction to Data Analysis and Cleaning presentation](../presentations/data/01-mam-ml-study-group-meetup/Introduction_to_Data_Analysis_and_Cleaning.pdf) by [Mark Bell](http://www.nationalarchives.gov.uk/about/our-research-and-academic-collaboration/our-research-and-people/staff-profiles/mark-bell/)
- [Intro to Data Cleaning](https://media.licdn.com/dms/document/C561FAQFELM5YZM3-Rw/feedshare-document-pdf-analyzed/0?e=1572184800&v=beta&t=9HybdEtMdRqdqA6bJQeN-0sMDrEU8yBZwAvTUTm6dTo)
- [✅ Data Cleaning With NumPy and Pandas ✅](https://www.linkedin.com/posts/asif-bhat_pythonprogramming-python-numpy-activity-6617773953014427648-pOKW)
- [CleverCSV](https://github.com/alan-turing-institute/CleverCSV)
- [Data Cleaning Software Options - Pros And Cons](https://www.pivigo.com/blog?post=Data%20Cleaning%20Software%20Options%20-%20Pros%20And%20Cons)
- [Data Cleansing with Apache Spark and Optimus](https://www.linkedin.com/posts/data-science-central_data-cleansing-with-apache-spark-and-optimus-activity-6606304380595556352-zYFJ)
- [Basic Data Cleaning for Machine Learning (That You Must Perform)](https://machinelearningmastery.com/basic-data-cleaning-for-machine-learning/)
- [How to Perform Data Cleaning for Machine Learning with Python](https://machinelearningmastery.com/basic-data-cleaning-for-machine-learning/)

### Missing values

- [Working with missing data](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html)
- [Missing values in a dataset](https://www.datasciencecentral.com/profiles/blogs/how-to-treat-missing-values-in-your-data-1)
- [Iterative Imputation for Missing Values in Machine Learning](https://machinelearningmastery.com/iterative-imputation-for-missing-values-in-machine-learning/)
- [MissForest: The Best Missing Data Imputation Algorithm?](https://towardsdatascience.com/missforest-the-best-missing-data-imputation-algorithm-4d01182aed3)

### Imbalanced data

- [Develop a Model for the Imbalanced Classification of Good and Bad Credit](https://machinelearningmastery.com/imbalanced-classification-of-good-and-bad-credit/)
- [One-Class Classification Algorithms for Imbalanced Datasets](https://machinelearningmastery.com/one-class-classification-algorithms/)
- [Step-By-Step Framework for Imbalanced Classification Projects](https://machinelearningmastery.com/framework-for-imbalanced-classification-projects/)

## Data preprocessing / Data wrangling / Data manipulation

- [Data Preprocessing vs. Data Wrangling in Machine Learning Projects](https://www.infoq.com/articles/ml-data-processing)
- [Improve Model Accuracy with Data Pre-Processing](https://machinelearningmastery.com/improve-model-accuracy-with-data-pre-processing/)
- [5 Steps to correctly prepare your data for your machine learning model](https://towardsdatascience.com/5-steps-to-correctly-prep-your-data-for-your-machine-learning-model-c06c24762b73?gi=6b4a6895ab1)
- [Introduction to Data Analysis and Cleaning presentation](../presentations/data/01-mam-ml-study-group-meetup/Introduction_to_Data_Analysis_and_Cleaning.pdf) by [Mark Bell](http://www.nationalarchives.gov.uk/about/our-research-and-academic-collaboration/our-research-and-people/staff-profiles/mark-bell/)
- [Data Engineering Cook book](https://media.licdn.com/dms/document/C561FAQHLSPhPny5ENg/feedshare-document-pdf-analyzed/0?e=1572793200&v=beta&t=AW9YeWaxF4PCcSxN64Lin_BIlbumyHLHnL0gF-jLNto)
- [The Data Engineering Cookbook  ➡V2.2 August 10, 2019⬅](https://www.linkedin.com/posts/asif-bhat_data-engineering-activity-6622229032404652032-THi8)
- [Waimak: Open Source framework to create complex data flows in Apache Spark](https://github.com/CoxAutomotiveDataSolutions/waimak)
- [Pandas](https://lnkd.in/gxSgfuQ)
- [SQLAlchemy](https://lnkd.in/gjvbm7h)
- [Extract online data (code in Java)](https://www.datasciencecentral.com/profiles/blogs/java-coding-sample-to-extract-online-data)
- [Data preparation for factor analysis](https://www.linkedin.com/posts/data-science-central_data-preparation-for-factor-analysis-activity-6608507889915092992-T1Vv)
- [7 Pandas Functions to Reduce Your Data Manipulation Stress by Andre Ye](https://towardsdatascience.com/7-pandas-functions-to-reduce-your-data-manipulation-stress-25981e44cc7d) [LinkedIn Post](https://www.linkedin.com/posts/towards-data-science_7-pandas-functions-to-reduce-your-data-manipulation-activity-6655006784069214208-R9Zn)
- [Data Prepossessing](https://www.linkedin.com/posts/nabihbawazir_datascience-machinelearning-aertificialintellegence-activity-6657362083665018880-2rzc)
- [How to Grid Search Data Preparation Techniques](https://machinelearningmastery.com/grid-search-data-preparation-techniques/)
- [How to Choose Data Preparation Methods for Machine Learning](https://machinelearningmastery.com/choose-data-preparation-methods-for-machine-learning/)
- [More awesome pandas Graphical User Experience? Check pandas_ui a user interface tool for pandas providing easy and fast data manipulation](https://www.linkedin.com/posts/philipvollet_datascience-python-notebook-activity-6729843870093447168-vhfj)

### Transformations

- [How to Use Quantile Transforms for Machine Learning](https://machinelearningmastery.com/quantile-transforms-for-machine-learning/)
- [pre-processing filters like denoising, smoothing, thresholding, rescaling with python. As a part 3 of this series, in this post I have explained other pre-processing methods like masking, blending, image sharpening, erosion, dilation, geometric transformation](https://www.let-the-data-confess.com/image-pre-processing-through-opencv-part-3/)

### Scaling and normalisation

- [Scale, Standardize, or Normalize with Scikit-Learn](https://towardsdatascience.com/scale-standardize-or-normalize-with-scikit-learn-6ccc7d176a02)
- [Scale, Standardize or Normalize with scikit-learn](https://www.kaggle.com/discdiver/guide-to-scaling-and-standardizing)
- [Mean normalization and Feature scaling](https://www.linkedin.com/posts/data-science-central_mean-normalization-and-feature-scaling-activity-6617945072522575872-My8E)
- [How to Easily Calculate the Mean Absolute Deviation in Excel - Statology](https://www.statology.org/how-to-easily-calculate-the-mean-absolute-deviation-in-excel/) | [Wikipedia](https://en.wikipedia.org/wiki/Average_absolute_deviation)
- [Normalising a distribution notebook](../notebooks/data/data-processing/Normalising-a-distribution.ipynb)

## Misc

- [Pandera - Pandas data validation](https://www.linkedin.com/posts/madewithml_path-explain-made-with-ml-activity-6691703758189150208-KUwr)
- [Data Preparation for Data Mining](http://www.temida.si/~bojan/MPS/materials/Data_preparation_for_data_mining.pdf)
- [Learning with Limited Labeled Data with Shioulin Sam](https://twimlai.com/twiml-talk-255-learning-with-limited-labeled-data-with-shioulin-sam/)
- See discussion on [how data cleaning/preprocessing](https://www.meetup.com/Kaggle-Days-Meetup-London/events/258570474/comments/500079284/?read=1&_xtd=gatlbWFpbF9jbGlja9oAJDczM2Q5MDExLWYyZTctNDliNy1hZTgzLTk5NjFlMGViOGQ4Mw&itemTypeToken=COMMENT) went wrong resulting in poorly performing model

# Contributing

Contributions are very welcome, please share back with the wider community (and get credited for it)!

Please have a look at the [CONTRIBUTING](../CONTRIBUTING.md) guidelines, also have a read about our [licensing](../LICENSE.md) policy.

---

Back to [Data page (table of contents)](README.md)</br>
Back to [main page (table of contents)](../README.md)
