## Do we know our data, as good as we know our tools?

### Slides

See [slides (PDF)](Do_we_know_our_data,_as_good_as_we_know_our_tools_.pdf)

### Video

See [YouTube video on the Devoxx UK channel (~51 mins)](https://www.youtube.com/watch?v=rR6v5lw80nA&list=PLRsbF2sD7JVqkOs-GFGxBmNf0KECELaiU&index=74)

### Speakers

- [Mani Sarkar](http://github.com/neomatrix369)
- [Jeremie Charlet](http://twitter.com/jeremiecharlet)

### Abstract

See [Devoxx UK 2019 - talk abstract](https://devoxxuk19.confinabox.com/talk/VEM-8021/Do_we_know_our_data_as_good_as_we_know_our_tools) | [cached version of Talk Abstract](https://webcache.googleusercontent.com/search?q=cache:ggqgEPqxb0MJ:https://devoxxuk19.confinabox.com/talk/VEM-8021/Do_we_know_our_data,_as_good_as_we_know_our_tools%253F+&cd=1&hl=en&ct=clnk&gl=uk)

For many of us who are developer turning data scientist, we are always concerned about how to build a model, train it, etc... And yes, we want the best accuracy (close to 99%).

But as every seasoned data scientist will always advise us, we need first and foremost to understand our data, ensure it’s clean and prepared before doing any training on it.

During the conference, we will explore multiple problems occurring during data analysis or preparation and for each a technique to solve them (from a list of them). You will go away with a number of resources to explore at your own pace.

We will cover these categories of problems:

    dirty data
    disparate datasets - needing normalisation
    too much information to process
    and others…

We will cover some of these techniques:

    analysis - detecting misleading data, outliers, specific time series issues
    cleaning - deal with missing/ambiguous values, outliers, generating synthetic data, resampling
    preparation - using statistical and physics functions, dimensionality reduction, feature selection, resampling

And using different kinds of plots relevant at different stages.

### Code

- [Simple Data generation code](../../../notebooks/jupyter/data/data-generation)

### Notebooks

- [Notebooks used during the talk](../../../notebooks/jupyter/data/)
- Also see towards the bottom of [Notebooks](../../../notebooks/README.md)

