title: pandas
category: page
slug: pandas
sortorder: 0316
toc: False
sidebartitle: pandas
meta: The Python Data Analysis Library (pandas) is a data structures and analysis library.


The [Python Data Analysis Library (pandas)](http://pandas.pydata.org/)
is a data structures and analysis library.

<a href="http://pandas.pydata.org/" style="border: none;"><img src="/img/logos/pandas.png" width="100%" alt="Python Data Analysis Library (pandas) logo." class="shot"></a>


### pandas resources
* [Intro to pandas data structures](http://www.gregreda.com/2013/10/26/intro-to-pandas-data-structures/), 
  [working with pandas data frames](http://www.gregreda.com/2013/10/26/working-with-pandas-dataframes/)
  and
  [Using pandas on the MovieLens dataset](http://www.gregreda.com/2013/10/26/using-pandas-on-the-movielens-dataset/)
  is a well-written three-part introduction to pandas blog series that 
  builds on itself as the reader works from the first through the third
  post.

* [pandas exercises](https://github.com/guipsamora/pandas_exercises) is
  a GitHub repository with Jupyter Notebooks that let you practice
  sorting, filtering, visualizing, grouping, merging and more with pandas.

* [A simple way to anonymize data with Python and Pandas](https://dev.to/r0f1/a-simple-way-to-anonymize-data-with-python-and-pandas-79g)
  is a good tutorial on removing sensitive data from your unfiltered 
  data sets.

* [Learn a new pandas trick every day!](https://www.dataschool.io/python-pandas-tips-and-tricks/)
  is a running list of great pandas tips that the author originally
  posted on Twitter and then aggregated onto a single webpage.

* [Time Series Analysis with Pandas](https://www.dataquest.io/blog/tutorial-time-series-analysis-pandas/)
  show you how to combine [Python 3.6](/why-use-python.html), pandas, 
  [matplotlib](/matplotlib.html) and [seaborn](/data-visualization.html)
  to analyze and visualize open data from Germany's power grid. This is 
  a great tutorial to learn these tools with a realistic data set.
  
* [Analyzing a photographer's flickr stream using pandas](https://www.turbowhale.com/posts/analyze_flickr_stream_pandas/)
  explains how the author grabbed a bunch of Flickr data using the 
  [flickr-api](https://github.com/alexis-mignon/python-flickr-api) library
  then analyzed the EXIF data in the photos using pandas.

* [Pandas Crosstab Explained](http://pbpython.com/pandas-crosstab.html)
  shows how to use the `crosstab` function in pandas so you can summarize
  and group data.

* [Calculating streaks in pandas](https://joshdevlin.com/blog/calculate-streaks-in-pandas/)
  shows how to measure and report on streaks in data, which is where 
  several events happen in a row consecutively.

* [How to Convert a Python Dictionary to a Pandas DataFrame](https://www.marsja.se/how-to-convert-a-python-dictionary-to-a-pandas-dataframe/)
  is a straightforward tutorial with example code for loading and adding 
  data stored in a typical Python dictionary into a DataFrame.

* This two-part series on loading data into a pandas DataFrame presents
  what to do
  [when CSV files do not match your expectations](http://blog.enthought.com/python/with-and-without-the-canopy-data-import-tool-loading-data-theres-no-such-thing-as-a-simple-csv-file/)
  and 
  [how to handle missing values](http://blog.enthought.com/enthought-canopy/data-import-tool/handling-missing-values-pandas-dataframes-hard-way-easy-way/)
  so you can start performing your analysis rather than getting frustrated
  with common issues at the beginning of your workflow.

* [Building a financial model with pandas](http://pbpython.com/amortization-model.html)
  explains how to create an amortization schedule with corresponding table
  and charts that show the pay off period broken down by interest and
  principal.

* [Efficiently cleaning text with pandas](https://pbpython.com/text-cleaning.html)
  provides a really great practical tutorial on different approaches
  for cleaning a large data set so that you can begin to do your analysis.
  The tutorial also shows how to use the 
  [sidetable](https://github.com/chris1610/sidetable) library, which
  creates summary tables of a DataFrame.

* [tabula-py: Extract table from PDF into Python DataFrame](https://blog.chezo.uno/tabula-py-extract-table-from-pdf-into-python-dataframe-6c7acfa5f302)
  presents how to use the Python wrapper for the 
  [Tabula](https://tabula.technology/) library that makes it easier to
  extract table data from PDF files.

* [Time Series Forecast Case Study with Python: Monthly Armed Robberies in Boston](https://machinelearningmastery.com/time-series-forecast-case-study-python-monthly-armed-robberies-boston/)
  walks through the data wrangling, analysis and visualization steps
  with a public data set of murders in Boston from 1966 to 1975. This
  particular data problem may not be your thing but by going through
  the process you can learn a lot that can be applied to any data set.

* [A Gentle Visual Intro to Data Analysis in Python Using Pandas](https://jalammar.github.io/gentle-visual-intro-to-data-analysis-python-pandas/)
  presents spreadsheet-like pictures to show conceptually what
  pandas is doing with your data as you apply various functions like
  `groupby` and `loc`.

* [Data Manipulation with Pandas: A Brief Tutorial](https://www.marsja.se/data-manipulation-pandas-tutorial/)
  uses some example data sets to show how the most commonly-used functions
  in pandas work.

* [Analyzing Pronto CycleShare Data with Python and Pandas](https://jakevdp.github.io/blog/2015/10/17/analyzing-pronto-cycleshare-data-with-python-and-pandas/)
  uses Seattle bikeshare data as a source for wrangling, analysis and 
  visualization.

* [Stylin' with pandas](https://pbpython.com/styling-pandas.html) shows how
  to add colors and sparklines to your output when using pandas for data 
  visualization.

* [Python and JSON: Working with large datasets using Pandas](https://www.dataquest.io/blog/python-json-tutorial/)
  is a well-done detailed tutorial that shows how to mung and analyze
  JSON data.

* [Fun with NFL Stats, Bokeh, and Pandas](https://j253.github.io/blog/fun-with-nfl-stats.html) 
  uses National (American) Football League data as a source for 
  wrangling and visualization.

* [Analyzing my Spotify Music Library With Jupyter And a Bit of Pandas](https://vsupalov.com/analyze-spotify-music-library-with-jupyter-pandas/)
  shows how to grab all of your user data from the Spotify API then
  analyze it using pandas in [Jupyter Notebook](/jupyter-notebook.html).

* [Scalable Python Code with Pandas UDFs](https://towardsdatascience.com/scalable-python-code-with-pandas-udfs-a-data-science-application-dd515a628896)
  explains that pandas operations can often be parallelized for better 
  performance using the Pandas UDFs feature in PySpark version 2.3
  or greater.

* [How to use Pandas read_html to Scrape Data from HTML Tables](https://www.marsja.se/how-to-use-pandas-read_html-to-scrape-data-from-html-tables/)
  has a bunch of great code examples that show how to load
  data from HTML directly into your DataFrames.

* [How to download fundamentals data with Python](http://theautomatic.net/2020/05/05/how-to-download-fundamentals-data-with-python/)
  shows how to obtain and use financial data, such as balance sheets,
  stock prices, and various ratios to perform your own analysis on.

* [How to convert JSON to Excel with Python and pandas](https://www.marsja.se/how-to-convert-json-to-excel-python-pandas/)
  provides instructions for creating a spreadsheet out of JSON file.

* [Loading large datasets in Pandas](https://towardsdatascience.com/loading-large-datasets-in-pandas-11bdddd36f7b)
  explains how to get around the `MemoryError` issue that occurs
  when using `read_csv` because the data set is larger than the
  available memory on a machine. You can use chunking with
  the `read_csv` function to divide the data set into smaller parts that
  each can be loaded into memory. Alternatively, you can use a 
  [SQLite database](/sqlite.html) to create a [relational database](/databases.html)
  with the data then use SQL queries or an 
  [object-relational mapper (ORM)](/object-relational-mappers-orms.html)
  to load the data and perform analysis in pandas.

* Real-world Excel spreadsheets are often a mess of unstructured data, so
  this tutorial on 
  [Reading Poorly Structured Excel Files with Pandas](https://pbpython.com/pandas-excel-range.html)
  gives example code for extracting only part of a file as well
  as reading ranges and tables.

