# Craigslist Filter
#### In Development...

Craigslist Filter (working title) is a web application built with python and flask which scrapes data about vehicles for sale from all Craigslist pages in America and allows users to filter them by criteria such as city, price, manufacturer, and odometer. This project is currently in development and is not yet completed.

## Installing

This application requires Python 3.6 or greater and several installable modules.

You will need to install flask, sqlite3, wtforms, lxml, and requests_html.

```
pip3 install Flask
```
```
pip3 install setuptools
```
```
pip3 install pysqlite3
```
```
pip3 install wtforms
```
```
pip3 install lxml
```
```
pip3 install requests_html
```
```
pip3 install flask_wtf
```
```
pip3 install flask_bootstrap
```
```
pip3 install geopy
```

## Deploying

To run this application locally you will first need to run both crawlCities.py and scrapeVehicles.py (or download a cached version [here](https://files.fm/u/p5z4fbkn)) in order to generate the databases used by the application.

Once these applications have completed, simply run app.py and copy and paste the address provided in the terminal into your browser.

## Specific Future Implementations

* Remain on the form page when a search yields no results.

* Allow users to specify a search radius and return more specific results when searching by location.

* Add Google maps API feature to allow users to browse sales in specific areas.

## Broad Future Implementations

* Integrate visualization project with filter, let users generate graphs to help narrow decisisons when purchasing cars e.g. show me a line graph of the average price of Ford pickups based on the odometer of the vehicle (this code has already been written with pandas, rewriting with SQL will take some time).

* Login/Logout functionality which allows users to save certain filter combinations and search results.

* Better site layout, less bootstrap-esque and more creative.

* Improved security.

* Frequent automated database updates.

* User-specific sale tracking (price has changed, listing has been removed, etc.).

## Blocked

* Pivot to multiprocessing to allow for many requests to be made at once, speeding up the scraper exponentially (I am worried about Craigslist blacklisting IPs).

## Completed Tasks

* Filter implemented.

* Improved filter form including dropdown lists automatically generated by column entries in the database.

* Scraped the map on the listing page to extract more specific location (lat/long) instead of just the region.

* Added a message that alerts a user when their search yields no results.

* Allowed for filtering between two values for fields such as price and odometer.

* Added photos to results page.

* Allow for users to search by any city using latitude and longitude instead of specific craigslist regions.

* Track which cities have been scraped recently to add order to the scraping process.

## Contributors

This application is being developed by Austin Reese.
