{% extends "layout.html" %}
{% from 'ruler.macro.html' import ruler %}
{% block body %}

{{ ruler() }}
<a style="display:none">src.html</a>
<div class="contents">
	<h2>License</h2>
	<p>This software is made available under the <a href="https://www.gnu.org/licenses/agpl-3.0.en.html">GNU Affero GPL 3 License.</a>. What this means is that is you deploy this software as part of networked software that is available to the public, you must make the source code available (and any modifications).
	</p>
	<p> From the GNU site:
	<blockquote>
	The GNU Affero General Public License is a modified version of the ordinary GNU GPL version 3. It has one added requirement: if you run a modified program on a server and let other users communicate with it there, your server must also allow them to download the source code corresponding to the modified version running there.
	</blockquote>
	</p>
	<h2>Download</h2>
	<p>
	The source code is available on GitHub:<a href="https://github.com/GoSecure/darkweb-search-engine">https://github.com/GoSecure/darkweb-search-engine</a>. The entirety of the project can be cloned from here: <a href="https://github.com/GoSecure/darkweb-search-engine.git">https://github.com/GoSecure/darkweb-search-engine.git</a>. We decided to fork this project to add functionalities and fix bugs. This is the original link to the project: <a href="https://github.com/dirtyfilthy/darkweb-search-engine">https://github.com/dirtyfilthy/darkweb-search-engine</a>
	</p>
	<h2>Infrastructure</h2>
	<p>(This came from the original project:<a href="https://github.com/dirtyfilthy/darkweb-search-engine">https://github.com/dirtyfilthy/darkweb-search-engine</a>)
	<br><br>
	Nexvisions runs on two servers, a frontend host running the database and hidden service web site, and a backend host running the crawler. Probably most interesting to the reader is the setup for the backend. TOR as a client is COMPLETELY SINGLETHREADED. I know! It's 2017, and along with a complete lack of flying cars, TOR runs in a single thread. What this means is that if you try to run a crawler on a single TOR instance you will quickly find you are maxing out your CPU at 100%.
	</p>
	<p>
	The solution to this problem is running multiple TOR instances and connecting to them through some kind of frontend that will round-robin your requests. The Nexvisions crawler runs eight Tor instances.
	</p>
	</p>
	Debian (and ubuntu) comes with a useful program "tor-instance-create" for quickly creating multiple instances of TOR. I used Squid as my frontend proxy, but unfortunately it can't connect to SOCKS directly, so I used "privoxy" as an intermediate proxy. You will need one privoxy instance for every TOR instance. There is a script in "scripts/create_privoxy.sh" to help with creating privoxy instances on debian systems. It also helps to replace /etc/privoxy/default.filter with an empty file, to reduce CPU load by removing unnecessary regexes.
	</p>
	<p>
	Additionally, this resource <a href="https://www.howtoforge.com/ultimate-security-proxy-with-tor"</a>https://www.howtoforge.com/ultimate-security-proxy-with-tor</a> might be useful in setting up squid. If all you are doing is crawling and don't care about anonymity, I also recommend running TOR in tor2web mode (required recompilation) for increased speed.
	</p>
</div>
{% endblock %}