\subsection{Consensus Protocols in Blockchains}

Despite the popularity of cryptocurrencies, general and all-purpose blockchain systems that accommodate various applications, such as Ethereum \cite{wood2014ethereum} have been proposed. However, although Ethereum is a Turing-complete system \cite{wood2014ethereum}, it is in essence designed for the cryptocurrency based on it. Hence the Proof-of-Work (PoW) consensus has been utilized used to support the valuation of the cryptocurrency in the socioeconomic sense, which results in poor efficiency. To facilitate general-purpose applications in a more efficient manner, other consensus protocols have been proposed to improve the performance of the blockchain systems in different scenarios such as Proof-of-Stake \cite{kiayias2017ouroboros}, Proof-of-Luck \cite{milutinovic2016proof}, and Proof-of-Membership \cite{kogias2016enhancing}. Hyperledger Sawtooth \cite{sawtooth} utilizes Intel Software Guard eXtentions (SGX, introduced below) to trust nodes and proposes Proof-of-Elapsed Time believed to be highly efficient. Additionally, EOS \cite{eosio} based on DPoS, NEO \cite{hoxha2018hashgraph} based on DBFT, Conflux \cite{li2018scaling}, Omniledger \cite{kokoris2018omniledger}, and Hyperledger Fabric \cite{cachin2016architecture} are all examples of new blockchain systems which are claimed to achieve more than 10k transaction rate \cite{bach2018comparative}.

\begin{table}
	\begin{tabular}{l*{6}{c}r}
		Consensus & Tolerated Percentage & Examples \\
		\hline		
		\textit{PoW} & $<25\%$ & Bitcoin \\
		\textit{PoS} & $<51\%$ & PeerCoin \\
		\textit{PBFT} & $<33.3\%$ & HyperLedger Fabric \\
		\textit{DPOS} & $<51\%$ & Bitshares, EOS\\
		\textit{Ripple} & $<20\%$ & Ripple \\
		\textit{Tendermint} & $<33.3\%$ & Tendermint \\
	\end{tabular}
	\caption{Consensus Algorithms and Their Tolerated Percentages}
	\label{tab:packet1}
\end{table}

\subsection{Peer-to-Peer Overlay Networks}

There have been tremendous efforts and many technical innovations in the Internet broadcasting in the past three decades \cite{liu2008opportunities}. Internet protocol (IP) multicast represented an earlier attempt to tackle this problem but failed largely due to concerns regarding scalability, deployment, and support for higher level functionality. In contrast, Peer-to-peer based broadcast has been shown to be cost effective and easy to deploy. This new paradigm brings a number of unique advantages such as scalability, resilience, and effectiveness in coping with dynamics and heterogeneity. With Network Function Virtualization (NFV), upper-layer applications are allowed to control the lower-layer functionalities of the network such as routing.\\
The most important feature of the p2p architecture is that individuals in their network are equal in role and function. Although each individual may handle different requests, the actual resources provided may differ after specific quantification, but they can all simultaneously provide and consume resources. If the resources in the entire network, including but not limited to computing power, storage space, network bandwidth, etc., are regarded as a total amount, the resource distribution in the p2p network is dispersed among individuals (maybe not necessarily evenly distributed). Therefore, the p2p network architecture is naturally decentralized and distributed.
%There are tremendous P2P networks which we divide them into two groups: one group focusing on node discovery and the other group focusing on message dissemination.

\subsubsection{Data Look-up}

Not every individual communicates with other peers in the network. This is actually a very important feature of the p2p network: an individual only needs to connect with a small part of nodes in the network. How many neighbors and how to connect vary from one to another. Basically, P2P networks are divided into unstructured and structured networks \cite{qiu2007towards}. Unstructured P2P networks are simple and easy to deploy, and the individuals in the local area of ​​the network can be arbitrarily distributed. When dealing with a large number of new individuals joining the network and the old individuals leaving the network (churn), unstructured P2P networks are very stable \cite{stutzbach2006understanding}. The disadvantage is that the efficiency of finding data in the network is too low. Because there is no foreseeable information, it is often necessary to send query requests throughout the network (at least most individuals) or use flooding, which will occupy a large part of the network resources and greatly slow down other businesses in the same network.

The individual distribution of structured P2P networks has been carefully designed, and the main purpose is to improve the efficiency of querying data and reduce the resource consumption caused by query data. The basic means to improve query efficiency is to index data \cite{risson2006survey}. The most common implementation of structured p2p networks is distributed hash table (DHT) \cite{galuba2009distributed}, which assigns a key to each data (value) to form $(key, value)$ pairs. Hashing can be used to uniquely identify a particular object from a group of similar objects by assigning each object a hash value. Nodes in the system are responsible for managing the mapping from keys to values in a way that minimizing the disruption caused to the participants. In this way, when looking for a certain item of data, the search area can be continuously reduced according to the key, thereby greatly reducing resource consumption. Although structured P2P network is efficient in data look-up, the robustness is a concern. Since each individual needs to maintain a large number of neighboring individuals, when the churn events in which a large number of new and old individuals frequently join and leave occur in the network, the performance of the entire network will be greatly deteriorated. Part of the resources are consumed when updating the neighbor list (including the update of the neighbor list, and the stored list is updated between each other), and the keys of many peers also need to be redefined. Most used DHTs are Chord \cite{stoica2001chord}, Pastry \cite{rowstron2001pastry}, and Tapestry \cite{zhao2004tapestry}.

%\textbf{Chord}  is an algorithm for P2P DHT. It indicates how keys are assigned to nodes, and how a node can look up the value for a given key by first locating the node responsible for that key. Chord queries a key from a client (usually a node) to find the successor($k$). If the key can not be found locally, the query is passed to a node’s successor, which leads to a $O(N)$ query time where $N$ is the number of nodes in the ring. The implementation of a faster search method which requires each node to keep a so called "finger table" can avoid the linear search above. The finger table contains up to $m$ entries, where $m$ is the number of entries in the hash key. With this kind of a finger table, the number of nodes that must be contacted in an N-node network to find a successor becomes $O(log N)$.

%\textbf{Pastry} , is another overlay and routing network for the implementation of a DHT. Pastry uses consistent hashing as a hash algorithm. The key value obtained by hashing is one-dimensional (in fact, 128-bit integer space is used). Pastry does not specify which hash algorithm should be used. In the Pastry protocol, each node has a 128-bit identity (Node Id). In order to ensure the uniqueness of the Node ID, the network identifier (such as the IP address) of the node is generally obtained by hashing. Each node in Pastry has a routing table, a neighboring node set and a leaf node set, which together form the node's state table. Each routing step is closer to the target node than the previous step, so the process is convergent. If the routing table is not empty, at least one prefix matching digit can be added to each route. Therefore, when the routing table is always valid, the number of routing steps is at most $O(logBN)$ where $B$ is the system parameter. Pastry’s hash table has a circular key-space, just like Chord’s hash table. Node IDs are used to represent position in the circular key-space. These are chosen randomly so as to ensure that adjacent node IDs represent geographically diverse peers. Pastry's routing takes advantage of the proven maximum mask matching algorithm, so it can be implemented with many off-the-shelf software algorithms and hardware frameworks for good efficiency. Compared to Chord, Pastry introduces the concept of a set of leaf nodes and neighbor nodes. When the application layer can obtain the node information of the two sets in time and accurately, the speed of the route search can be greatly accelerated, and the network transmission overhead caused by the route can be reduced; but how to do this ideally in the dynamically changing P2P network does have some difficulty.

\subsubsection{Message Dissemination}

Gossip based protocols are developed for providing high reliability and scalability of message delivery \cite{islam2009efficient}. Gossip protocols are highly used for reducing control message overhead \cite{gupta2002fighting}. Gossip protocols are scalable because they do not require as much synchronization as traditional reliable multicast protocols. In gossip-based protocols, each node contacts one or a few nodes in each round usually chosen at random, and exchanges information with these nodes. The dynamics of information spread algorithm behavior stems from the work in epidemiology, and leads to high fault tolerance. Gossip-based protocols usually do not require error recovery mechanisms, and thus enjoy a large advantage in simplicity, while often incurring only moderate overhead compared to optimal deterministic protocols.

Unfortunately, gossip algorithms suffer from repeated messages which may lead to traffic congestion when broadcast frequency grows. There are several improvements made on gossip algorithms. Directional gossip uses a gossip server to construct spanning tree but it is not scalable. Intelligent select node selects directional children to build a tree. Some other improvements add TTL, use UID to reduce redundancy but still has overhead.

In addition, tree-based and data-driven broadcast/multicast algorithms in video streaming or file sharing does not fit in blockchain system context \cite{liu2008opportunities}. Tree-based approaches like SplitStream \cite{castro2003splitstream} and CoopNet \cite{padmanabhan2002distributing} is efficient but need to be maintained. It works poorly when dealing with node failures. For DHT, the cost of correcting the routing table of each node is also high. In addition to gossip, other data-driven approaches like ChainSaw \cite{pai2005chainsaw}, Bullet \cite{kostic2003bullet}, and CoolStream \cite{zhang2005coolstreaming} have reduced the redundancy of gossip a lot. However, pull-based broadcast and single-source broadcast/multicast do not fit into the context of blockchain systems.

\subsection{P2P Networks in Blockchain Systems}

\subsubsection{Requirements from Blockchain Systems}

Blockchain systems' requirements are different from other Peer-to-Peer applications: (i) on-demand streaming allows users to look up data in the P2P network and download stream data from the source, e.g. BitTorrent-based streaming systems like BASS \cite{dana2005bass}, Peer-Assisted \cite{carlsson2007peer}, LiveBT \cite{lv2007livebt}, and Give-To-Get \cite{mol2008give}; (ii) audio/video conferencing applications deal with small scale point-to-point connected networks which requires low latency, e.g. Skype \cite{baset2004analysis}; (iii) peer-to-peer file sharing makes efficient indexing and searching possible, e.g. Napster \cite{saroiu2003measuring}, Gnutella \cite{ripeanu2001peer}, and KaZaA \cite{good2003usability}; (iv) video streaming applications enables single-source broadcasting efficient, e.g. SplitStream \cite{castro2003splitstream}, Bullet \cite{kostic2003bullet}, and ChainSaw \cite{pai2005chainsaw}. (i) and (ii) are not relevant to the context of blockchain systems since nodes in a blockchain system network should be in a large scale and broadcasting a message is an active operation instead of searching and downloading data. (iii) and (iv) are more similar to blockchain systems' use case. However, P2P file sharing is not real-time and the broadcast model in a blockchain system is not indexing and searching. In video streaming, time is stringent and the network size can be large-scale. However, it is a data or bandwidth-intensive communication which means control messages in a broadcast operation are relatively small compared to the data to transmit.

\subsubsection{State-of-Art P2P Networks in Blockchain Systems}

Blockchain systems are either based random unstructured network or DHT-based structured network. Ethereum is implemented based on Kademlia \cite{maymounkov2002kademlia}, which is also a distributed hash table for decentralized P2P networks. Kademlia uses UDP for communication among peers and specifies the structure of the network and the exchange of information through node lookups. Similar to Pastry, each node is identified by a Node ID. Kademlia has many ideal features that previous DTHs could not provide at the same time. By incorporating broadcast configuration information into the loop-up messages, it minimizes the configuration messages that nodes must send in order to understand each other. Nodes have enough knowledge and flexibility to route queries through low latency paths. Kademlia uses concurrent asynchronous queries to avoid timeouts caused by node failures. Nodes record each other's existence against certain basic denial of service attacks.

While searching for $n$ nodes in a system, Kademlia only contacts $O(log(n))$ nodes, which is very efficient. Unlike first or second generation P2P file sharing networks such as Napster\cite{saroiu2001measurement} or the Gnutella\cite{ripeanu2001peer}, Kademlia uses DHTs to look up files in the network. Many of the advantages stem from the use of novel XOR metrics to define the distance between two points in the primary key space. XOR is symmetric, which allows Kademlia participants to receive query requests from the exact same node distribution contained in their routing tables. Without this feature, systems like Chord cannot learn useful routing information from the queries they receive. Worse, asymmetry can make routing tables less flexible. For example, in Chord, each finger table must store the exact nodes before an interval. In fact, any node within the interval and those nodes before the same interval may be physically far apart. In contrast, Kademlia can send queries to any node within an interval, which allows it to select the optimal route based on the delay, and even asynchronously query several equally suitable nodes in parallel.

Most existing blockchain systems use some form of gossiping to disseminate transactions, blocks, and membership information. By utilizing gossip, participants will eventually receive all transactions and blocks with high probability. For example, participants in Bitcoin \cite{nakamoto2008bitcoin} gossip with neighboring peers about recent transactions, blocks, and advertise membership of other participants. Hyperledger Fabric \cite{androulaki2018hyperledger} is a platform for deploying and operating permissioned blockchains. In Fabric, participants gossip about blocks, transactions, and membership information. Fabric divides gossiping into two modes: pull and push, where participants request state from other peers during pulling, and sends their state while pushing. Algorand \cite{gilad2017algorand} uses a similar gossip approach as Bitcoin \cite{nakamoto2008bitcoin}, where participants select a small subset of peers to gossip with.

\subsection{Trusted Execution Environment (TEE)}

Trusted computing has been defined to help systems to achieve secure computation, privacy and data protection. Originally, the Trusted Platform Module (TMP) allows a system to provide evidence of its integrity in a separate hardware module. In recent years, a new approach to address trusted computing has emerged, which allow the execution of arbitrary code within a confined environment that provides tamper-resistant execution to its applications - trusted execution environment (TEE) \cite{sabt2015trusted}. TEE is a secure, integrity-protected processing environment, consisting memory and storage capabilities \cite{asokan2014mobile}.

Intel SGX is one popular instance of TEE which is a set of extensions to the Intel architecture that aims to provide integrity and confidentiality guarantees to security sensitive computation performed on a computer where all the privileged software (kernel, hypervisor, etc.) is potentialy malicious \cite{costan2016intel}. Intel SGX provides two kinds of attestations (local and remote) to prove that particular piece of code is running in a genuine SGX-enabled CPU \cite{anati2013innovative, costan2016intel, mckeen2016intel}.
In a local attestation, an enclave directly attests
another enclave on the same machine using CPU instructions. 

In a remote attestation, SGX produces a report of measurements of an enclave (e.g., code, CPU generations), signs it and send it to a challenger. The challenger then connects to a Intel Attestation Service (IAS) and get a QUOTE to confrm that the code is running in a genuine Intel CPU. SGX also provides a linkable attestation mode for challengers to identify enclaves from the same machines. During a remote attestation, the challenger establish a secure communication channel with the help of key-exchange protocols (DifeHellman Key Exchange \cite{bresson2001provably}).

In addition to remote attestation, \xxx also uses a trustworthy source of random number provided by Intel SGX \cite{intelsgx} via its \texttt{sgx\_read\_rand} API \cite{intelsgx} which calls the hardware based pseudorandom generator (PRNG) through RDRAND on Intel CPUs \cite{costan2016}. Previous studies show that this random number generator is safe and cannot be altered from outside the enclave \cite{aumasson2016sgx, hamburg2012analysis, mofrad2017leveraging}.

%Currently there is one related work which uses Intel SGX to provide reliable broadcast for P2P network \cite{jia2017robust}. However, there is no related work on using Intel SGX to improve asynchronous P2P network performance, which is the main focus of this project.


%\subsection{Design Motivation}

%Talk about the design motivation, kind of combining the two.