A blockchain is essentially a distributed ledger that permanently records the transactions among parties \cite{iansiti2018truth}. The transactions recorded are verifiable and resistant to modification without the need of a centralized third party. The decentralization nature and proved immutability have led to the emergence of cryptocurrencies which leverages the blockchain system as their cornerstone, the most salient example being Bitcoin \cite{nakamoto2008bitcoin}. Blockchain systems are based on consensus protocols to achieve chain consistency. The communication is based on a Peer-to-Peer network. With tremendous works focused on consensus layer to increase the blockchain system efficiency and transaction rate, broadcast frequency in the P2P network is growing.

%Despite the popularity of cryptocurrencies, general and all-purpose blockchain systems that accommodate various applications, such as Ethereum \cite{wood2014ethereum} have been proposed. However, although Ethereum is a Turing-complete system \cite{wood2014ethereum}, it is in essence designed for the cryptocurrency based on it. Hence the Proof-of-Work (PoW) consensus has been utilized used to support the valuation of the cryptocurrency in the socioeconomic sense, which results in poor efficiency. To facilitate general-purpose applications in a more efficient manner, other consensus protocols have been proposed to improve the performance of the blockchain systems in different scenarios such as Proof-of-Stake \cite{kiayias2017ouroboros}, Proof-of-Luck \cite{milutinovic2016proof}, and Proof-of-Membership \cite{kogias2016enhancing}. Hyperledger Sawtooth \cite{sawtooth} utilizes Intel Software Guard eXtentions (SGX, introduced below) to trust nodes and proposes Proof-of-Elapsed Time believed to be highly efficient. Additionally, EOS \cite{eosio} based on DPoS, NEO \cite{hoxha2018hashgraph} based on DBFT, and Hyperledger Fabric \cite{cachin2016architecture} are all examples of new blockchain systems which are claimed to achieve more than 10k transaction rate \cite{bach2018comparative}.

Unfortunately, experiments and studies \cite{cachin2016architecture, xueos, li2018scaling} already show that the network layer became the bottleneck of further growing the transaction rate. For example, EOS report \cite{xueos} shows that EOS is sensitive to network latency. Higher network latency caused by other bandwidth-intensive applications in the same network or high client transaction input rate can both lead to significant drop of the throughput. Hyperledger Fabric (HLF) \cite{cachin2016architecture} also shows that its throughput is capped by the low efficiency of the underlying P2P network. 
% However, current blockchain systems are based on Distributed Hash Table (DHT) structure and message broadcast in a Gossip manner \cite{eugster2004epidemic}.

However, broadcasting messages suffers from traffic congestion and inefficient convergence problems when transaction rate gets higher. Our key insight is that the gossip algorithm \cite{eugster2004epidemic} used to broadcast a message does not fit in the demand for P2P networks from blockchain systems. Although many improvements have been made on gossip algorithm such as adding unique message ID, using Time-to-Live field to control flooding, using pull-version sending mechanism to reduce repeated messages, our evaluations show that they are not efficient enough, and still suffer from traffic congestion problem. It is not an optimal model for blockchain system. Unstructured networks are robust but lead to inefficiency in both node discovery and broadcast. Structured networks based on DHT are efficient in terms of node discovery and data look up (they are designed for this use \cite{stoica2001chord}), however, gossiping on such a network is inefficient. Since blockchain systems emphasize broadcast more than node discovery, a new P2P network protocol is needed.

% Blockchain systems' requirements are different from other Peer-to-Peer applications: (i) on-demand streaming allows users to look up data in the P2P network and download stream data from the source, e.g. BitTorrent-based streaming systems like BASS \cite{dana2005bass}, Peer-Assisted \cite{carlsson2007peer}, LiveBT \cite{lv2007livebt}, and Give-To-Get \cite{mol2008give}; (ii) audio/video conferencing applications deal with small scale point-to-point connected networks which requires low latency, e.g. Skype \cite{baset2004analysis}; (iii) peer-to-peer file sharing makes efficient indexing and searching possible, e.g. Napster \cite{saroiu2003measuring}, Gnutella \cite{ripeanu2001peer}, and KaZaA \cite{good2003usability}; (iv) video streaming applications enables single-source broadcasting efficient, e.g. SplitStream \cite{castro2003splitstream}, Bullet \cite{kostic2003bullet}, and ChainSaw \cite{pai2005chainsaw}. (i) and (ii) are not relevant to the context of blockchain systems since nodes in a blockchain system network should be in a large scale and broadcasting a message is an active operation instead of searching and downloading data. (iii) and (iv) are more similar to blockchain systems' use case. However, P2P file sharing is not real-time and the broadcast model in a blockchain system is not indexing and searching. In video streaming, time is stringent and the network size can be large-scale. However, it is a data or bandwidth-intensive communication which means control messages in a broadcast operation are relatively small compared to the data to transmit.

Blockchain systems require two main functions from the underlying P2P network from : peer discovery and message dissemination. For peer discovery, DHT-based protocols such as Chord \cite{stoica2001chord}, Pastry \cite{rowstron2001pastry}, Tapestry \cite{zhao2004tapestry}, and Kademlia \cite{maymounkov2002kademlia} are used to achieve efficiency. For message dissemination, gossip algorithms are used due to its robustness and simplicity. Under 50\% failure, gossip can send twice amount of messages to cover the remaining nodes. However, the robustness of gossip exceeds too much of the requirement from the consensus protocols in most blockchain systems. As a side effect, gossip generates redundant messages in the network which lead to traffic congestion. To tackle the problem, our key idea is that broadcasting using the DHT-based structure in a hidden and secure way can improve the broadcast efficiency in terms of both time complexity and message complexity.

We implement our idea in HGFRR, which is a \textbf{H}idden \textbf{G}eographical \textbf{F}ractal \textbf{R}andom \textbf{R}ing structured P2P network. HGFRR contains multi-level fractal random rings. Each ring at the lower level will have a couple of contact nodes which is randomly selected to represent the ring in the upper level ring. Unlike DHT-based protocols which indexes the whole network as a ring, HGFRR recursively constructs rings based on the proximity of peers and the number of peers in a ring. The broadcast on HGFRR is recursively performed on each ring. The in-ring broadcast uses the k-ary distributed spanning tree formed within the ring. Both the proof and evaluation show that the broadcast operation in HGFRR is more efficient than the P2P network in Ethereum, in terms of time complexity and message complexity. The message complexity of our network with $N$ nodes is $O(N)$ and time complexity of message broadcast is logarithm, which are both better than extant work.

The paper makes the following contributions. First, the paper identifies requirements for the P2P networks from blockchain systems and pointed out the over-robustness and thus the inefficiency of current message dissemination algorithm. Second, it presents and proves a new network protocol HGFRR that improves message dissemination efficiency and thus the overall throughput of the blockchain system. Third, HGFRR is the first P2P network in blockchain systems that addresses geographical locality and security problems. Fourth, HGFRR is implemented in C++ which is portable and runnable across-platform and intensively evaluated.

The remaining of this paper is organized as follows. Section \cref{background} introduces the background of this problem and related works. Section \cref{design} presents the design overview of HGFRR including topology and protocols. Section \cref{analysis} gives a proof for time and message complexity and analyzes the robustness of HGFRR on node failures. Section \cref{implementation} describes the implementation details. Section \cref{eval} presents and discusses the evaluation results. Section \cref{conclusion} concludes the paper.