Using provided scripts to simulate a large E-Chord network on a single computer

The purpose of this tutorial is to use scripts provided with E-Chord to set up a large E-Chord network. As in previous tutorials, the network will run on a single computer. As such, we refer to it as a simulation, rather than a real network.

To follow along, you'll need to get both the seed server and the main E-Chord repositories. For more information on how to do this, refer to the setting up guide.

Starting the seed server

As usual, the first step to setting up an E-Chord network is executing the seed server. If you've done this before, feel free to repeat that process and skip this section.

Open a terminal at the location of the seed server and execute the command python3 server.py. You should see output similar to the following:

MainThread-INFO: Loaded params
MainThread-INFO: Starting node on 192.168.1.12:8000

If this isn't the case, the most common issue will be the unavailability of the port used. Open the params.json file, located in the same directory, and change the port under host to a different number. Try running the server again, until you you get output like the above.

Take note of the IP address and port number of the seed server, as seen in the output. You'll need this information for the next part.

Configuring node parameters

Before adding any nodes to the network, we must make sure we've set up the execution parameters as required. Those can be found in the main E-Chord repository, at config/params.json.

Open that file using a text editor of your choice. The contents should look something like this:

{
  "seed_server": {
    "ip": "192.168.1.12",
    "port": 8000,
    "attempt_limit": 4
  },
  "host": {
    "server_port": 9150
  },
  "ring": {
    "bits": 14,
    "fallback_fingers": 2,
    "stabilize_delay": 3
  },
  "net": {
    "timeout": 10,
    "connection_lifespan": 60,
    "data_size": 8192
  },
  "logging": {
    "level": "INFO"
  },
  "testing": {
    "initial_port": 9150,
    "total_nodes": 100,
    "stabilize_wait_period": 60,
    "percentage_to_remove": 50
  }
}

If these look overwhelming, don't worry. We'll only use a few here, so you can ignore most of these. For a complete list of parameters and their function, refer to the corresponding reference guide.

The first parameters we'll need to change can be seen first, under seed_server. We'll need to change the values for ip and port to the values we got when we started the seed server. The commonly used format for these is [ip address]:[port], so replace the default IP address and port with the ones your seed server is using.

For instance, our seed server output was 192.168.1.12:8000. Therefore, we set the IP address to 192.168.1.12, and the port to 8000, as seen above.

The next set of parameters we'll concern ourselves with are those under testing. These are parameters that describe the function of the simulation scripts. Specifically, we'll change the value of the total_nodes parameter to 40. This parameter describes the number of nodes that will actually be created in our simulation. In other words, the number of simulated computers in our network.

Feel free to use a different value if you want to! Just make sure it isn't too large, since it will affect your waiting time later on.

Once you've saved your changes, you're ready to move on to the next part.

Executing the visualizer

Next up, make sure to execute the visualizer script. Open a terminal window at the scripts directory, located in the main E-Chord directory, and execute python3 visualizer.py.

What you should see is an empty table. This is expected, since we haven't yet added any nodes to the network.

Adding nodes to the network

Now that our parameters are configured, we're ready to execute our first script. If you've followed the first tutorial, you might have noticed that adding each node to the network individually can be time-consuming. To avoid this, we can instead use a script to add a number of nodes, as determined by the parameter we set.

Open a new terminal window at the scripts directory and execute the command python3 mass_node_join.py. You'll start getting a large amount of output. You can ignore that and switch back to the visualizer window.

Once you do so, you'll see that every time the script updates, more nodes are added to the network. Pay attention to the IDs for each node's successor. You might see that for a lot of them, the successor isn't the same as the ID of the node below it. This is because the network hasn't yet stabilized, as indicated by the stability percentage at the bottom.

Wait and observe the visualizer. With time, you'll notice the stability percentage going up. If you look at individual nodes in the table, you'll find that more and more of them start pointing to the correct successor, the node under them in the table.

Eventually, the network stability percentage should reach 100%. Once it does, you're ready to move on to the next step (in fact, you can even move on if your stability percentage is high, but not yet 100%).

Inserting data

For this part, we'll need some additional data, in JSON format. Specifically, we'll want a JSON file that consists of a JSON Object, containing multiple keys. If you already have a file with that format, you may use that. If not, you can use this randomly generated sample file.

Open a new terminal window at the scripts directory and execute the command python3 mass_data.py [filedir] i. Replace [filedir] with the directory of the data file you're using (relative or absolute).

If the script executes correctly, you will see the script slowly inserting data into the network. You may switch over to the visualizer as well, where you'll notice the total number of keys increasing, as well as various nodes starting to hold those keys.

The mass_data script will also notify you of the failure percentage for the insertions. This is generally expected to be 0, but do not panic if it isn't! E-Chord is a distributed system and, as such, routing failures will always happen. In this case, the script will ignore keys that fail to be inserted, but in a real scenario, we would attempt to insert them again until we succeed.

Once the insertion script finishes, observe the visualizer window. Below is a sample of what part of the table should look like.

╔════════════════╦════════════════╦════════════╗
║   Successors   ║  Predecessors  ║    Keys    ║
╠════════════════╬════════════════╬════════════╣
║   493 -> 665   ║ 16263 <- 15162 ║  493: 380  ║
╠════════════════╬════════════════╬════════════╣
║   665 -> 1558  ║ 15162 <- 15000 ║  665: 106  ║
╠════════════════╬════════════════╬════════════╣
║  1558 -> 2178  ║ 15000 <- 14250 ║  1558: 604 ║
╠════════════════╬════════════════╬════════════╣
║  2178 -> 2358  ║ 14250 <- 13662 ║  2178: 350 ║
╠════════════════╬════════════════╬════════════╣
║  2358 -> 2449  ║ 13662 <- 13637 ║  2358: 115 ║
╠════════════════╬════════════════╬════════════╣

You'll find that the keys have been spread to multiple nodes, instead of all of them being saved in one. Pay attention to how the keys are distributed. On average, it will probably look like each node stores a similar number of keys, with the occasional extremes. A more careful look might, however, reveal that the number of keys stored in each node is somehow proportional to the distance between the node's ID and the previous node ID. The reason for this relates to the Chord protocol itself. For further analysis, visit the Chord protocol discussion.

At the bottom of the script, you'll get a sum of the total keys saved in the network:

Total keys: 9808
Stability: 100.00%

Because of routing failures, we were only able to save 9808 out of the 10000 keys we had. This might be different in your case.

Looking up inserted data

Now that we've inserted our data, it is time to figure out whether we can actually retrieve what we stored. To do this, we use the same script as before, only with l instead of i as a final argument. Specifically, the command we'll execute (at the scripts directory) is python3 mass_data.py [filedir] l.

Once the script is correctly executed, the output should look similar to the one of the insertion variant. This time, the failure percentage we see indicates lookup failures.

You'll observe that that percentage is likely higher than the insertion percentage from before. In fact, the lowest possible failure percentage you can expect here is the insertion failure percentage. This is because, since we're using the entire file again, we're looking up every key, including those that weren't inserted.

Still, if everything goes well, you should see a relatively low failure percentage. If you're instead getting very high failure percentages, it helps to restart your computer and try setting up the network again.

Removing half the network's nodes

A quality that is generally desired of any distributed system is the ability for fault tolerance. Distributed systems, by their nature, are always prone to various types of faults. One of those faults is the sudden loss of network nodes (because a computer went offline, or lost power, etc).

To test E-Chord's ability for tolerance to such faults, we're going to intentionally drop half the network's nodes. We'll then let the system stabilize and try to look up our data again.

To start this process, go to the scripts directory and execute the python3 mass_node_leave.py command. This will remove half the nodes from the network. If you return to the visualizer, you'll notice that some keys have been lost, and stability has dropped.

At this point, wait and observe the visualizer again. It will likely take a while, but eventually, stability should return to 100%. Nevertheless, once stability is high enough, we can move on to the next step.

Looking up data after simulated failure

Now that we've intentionally caused a mass node failure, and after the network has been given time to stabilize, we can look up our data again. At the scripts directory, execute the mass_data script again, in the same way we did before when we initially looked up the data.

You should notice that, even though the failure percentage will generally be higher, the network has more or less been able to come back from losing even as high a number of nodes as half its nodes at once. In fact, most of the failures will occur due to missing data, either because insertion itself failed, or because data might have been lost due to the node failures.

Conclusion

In this tutorial, you set up a simulation of a large E-Chord network on a single computer. Using the scripts, you managed to insert data into the network, and subsequently retrieve that data.

Afterwards, you caused a massive node failure and observed how the network re-stabilizes over time. Finally, you looked up your data again.