Communicating with an active E-Chord network through separate software

In this tutorial, we will create software that communicates with an active E-Chord network and uses it to store and retrieve data.

We'll assume that you already have an E-Chord network up and running. This can either be a network simulation (running on one machine), or an actual E-Chord network. If you haven't done this yet, consider taking a look at the previous tutorials.

A basic program using a hash table

Since E-Chord is a distributed hash table, we'll first write a program that uses a regular hash table. After this, we'll replace the hash table with E-Chord, and show that the functionality is the same.

We'll write the program in Python, where the equivalent of a hash table is a dictionary. The program will implement a simple login system. Users will be able to register using a username and password, and login to the system.

Obviously, we won't focus on any sort of security for this. We also won't implement any form of defensive programming. This will only serve to demonstrate the function of E-Chord as a distributed hash table.

users = {}

while True:
    print("Welcome to Login System!")
    c = input("Would you like to login or register (l/r)? ")

    if c == "l":
        username = input("Username: ")
        password = input("Password: ")

        if username not in users or users[username] != password:
            print("Incorrect credentials!")
            continue

        print(f"Welcome, {username}!")

    elif c == "r":
        username = input("Username: ")
        password = input("Password: ")

        if username in users:
            print("Username taken!")
            continue

        users[username] = password
        print("Successfully registered!")

The above program uses a hash table to store usernames, with the value corresponding to their password. Whenever a new user registers, the hash table is checked for the existence of the username. If it doesn't exist, it is added with the provided password as a value.

When a user tries to log in, the hash table is checked for the existence of the username. If it exists and the value is the same as the provided password, login is successful.

Below is a sample execution of the program:

Welcome to Login System!
Would you like to login or register (l/r)? r
Username: Emily
Password: ilovecats2
Successfully registered!
Welcome to Login System!
Would you like to login or register (l/r)? l
Username: Emily
Password: ilovewats2
Incorrect credentials!
Welcome to Login System!
Would you like to login or register (l/r)? l
Username: Emily
Password: ilovecats2
Welcome, Emily!

Communication with the E-Chord network

When communicating with an E-Chord network, we essentially need to talk to one of the nodes of the network. As such, we need to know the address of some node. If you've started the network yourself, you should know the addresses of the nodes within it, so use one of them.

If you've used the scripts to create the network, open the config/params.json file. Under testing, you'll find the parameter initial_port. Its value is the port of the first node that was started by the script, incremented by 1 for each new node. As such, one possible address you can use is localhost:[initial_port]. If that one doesn't work, increment the port value by 1 and try again.

Another way to find an active node in the network is simply by observing the output of the seed server. The seed server regularly polls nodes to confirm they're still there, so looking at its output, you'll find the addresses of recently polled nodes. Remember that an empty IP address string is equivalent to localhost.

E-Chord nodes communicate using TCP sockets. Additionally, messages between nodes are all in JSON, and follow a specific format. For an in-depth explanation on how to structure your messages when communicating with the network, refer to this guide and the equivalent reference guide.

In order to simplify this process here, we will use an existing function. This function, called ask_node, connects to the node with the address we provide and makes a request of a certain type.

import socket
import json

def ask_node(peer_addr, req_type, body_dict, custom_timeout=10):
    """
    Sends a request and returns the response
    :param peer_addr: (IP, port) of peer
    :param req_type: type of request for request header
    :param body_dict: dictionary of body
    :param custom_timeout: timeout for request
    :return: string response of peer
    """
    request_dict = {"header": {"type": req_type}, "body": body_dict}
    request_msg = json.dumps(request_dict, indent=2)

    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as client:
        client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        client.settimeout(custom_timeout)
        try:
            client.connect(peer_addr)
            client.sendall(request_msg.encode())
            data = client.recv(8192).decode()
        except (socket.error, socket.timeout):
            return None

    if not data:
        return None

    return json.loads(data)

There are various request types that can be made. As mentioned before, a full list can be viewed in the corresponding guides. As we move forward, we'll translate the dictionary (i.e hash map) operations into the appropriate requests for the E-Chord network.

Points of interest in the hash table program

Let's return to the Login System program from before and identify the points where dictionary operations are used. We'll then discuss how these operations can be translated into their equivalents for E-Chord.

Firstly, at the very top, we create an empty dictionary.

The next time the dictionary is used is when a user tries to login.

if username not in users or users[username] != password:

In fact, there are two dictionary operations here. The first one is username not in users, which checks for the existence of a key in the dictionary, while the second is users[username], which retrieves a value from the dictionary by key.

Finally, we use the dictionary twice when a user registers. Firstly, like before, we check for the existence of a username:

if username not in users:

If a username doesn't exist, we register the user with the credentials given:

users[username] = password

In total, we have four points where we perform dictionary operations. Two of those consist of checking for the existence of a key, one of getting the value for a key and one of inserting a new pair.

Equivalent E-Chord operations

To perform these operations in the E-Chord network, we'll have to make the appropriate requests to a node in the network. We have three types of operations, each of which will require some type of request.

To check the existence of a key in the network, we need to make a request of type find_key. This request type also expects the key we're looking for as a parameter.

To retrieve the value for a given key, we also use the same request type, find_key.

To insert a new pair into the network, we use the request type find_and_store_key. This request type expects two parameters, the key we want to insert and the value to store for that key.

If you take a look at the ask_node function's arguments, you'll notice that there are three arguments we must pass to it. Those are peer_addr, req_type and body_dict. The first one is a tuple, containing the IP address (as a string) and the port (as an integer) of the node to connect to. The second is the request type, as described above. The third is a dictionary of the parameters that must be included for that request type.

Handling responses

Since E-Chord is a network, whenever we make a request to one of its nodes, we expect a response. Just like the requests, E-Chord's responses follow a specific format. In general, after we make a request, we receive a response that contains a header and a body. The header contains metadata about the response, while the body contains the response data itself.

Usually, this means that we check the header for the request status and the body for resulting data. In fact, every response will always contain a status field in the header. Its value is an integer code, borrowed from HTTP response status codes, that indicates the response status.

The same program using E-Chord

Let's now attempt to rewrite the Login System program, using E-Chord as our dictionary. We'll start by placing the code for the ask_node function in a separate file, ask_node.py, and then import it into our program, using the line from ask_node import ask_node.

Since the E-Chord network is already running, we don't need to do anything when it comes to creating our dictionary. Next, let's look at the first two instances where the dictionary is used, when a user logs in.

if username not in users or users[username] != password:

Here, we need to check for the existence of the username. If the username exists, we need to check whether the provided password matches the one we have in storage.

Remember how the request type for checking whether a key exists and for retrieving a key's value is the same? This is convenient, because we would prefer to only make one request here, not two. As such, we'll ask our node to find the provided username. The response will indicate if it exists and, if it does, will contain the password.

response = ask_node(NODE_ADDR, "find_key", {"key": username})

We then need to determine whether the key exists. To do this, we check the response status. It will either be 200, which means the key exists, or 404, which means it doesn't. If it does, we move to the response body, where the value field will contain the value for the key, which is the password.

if response["header"]["status"] == 404 or response["body"]["value"] != password:
    print("Incorrect credentials!")
    continue

As for the register part, we need to check if a username already exists, so as not to override it. This is done exactly as above. In case the username does not exist, we need to store a pair for the new user. To do this, we use the find_and_store_key request type.

response = ask_node(NODE_ADDR, "find_and_store_key", {"key": username, "value": password})

In theory, this request should never fail. In practice, a routing error could cause the response status to be 404. If that were to happen, we could repeat the request, possibly to a different node. For the purposes of this tutorial, however, that will not be necessary.

This is it! The full version of the program, using E-Chord as its dictionary, can be seen below.

from ask_node import ask_node

NODE_ADDR = ("localhost", 9150)

while True:
    print("Welcome to Login System!")
    c = input("Would you like to login or register (l/r)? ")

    if c == "l":
        username = input("Username: ")
        password = input("Password: ")

        response = ask_node(NODE_ADDR, "find_key", {"key": username})

        if response["header"]["status"] == 404 or response["body"]["value"] != password:
            print("Incorrect credentials!")
            continue

        print(f"Welcome, {username}!")

    elif c == "r":
        username = input("Username: ")
        password = input("Password: ")

        response = ask_node(NODE_ADDR, "find_key", {"key": username})

        if response["header"]["status"] == 200:
            print("Username taken!")
            continue

        response = ask_node(NODE_ADDR, "find_and_store_key", {"key": username, "value": password})

        print("Successfully registered!")

Let's now try the same sample execution as with the regular version of the program.

Welcome to Login System!
Would you like to login or register (l/r)? r
Username: Emily
Password: ilovecats2
Successfully registered!
Welcome to Login System!
Would you like to login or register (l/r)? l
Username: Emily
Password: ilovewats2
Incorrect credentials!
Welcome to Login System!
Would you like to login or register (l/r)? l
Username: Emily
Password: ilovecats2
Welcome, Emily!

As expected, our output is exactly the same as before! This means E-Chord has correctly stored and retrieved the pair for us, and we have communicated with it through our own software.

At this point, play around with the Login System programs. Try different inputs for both, and see if the version using E-Chord works as expected.

Do not forget that, since the network is running separately to your program, if you quit and start your program again, the users you've already created will still be there (since they're stored on the network). This is contrary to the original version, where restarting the program erases the dictionary data you've previously inserted.

If the E-Chord network you're using is simulated on one computer, you may also run the visualizer script at the scripts directory to view the nodes in the network and see which of them store the keys you insert.

Conclusion

In this tutorial, you set up a simple program of your own, which communicates with an E-Chord network directly and uses it as its dictionary to store data. You made requests towards a node in the network and read its responses to achieve this.

For a full list of requests and their expected responses, visit the communication reference.