Wesh: Flight of a Byte

In the previous blog post, we made an example app where the Wesh user account of “client2” sends a message “Hello” to a Contact group which is received by the other user account of “client1”. You can use the example to start building your Wesh app, but for development and debugging it helps to understand more about how it works. So this blog post will be a theory dive into some of the details of Wesh communication.

When the Wesh app on the computer for client2 sends the bytes of a message, what is the “flight path” of the bytes of “Hello” as they travel to the computer of client1? It's different from “traditional” networking where, for example, a computer sends the message in a UDP packet to the IP address of the other computer, as described in this Wikipedia article. Instead of a specific IP address, Wesh communication happens between libp2p peers, where the client's PeerID is the value that we've seen in previous examples and is the same no matter how the client is connected to the network.

Peer discovery

The “glue” between a libp2p peers and a more traditional network address is peer discovery. There are a lot of details that we won't go into regarding types of discovery, peer authentication, etc. In this blog post, we just describe the minimal steps for how client1 receives the message from client2. When the libp2p service of client1 discovers the computer of client2, it opens a network connection (code) and reads from it using a traditional byte stream (code). (We give links to code for curious readers, but this isn't a code tutorial and you can skip the code links.) We've mentioned the traditional networking code “under the hood” to remove some of the mystery. But that code doesn't immediately read the message from client2. Instead, there are some communication steps which allow Wesh to use libp2p to operate in a peer-to-peer environment.

Message log and “head”

To send a message in traditional networking, one user connects to the network address of the other user and transmits the message. Or the user connects to a central server and appends to the message log there, while the other user connects to the same central server and reads the message log. But this doesn't work in a peer-to-peer environment. Instead, the goal is for each group member to have a copy of the message log, where there is a “head” (the latest message) which refers to all the previous messages in a chain. To send a message to the group, a member adds to the local copy of the message log and updates the “head”. This will be synchronized with the other members.

Without a central server, each group member may add to the message log and receive updates from others in a different order. But the synchronization algorithm ensures “eventual consistency”: after exchanging enough messages, eventually all the users will have the same message log in the same order. At that point they will all have the same “head” message, which refers to all the previous messages in a chain. Wesh currently uses OrbitDB for this, and you can read the details in their manual. (In the future, Wesh may use a simpler library but it will still have the same concept of a synchronized message log with a head.)

As mentioned, client2 doesn't send the message “Hello” directly to the other group members. Instead, it puts the message in its own message log and updates the head. The other members learn about the new head which means there is a new message to fetch.

libp2p pubsub

So now the question is, how does client1 learn that client2 says there is a new message log head? This uses libp2p pubsub where peers can subscribe to a topic and publish a pubsub message to that topic, or receive pubsub messages. (We say “pubsub message” to not confuse with the ordinary messages in the log which we'll discuss below.)

libp2p pubsub is a powerful tool with algorithms which allow it to scale to many peers and to make sure they all get the pubsub message. You can read more in its documentation, but for this blog post we only need to say that the Wesh group has a libp2p topic name like   /orbitdb/bafyreihpdptqwgpuwu4ob6nusxebmdx6fm6mq5pma5hva6eu5u74pflo7e/2ac4e88c5272f900c76fd36989f9780e5b2c95d75d38fe5bcef0345b34bc4806_wesh_group_messages. Client2 publishes a pubsub message to this topic that contains the message log head. The libp2p on client1 receives this pubsub message and sends it (code) internally to all processes which are subscribed to the topic. The Wesh application of client1 has used ActivateGroup and is therefore subscribed to this topic and receives the pubsub message (code).

IPFS to fetch the message

libp2p pubsub doesn't care about the meaning of a pubsub message. It only makes sure that all subscribers get it. Remember that the next step is to synchronize the message log, which is handled by the local message store of client1. After checking that it's not a repeat, the pubsub message is forwarded (code) as an EventPubSubMessage to the message store which receives it (code). This confirms that the type of pubsub message is to update the message log head, and it checks to make sure the head was not already processed. (If client1 has already received the same message log head, maybe from another peer, then it doesn't need to do more processing. This may seem trivial, but it's important because it's efficient to exchange these small “head” messages frequently between peers.)

The message store receives (code) the event for a new head which contains the CID of the message. The message (which may be large) is fetched with IPFS, not with libp2p pubsub (which is better for small messages). If you don't know what a CID is and how IPFS uses it to fetch content, then you can read all about it. Notice that using a CID to fetch the message (code) like “Hello” is a separate process from synchronizing heads. Maybe IPFS has already fetched the same CID, or maybe it can be fetched from another group member that is not client2. Wesh takes advantage of the efficient IPFS algorithms.

Decrypting and finishing

We're almost done! When the “Hello” message is fetched, it's placed in the local copy of the message log in the correct order. (Messages in the device's storage remain encrypted.) This internally sends the EventReplicated event which is received (code) by the Wesh message monitor. We started at the libp2p “level” which doesn't care about the meaning of pubsub messages, up through the message log level which only cares about the correct order of messages. Now at the “top” level, Wesh must use client1's cryptographic keys to decrypt (code) the message.

The decrypted message is sent internally (code) to all monitoring processes, including the GroupMessageList service (code). The client1 application has called GroupMessageList, so the “Hello” message is sent (code) to the application which prints the message.

Flight of a Byte

In summary, the flight of a message from client2 to client1 starts when the client2 peer is discovered, subscribes to the libp2p pubsub topic for the group messages and publishes its message log head. client1 doesn't have this head yet, so it uses the CID to fetch the message using IPFS. When the message arrives, it is added to client1's copy of the message log in the correct order. Wesh sends this new message to processes using the GroupMessageList service, such as the sample application which receives it and prints “Hello”. Easy, right? Each of these steps is needed to make sure communication can still happen even if a peer disconnects or reconnects at a different network address.

Now that you know what to expect when your Wesh application communicates, we'll return to some more examples in future blog posts.

Posted by Jeff on 20/07/2023