Prototyping is exciting, building something that lasts is meaningful

Hey nerds and normies!

As I have been working on FileOrbit and its underlying protocol Substratum.cloud. I have faced countless challenges. Computers dying, UI hiccups, cloud hosting hurdles, Podman versus Docker fun times, and early missteps in the initial prototype.

When I started I set out to prove can I make my MacBook, and Windows PC exchange files using AT Proto as my source of truth for who owns the file, and allow the two to leverage libp2p to communicate to look for that record on the PDS (Personal Data Server) and request it from the private IPFS Swarm, while most importantly securing the libp2p network to the specific AT Proto Identity so others can't just request any file if they join the network.

Once that was proven I set out to add file sharing, where you share a file across AT Proto identities allowing the secure libp2p Network Behavior to look at not only who owns the content but also Access control Lists (ACLs). It all worked swimmingly I logged into one account on my MacBook, and another on my Windows gaming PC and boom I would share a file that otherwise would not be available to my user since they were not the owner, in a similar way to other drive storage where there is a "Shared with Me" page.

Fast forward months later after that walking skeleton was built, I started to figure out how I would like to leverage a users PDS to handle Peer discovery instead of Distributed Hash Tables (DHTs) since this is a private swarm and I do not want everyone's file content Ids (CIDs) advertised. Allowing me to not need a VPN like my prototype required to have static IPs.That led me to Hybrid Public Key Encryption to that I could exchange the private swarm key using AT Proto a trusted source of truth you are who you say you are. That enabled the application to start to feel like any other cloud drive storage but instead using your own identity you control, and the storage you have. The cloud is just another peer that leverages all the same bells and whistles.

I was on top of the world, my mom and friend Elliott Quinn started testing it out and we ran into some silly issues with dev-dependencies that made it work on my machine (classic developer move ;P). I decoupled the app from a package managed PostGres and built it from source to make the installer standalone, polished Windows 10 versus 11 issues with credential storage, and voila! I had a working app, my family and friends could now upload to their local place and it would sync with the Cloud Safety-Net storage in realtime! I was so proud.

Of course then came sharing...Elliott went to share a file with me, and boom he noticed it only showed up in the Cloud/Web Version where he shared it. Then it hit me...I built that on a closed network where the two machines were talking and frankly I did not consider the mature applications relay/proxying of the same information which utilize the Gossip Protocol beyond its direct connected peers. In the app I call these messages Whisper events, they should be light and only as hints for the main app to consult the AT Proto users' PDS records since that's the authenticated source of truth for any given user. One other issue is cross account sharing is a matter of both users being logged-in on a common peer so that the application can fetch from the sharers PDS Passport record and check the ACL to ensure they were actually added and not removed shortly after. Again the PDS is the authority, the places/peers are just an AppView/Cache of the aggregated information and has access to the Private IPFS swarm. This is where prototype meets maintainable.

I launched the app in Alpha a month ago, yet with Sharing and Invites with a glaring gap I had to act fast. First I reset expectations, making a new lexicon for experimental features. Configuring Sharing and Invites behind their own feature flags and accessible in a Advanced Settings section on the Account page. This stopped the bleeding, but I needed to make sure when this is ready to test again I am confident in what I am releasing, I don't want to waste my friends and families time.

This brought me back to my testing strategy. Over the course of the last few weeks I have doubled down on my tests. Unit tests prove that the pieces work as expected, integration proves that the subsystems between the database, swarm, and web services work, yet my E2E tests were lacking. They only tested on one node, so of course my sharing and invite mechanisms looked like they worked they shared persistence. The trouble with a P2P system the persistence is a cache not a source of truth, say it with me folks! The PDS is the source of truth! 🗣️ So I needed tests in multiple areas first I dabbled in TestGround IPFS mesh tests. Once I felt confident in the relaying aspect of the Gossip Pub/Sub I moved onto multi node tests with a PDS in the middle to simulate the real world conditions of this distributed P2P system.

I am really satisfied with the testing coverage and results and optimistic to update the File Orbit Roadmap with the Invites and Sharing revamp supporting P2P relaying and tests with video and logging proof everyone can see via the new Allure Reports hosted on the docs site in the next week or so.

Huge thank you and shout out to Elliott; and my Merm!

Until next time peace nerds and normies,

~ Triforce