Docs: Mandocs

builddocs

@@ -7,3 +7,5 @@ rm -rf docs.tmp
 mv docs/index.md docs/readme.md
 sed -i '1s;^;![PierMesh logo](https://git.utopic.work/PierMesh/piermesh/raw/branch/main/piermeshicon.png)\n\n;' docs/readme.md
 sed -i '1s;^;![Daisy logo](https://git.utopic.work/PierMesh/piermesh/raw/branch/main/imgs/daisydisplay.png)\n\n;' docs/Daisy/Daisy.md
+echo $'\n' >> docs/readme.md
+cat src/readme_append.md >> docs/readme.md

docs/readme.md

@@ -180,3 +180,88 @@ sphinx-quickstart on Fri Jul 26 23:30:55 2024. -->
     * [`Server.app`](/PierMesh/piermesh/src/branch/main/docs/Splash/serve.md#Splash.serve.Server.app)
     * [`Server.catch`](/PierMesh/piermesh/src/branch/main/docs/Splash/serve.md#Splash.serve.Server.catch)
     * [`Server.sendToPeer()`](/PierMesh/piermesh/src/branch/main/docs/Splash/serve.md#Splash.serve.Server.sendToPeer)
+
+
+# System Overview
+
+PierMesh has two main events loops to learn about: the TUI and the service.
+
+## TUI
+
+[🔗 Docs](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs/ui.md)
+
+[🔗 Source](https://git.utopic.work/PierMesh/piermesh/src/branch/main/src/ui.py)
+
+The TUI is provided via [Textual's](https://textual.textualize.io/) library. It's a relatively simple application that gives us a quick overview of system statistics in the way of memory and cpu usage as well as scrollable logs. You can toggle full screen logs with f and close the TUI and service with q.
+
+## The Service
+
+PierMesh runs a number of loops under the hood. These are primarily initialized in the main loop of run with a special logging loop outside of that.
+
+Note that we make heavy use of [Meshtastic's Python API](https://github.com/meshtastic/python).
+
+### run
+
+[🔗 run.py docs](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs/run.md)
+
+[🔗 run.py source](https://git.utopic.work/PierMesh/piermesh/src/branch/main/src/run.py)
+
+#### run.main
+
+In run.main we (in order)
+1. Initialize the `Node` class to a global `nodeOb`.
+2. Wait for x seconds determined by command line arguments (see the falin and marcille scripts in scripts)
+3. Initialize our `Transceiver` class (and the corresponding hardware)
+4. Initialize our `Server` class
+5. Create an async task running the `Node.spongeListen` method which looks for updates from the filter system
+6. Kick the loop on with `await asyncio.sleep(1)` (we do this to kick on all async loops so I will omit this step going forward)
+7. Create an async task running the `Transceiver.checkProgress` method which checks for packet reception progress and resends packets if necessary
+8. Create an async task running `Node.monitor` which checks and reports system usage
+9. Create an async task running the `Transceiver.announce` method which broadcasts a `Packets.Message` containing network mapping information
+10. Last we start the `Microdot` server loop
+
+### ..
+Travelling out of the run.main thread to the primary ____main____ code we see the other two threads: one running the `logPassLoop` loop which is created in the same way as the main thread: `lplThread = threading.Thread(target=asyncio.run, args=(logPassLoop(),))` and one running the TUI loop which has it's own system we just kick on by instantiating the TUI class and calling .run() on.
+
+## Packet lifecycle
+
+Prior to this we've been talking relatively high level but if you've read this far you probably want more low level details. Let's talk about packets.
+
+### Packets.Packet
+
+This is a base singular packet. At the very least a packet will contain: a packets ID which determines what set of packets (message) a singular packet belongs to if any, a packetNumber which determines which order the data will be put in, a packetCount which is, well a count of all the packets in a set of packets and in most cases data which is an lzma compressed msgpack encoded chunk of data to be fused together when the full set of packets is received.
+
+### Packets.HeaderPacket
+
+This is the metadata packet for a set of packets (message) which handles the details necessary for routing and action triggering
+
+### Packets.Message
+
+A set of packets instantiated primarily from lzma compressed msgpack encoded bytes
+
+### You keep saying msgpack what is that?
+
+[msgpack](https://msgpack.org/) is what I landed on for bundling metadata and arbitrary bytes after initially using bson as msgpack is leaner, simpler and better adopted for cross platform data interchange
+
+### Transmitting
+
+Once we have our msgpack encoded lzma compressed data we can send it with `Transceiver.addPackets` this method creates a `Message` and sends each `Packet` (dumped to binary data) over LoRa using Meshtastic's Python interface via the `Transceiver.send` method.
+
+### Receiving
+
+On the other end when we receive a `Packet` we pass it directly into `Sponge.base.sieve` which checks: if it's msgpack encoded and if it's a packet sent by the same node. In both cases it skips the packet, otherwise we sieve!
+
+### Sieving
+
+Primarily in this stage we are gathering packets and adding them to an object containing all of the packet info and data we've received for that `Message`. 
+
+Once a `Message` is completed we pass it on to the appropriate protocol from `Sponge.Protocols`. These will pass the `Message` wherever it needs to go from there which can be any number of places. 
+
+To avoid unnecessary manual documentation labor for future devs I will not list everything that could happen here but here are some examples:
+- Updating a Catch
+- Sending a message up to a peer on the node
+- Sending a Diffie Hellman handshake
+
+# Fin
+
+Well not really, there's more to learn but these are the basics. There's more in the docs above and you can always submit an issue or reach me at info@piermesh.net or on Tumblr at utopicwork.

readme.md

@@ -6,7 +6,15 @@ This is the monorepo for PierMesh.
 
 # Docs
 
+<<<<<<< HEAD
+<<<<<<< HEAD
 You can find the full docs here: [docs/](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs)
+=======
+You can find the full docs here: [docs/](docs/)
+>>>>>>> 3ad19a9 (Docs: Clean up primary readme)
+=======
+You can find the full docs here: [docs/](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs)
+>>>>>>> 6f5bfe4 (Docs link fix)
 
 # How to use
 

src/Cryptography/WhaleSong.py

@@ -257,6 +257,7 @@ class DHEFern:
         """
         return os.urandom(16)
 
+    # TODO: Build in transport security (node/node)
     def encrypt(self, data, nodeID: str, isDict: bool = True):
         """
         Do Fernet encryption

src/Packets/Message.py

@@ -27,6 +27,7 @@ class Message:
         senderDisplayName: int,
         recipient: int,
         recipientNode: int,
+        cryptographyInfo,
         dataSize: int = 128,
         wantFullResponse: bool = False,
         packetsClass: int = 0,
@@ -71,7 +72,10 @@ class Message:
                 )
             self.packets = packets
         else:
-            bytesObject = lzma.compress(bytesObject)
+            # Data passed in by peers should already have been e2ee encrypted by SubtleCrypto
+            # Transport encryption
+            # bytesObject = lzma.compress(bytesObject, str(recipientNode).zfill(6), isDict=False)
+            bytesObject = cryptographyInfo.encrypt(bytesObject, self.no)
             packets = []
             self.packetsID = random.randrange(0, 999999)
             pnum = 1

src/Transceiver/Transceiver.py

@@ -227,6 +227,7 @@ class Transceiver:
             senderName,
             recipient,
             recipientNode,
+            self.cryptographyInfo,
             packetsClass=packetsClass,
         )
         for p in tp.packets:

src/getcloc

@@ -0,0 +1,2 @@
+cloc --exclude-dir=Splash,stale .
+cloc Splash/serve.py

src/readme_append.md

@@ -0,0 +1,83 @@
+# System Overview
+
+PierMesh has two main events loops to learn about: the TUI and the service.
+
+## TUI
+
+[🔗 Docs](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs/ui.md)
+
+[🔗 Source](https://git.utopic.work/PierMesh/piermesh/src/branch/main/src/ui.py)
+
+The TUI is provided via [Textual's](https://textual.textualize.io/) library. It's a relatively simple application that gives us a quick overview of system statistics in the way of memory and cpu usage as well as scrollable logs. You can toggle full screen logs with f and close the TUI and service with q.
+
+## The Service
+
+PierMesh runs a number of loops under the hood. These are primarily initialized in the main loop of run with a special logging loop outside of that.
+
+Note that we make heavy use of [Meshtastic's Python API](https://github.com/meshtastic/python).
+
+### run
+
+[🔗 run.py docs](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs/run.md)
+
+[🔗 run.py source](https://git.utopic.work/PierMesh/piermesh/src/branch/main/src/run.py)
+
+#### run.main
+
+In run.main we (in order)
+1. Initialize the `Node` class to a global `nodeOb`.
+2. Wait for x seconds determined by command line arguments (see the falin and marcille scripts in scripts)
+3. Initialize our `Transceiver` class (and the corresponding hardware)
+4. Initialize our `Server` class
+5. Create an async task running the `Node.spongeListen` method which looks for updates from the filter system
+6. Kick the loop on with `await asyncio.sleep(1)` (we do this to kick on all async loops so I will omit this step going forward)
+7. Create an async task running the `Transceiver.checkProgress` method which checks for packet reception progress and resends packets if necessary
+8. Create an async task running `Node.monitor` which checks and reports system usage
+9. Create an async task running the `Transceiver.announce` method which broadcasts a `Packets.Message` containing network mapping information
+10. Last we start the `Microdot` server loop
+
+### ..
+Travelling out of the run.main thread to the primary ____main____ code we see the other two threads: one running the `logPassLoop` loop which is created in the same way as the main thread: `lplThread = threading.Thread(target=asyncio.run, args=(logPassLoop(),))` and one running the TUI loop which has it's own system we just kick on by instantiating the TUI class and calling .run() on.
+
+## Packet lifecycle
+
+Prior to this we've been talking relatively high level but if you've read this far you probably want more low level details. Let's talk about packets.
+
+### Packets.Packet
+
+This is a base singular packet. At the very least a packet will contain: a packets ID which determines what set of packets (message) a singular packet belongs to if any, a packetNumber which determines which order the data will be put in, a packetCount which is, well a count of all the packets in a set of packets and in most cases data which is an lzma compressed msgpack encoded chunk of data to be fused together when the full set of packets is received.
+
+### Packets.HeaderPacket
+
+This is the metadata packet for a set of packets (message) which handles the details necessary for routing and action triggering
+
+### Packets.Message
+
+A set of packets instantiated primarily from lzma compressed msgpack encoded bytes
+
+### You keep saying msgpack what is that?
+
+[msgpack](https://msgpack.org/) is what I landed on for bundling metadata and arbitrary bytes after initially using bson as msgpack is leaner, simpler and better adopted for cross platform data interchange
+
+### Transmitting
+
+Once we have our msgpack encoded lzma compressed data we can send it with `Transceiver.addPackets` this method creates a `Message` and sends each `Packet` (dumped to binary data) over LoRa using Meshtastic's Python interface via the `Transceiver.send` method.
+
+### Receiving
+
+On the other end when we receive a `Packet` we pass it directly into `Sponge.base.sieve` which checks: if it's msgpack encoded and if it's a packet sent by the same node. In both cases it skips the packet, otherwise we sieve!
+
+### Sieving
+
+Primarily in this stage we are gathering packets and adding them to an object containing all of the packet info and data we've received for that `Message`. 
+
+Once a `Message` is completed we pass it on to the appropriate protocol from `Sponge.Protocols`. These will pass the `Message` wherever it needs to go from there which can be any number of places. 
+
+To avoid unnecessary manual documentation labor for future devs I will not list everything that could happen here but here are some examples:
+- Updating a Catch
+- Sending a message up to a peer on the node
+- Sending a Diffie Hellman handshake
+
+# Fin
+
+Well not really, there's more to learn but these are the basics. There's more in the docs above and you can always submit an issue or reach me at info@piermesh.net or on Tumblr at utopicwork.