From 161944eab7f79d724531ad0ea7176933a38ae88f Mon Sep 17 00:00:00 2001 From: Agie Ashwood Date: Fri, 2 Aug 2024 12:03:52 +0000 Subject: [PATCH] Docs: Mandocs --- builddocs | 2 + docs/readme.md | 85 ++++++++++++++++++++++++++++++++++ readme.md | 8 ++++ src/Cryptography/WhaleSong.py | 1 + src/Packets/Message.py | 6 ++- src/Transceiver/Transceiver.py | 1 + src/getcloc | 2 + src/readme_append.md | 83 +++++++++++++++++++++++++++++++++ 8 files changed, 187 insertions(+), 1 deletion(-) create mode 100755 src/getcloc create mode 100644 src/readme_append.md diff --git a/builddocs b/builddocs index 5e064098..a5b44fdb 100755 --- a/builddocs +++ b/builddocs @@ -7,3 +7,5 @@ rm -rf docs.tmp mv docs/index.md docs/readme.md sed -i '1s;^;![PierMesh logo](https://git.utopic.work/PierMesh/piermesh/raw/branch/main/piermeshicon.png)\n\n;' docs/readme.md sed -i '1s;^;![Daisy logo](https://git.utopic.work/PierMesh/piermesh/raw/branch/main/imgs/daisydisplay.png)\n\n;' docs/Daisy/Daisy.md +echo $'\n' >> docs/readme.md +cat src/readme_append.md >> docs/readme.md diff --git a/docs/readme.md b/docs/readme.md index 8eb2afb8..ada8c20e 100644 --- a/docs/readme.md +++ b/docs/readme.md @@ -180,3 +180,88 @@ sphinx-quickstart on Fri Jul 26 23:30:55 2024. --> * [`Server.app`](/PierMesh/piermesh/src/branch/main/docs/Splash/serve.md#Splash.serve.Server.app) * [`Server.catch`](/PierMesh/piermesh/src/branch/main/docs/Splash/serve.md#Splash.serve.Server.catch) * [`Server.sendToPeer()`](/PierMesh/piermesh/src/branch/main/docs/Splash/serve.md#Splash.serve.Server.sendToPeer) + + +# System Overview + +PierMesh has two main events loops to learn about: the TUI and the service. + +## TUI + +[🔗 Docs](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs/ui.md) + +[🔗 Source](https://git.utopic.work/PierMesh/piermesh/src/branch/main/src/ui.py) + +The TUI is provided via [Textual's](https://textual.textualize.io/) library. It's a relatively simple application that gives us a quick overview of system statistics in the way of memory and cpu usage as well as scrollable logs. You can toggle full screen logs with f and close the TUI and service with q. + +## The Service + +PierMesh runs a number of loops under the hood. These are primarily initialized in the main loop of run with a special logging loop outside of that. + +Note that we make heavy use of [Meshtastic's Python API](https://github.com/meshtastic/python). + +### run + +[🔗 run.py docs](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs/run.md) + +[🔗 run.py source](https://git.utopic.work/PierMesh/piermesh/src/branch/main/src/run.py) + +#### run.main + +In run.main we (in order) +1. Initialize the `Node` class to a global `nodeOb`. +2. Wait for x seconds determined by command line arguments (see the falin and marcille scripts in scripts) +3. Initialize our `Transceiver` class (and the corresponding hardware) +4. Initialize our `Server` class +5. Create an async task running the `Node.spongeListen` method which looks for updates from the filter system +6. Kick the loop on with `await asyncio.sleep(1)` (we do this to kick on all async loops so I will omit this step going forward) +7. Create an async task running the `Transceiver.checkProgress` method which checks for packet reception progress and resends packets if necessary +8. Create an async task running `Node.monitor` which checks and reports system usage +9. Create an async task running the `Transceiver.announce` method which broadcasts a `Packets.Message` containing network mapping information +10. Last we start the `Microdot` server loop + +### .. +Travelling out of the run.main thread to the primary ____main____ code we see the other two threads: one running the `logPassLoop` loop which is created in the same way as the main thread: `lplThread = threading.Thread(target=asyncio.run, args=(logPassLoop(),))` and one running the TUI loop which has it's own system we just kick on by instantiating the TUI class and calling .run() on. + +## Packet lifecycle + +Prior to this we've been talking relatively high level but if you've read this far you probably want more low level details. Let's talk about packets. + +### Packets.Packet + +This is a base singular packet. At the very least a packet will contain: a packets ID which determines what set of packets (message) a singular packet belongs to if any, a packetNumber which determines which order the data will be put in, a packetCount which is, well a count of all the packets in a set of packets and in most cases data which is an lzma compressed msgpack encoded chunk of data to be fused together when the full set of packets is received. + +### Packets.HeaderPacket + +This is the metadata packet for a set of packets (message) which handles the details necessary for routing and action triggering + +### Packets.Message + +A set of packets instantiated primarily from lzma compressed msgpack encoded bytes + +### You keep saying msgpack what is that? + +[msgpack](https://msgpack.org/) is what I landed on for bundling metadata and arbitrary bytes after initially using bson as msgpack is leaner, simpler and better adopted for cross platform data interchange + +### Transmitting + +Once we have our msgpack encoded lzma compressed data we can send it with `Transceiver.addPackets` this method creates a `Message` and sends each `Packet` (dumped to binary data) over LoRa using Meshtastic's Python interface via the `Transceiver.send` method. + +### Receiving + +On the other end when we receive a `Packet` we pass it directly into `Sponge.base.sieve` which checks: if it's msgpack encoded and if it's a packet sent by the same node. In both cases it skips the packet, otherwise we sieve! + +### Sieving + +Primarily in this stage we are gathering packets and adding them to an object containing all of the packet info and data we've received for that `Message`. + +Once a `Message` is completed we pass it on to the appropriate protocol from `Sponge.Protocols`. These will pass the `Message` wherever it needs to go from there which can be any number of places. + +To avoid unnecessary manual documentation labor for future devs I will not list everything that could happen here but here are some examples: +- Updating a Catch +- Sending a message up to a peer on the node +- Sending a Diffie Hellman handshake + +# Fin + +Well not really, there's more to learn but these are the basics. There's more in the docs above and you can always submit an issue or reach me at info@piermesh.net or on Tumblr at utopicwork. \ No newline at end of file diff --git a/readme.md b/readme.md index 10c7342c..4a4c9ee2 100644 --- a/readme.md +++ b/readme.md @@ -6,7 +6,15 @@ This is the monorepo for PierMesh. # Docs +<<<<<<< HEAD +<<<<<<< HEAD You can find the full docs here: [docs/](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs) +======= +You can find the full docs here: [docs/](docs/) +>>>>>>> 3ad19a9 (Docs: Clean up primary readme) +======= +You can find the full docs here: [docs/](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs) +>>>>>>> 6f5bfe4 (Docs link fix) # How to use diff --git a/src/Cryptography/WhaleSong.py b/src/Cryptography/WhaleSong.py index a7dcbeeb..2c144284 100755 --- a/src/Cryptography/WhaleSong.py +++ b/src/Cryptography/WhaleSong.py @@ -257,6 +257,7 @@ class DHEFern: """ return os.urandom(16) + # TODO: Build in transport security (node/node) def encrypt(self, data, nodeID: str, isDict: bool = True): """ Do Fernet encryption diff --git a/src/Packets/Message.py b/src/Packets/Message.py index de82aef3..1651dc1d 100755 --- a/src/Packets/Message.py +++ b/src/Packets/Message.py @@ -27,6 +27,7 @@ class Message: senderDisplayName: int, recipient: int, recipientNode: int, + cryptographyInfo, dataSize: int = 128, wantFullResponse: bool = False, packetsClass: int = 0, @@ -71,7 +72,10 @@ class Message: ) self.packets = packets else: - bytesObject = lzma.compress(bytesObject) + # Data passed in by peers should already have been e2ee encrypted by SubtleCrypto + # Transport encryption + # bytesObject = lzma.compress(bytesObject, str(recipientNode).zfill(6), isDict=False) + bytesObject = cryptographyInfo.encrypt(bytesObject, self.no) packets = [] self.packetsID = random.randrange(0, 999999) pnum = 1 diff --git a/src/Transceiver/Transceiver.py b/src/Transceiver/Transceiver.py index 4cefa5cc..5a943496 100644 --- a/src/Transceiver/Transceiver.py +++ b/src/Transceiver/Transceiver.py @@ -227,6 +227,7 @@ class Transceiver: senderName, recipient, recipientNode, + self.cryptographyInfo, packetsClass=packetsClass, ) for p in tp.packets: diff --git a/src/getcloc b/src/getcloc new file mode 100755 index 00000000..e93b548b --- /dev/null +++ b/src/getcloc @@ -0,0 +1,2 @@ +cloc --exclude-dir=Splash,stale . +cloc Splash/serve.py diff --git a/src/readme_append.md b/src/readme_append.md new file mode 100644 index 00000000..20b982f1 --- /dev/null +++ b/src/readme_append.md @@ -0,0 +1,83 @@ +# System Overview + +PierMesh has two main events loops to learn about: the TUI and the service. + +## TUI + +[🔗 Docs](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs/ui.md) + +[🔗 Source](https://git.utopic.work/PierMesh/piermesh/src/branch/main/src/ui.py) + +The TUI is provided via [Textual's](https://textual.textualize.io/) library. It's a relatively simple application that gives us a quick overview of system statistics in the way of memory and cpu usage as well as scrollable logs. You can toggle full screen logs with f and close the TUI and service with q. + +## The Service + +PierMesh runs a number of loops under the hood. These are primarily initialized in the main loop of run with a special logging loop outside of that. + +Note that we make heavy use of [Meshtastic's Python API](https://github.com/meshtastic/python). + +### run + +[🔗 run.py docs](https://git.utopic.work/PierMesh/piermesh/src/branch/main/docs/run.md) + +[🔗 run.py source](https://git.utopic.work/PierMesh/piermesh/src/branch/main/src/run.py) + +#### run.main + +In run.main we (in order) +1. Initialize the `Node` class to a global `nodeOb`. +2. Wait for x seconds determined by command line arguments (see the falin and marcille scripts in scripts) +3. Initialize our `Transceiver` class (and the corresponding hardware) +4. Initialize our `Server` class +5. Create an async task running the `Node.spongeListen` method which looks for updates from the filter system +6. Kick the loop on with `await asyncio.sleep(1)` (we do this to kick on all async loops so I will omit this step going forward) +7. Create an async task running the `Transceiver.checkProgress` method which checks for packet reception progress and resends packets if necessary +8. Create an async task running `Node.monitor` which checks and reports system usage +9. Create an async task running the `Transceiver.announce` method which broadcasts a `Packets.Message` containing network mapping information +10. Last we start the `Microdot` server loop + +### .. +Travelling out of the run.main thread to the primary ____main____ code we see the other two threads: one running the `logPassLoop` loop which is created in the same way as the main thread: `lplThread = threading.Thread(target=asyncio.run, args=(logPassLoop(),))` and one running the TUI loop which has it's own system we just kick on by instantiating the TUI class and calling .run() on. + +## Packet lifecycle + +Prior to this we've been talking relatively high level but if you've read this far you probably want more low level details. Let's talk about packets. + +### Packets.Packet + +This is a base singular packet. At the very least a packet will contain: a packets ID which determines what set of packets (message) a singular packet belongs to if any, a packetNumber which determines which order the data will be put in, a packetCount which is, well a count of all the packets in a set of packets and in most cases data which is an lzma compressed msgpack encoded chunk of data to be fused together when the full set of packets is received. + +### Packets.HeaderPacket + +This is the metadata packet for a set of packets (message) which handles the details necessary for routing and action triggering + +### Packets.Message + +A set of packets instantiated primarily from lzma compressed msgpack encoded bytes + +### You keep saying msgpack what is that? + +[msgpack](https://msgpack.org/) is what I landed on for bundling metadata and arbitrary bytes after initially using bson as msgpack is leaner, simpler and better adopted for cross platform data interchange + +### Transmitting + +Once we have our msgpack encoded lzma compressed data we can send it with `Transceiver.addPackets` this method creates a `Message` and sends each `Packet` (dumped to binary data) over LoRa using Meshtastic's Python interface via the `Transceiver.send` method. + +### Receiving + +On the other end when we receive a `Packet` we pass it directly into `Sponge.base.sieve` which checks: if it's msgpack encoded and if it's a packet sent by the same node. In both cases it skips the packet, otherwise we sieve! + +### Sieving + +Primarily in this stage we are gathering packets and adding them to an object containing all of the packet info and data we've received for that `Message`. + +Once a `Message` is completed we pass it on to the appropriate protocol from `Sponge.Protocols`. These will pass the `Message` wherever it needs to go from there which can be any number of places. + +To avoid unnecessary manual documentation labor for future devs I will not list everything that could happen here but here are some examples: +- Updating a Catch +- Sending a message up to a peer on the node +- Sending a Diffie Hellman handshake + +# Fin + +Well not really, there's more to learn but these are the basics. There's more in the docs above and you can always submit an issue or reach me at info@piermesh.net or on Tumblr at utopicwork. \ No newline at end of file