piermesh/docs/readme.md

PierMesh documentation

Contents:

• run: PierMesh service runner
  • Node
    • Node.toLog
    • Node.actions
    • Node.todo
    • Node.network
    • Node.catch
    • Node.cache
    • Node.nodeInfo
    • Node.onodeID
    • Node.oTransceiver
    • Node.processed
    • Node.proc
    • Node.mTasks
    • Node.action_initNodeDH()
    • Node.action_keyDeriveDH()
    • Node.action_map()
    • Node.action_sendCatch()
    • Node.action_sendToPeer()
    • Node.cLog()
    • Node.spongeListen()
• ui: TUI application
  • TUI
    • TUI.visibleLogo
    • TUI.nodeOb
    • TUI.done
    • TUI.CSS_PATH
    • TUI.action_quitFull()
    • TUI.action_toggleFullscreen()
    • TUI.compose()
    • TUI.do_set_cpu_percent()
    • TUI.do_set_mem()
    • TUI.do_write_line()
    • TUI.on_mount()
• Network: Network map representation
  • Network
    • Network.omap
    • Network.imap
    • Network.emaps
    • Network.lookup
    • Network.addLookup()
    • Network.addie()
    • Network.addin()
    • Network.addoe()
    • Network.addon()
    • Network.doLookup()
    • Network.export()
      • Network.path
    • Network.getRoute()
    • Network.mimport()
    • Network.render()
• hopper: Small internet interop utilities
  • get()
  • post()
• Daisy based cache
  • Cache
    • Cache.create()
    • Cache.get()
    • Cache.refresh()
    • Cache.search()
• Daisy Catch cache
  • Catch
    • Catch.get()
    • Catch.sget()
• Daisy
  • Daisy
    • Daisy.filepath
    • Daisy.msg
    • Daisy.get()
    • Daisy.read()
    • Daisy.sublist()
    • Daisy.write()
• Soil: Daisy signal management
  • Compound
    • Compound.on_any_event()
• Store: Daisy key value store
  • Store
    • Store.getRecord()
    • Store.update()
• WhaleSong: Diffie hellman ephemeral Fernet based encryption
  • DHEFern
    • DHEFern.cLog
    • DHEFern.loadedParams
    • DHEFern.loadedKeys
    • DHEFern.nodeNickname
    • DHEFern.cache
    • DHEFern.publicKey
    • DHEFern.privateKey
    • DHEFern.checkInMem()
    • DHEFern.decrypt()
    • DHEFern.encrypt()
    • DHEFern.genKeyPair()
    • DHEFern.genParams()
    • DHEFern.getParamsBytes()
    • DHEFern.getRecord()
    • DHEFern.getSalt()
    • DHEFern.initStore()
    • DHEFern.keyDerive()
    • DHEFern.loadParamBytes()
    • DHEFern.loadRecordToMem()
• base: Primary filtering functionality
  • Filter
    • Filter.cache
    • Filter.mCheck()
    • Filter.protoMap()
    • Filter.protoRoute()
    • Filter.selfCheck()
    • Filter.sieve()
• Protocols
  • bubble.filter()
  • catch.filter()
  • cryptography.filter()
  • map.filter()
• Header packet: Metadata packet
  • Header
    • Header.sender
    • Header.senderDisplayName
    • Header.recipient
    • Header.recipientNode
    • Header.subpacket
    • Header.wantFullResponse
    • Header.pAction
    • Header.dump()
    • Header.usePreset()
• Packet: Base packet
  • Packet
    • Packet.data
    • Packet.packetsID
    • Packet.packetNumber
    • Packet.packetCount
    • Packet.packetsClass
    • Packet.dump()
    • Packet.parsePayload()
• SinglePacket: Singular packet messages for very low data applications
  • SinglePacket
• SubMessage: Additional data for larger messages
  • SubMessage
• SubPacket: Packets for submessages
  • SubPacket
• Transceiver: Layer 0 data transceiving
  • Transceiver
    • Transceiver.cLog
    • Transceiver.cryptographyInfo
    • Transceiver.filter
    • Transceiver.tcache
    • Transceiver.tcatch
    • Transceiver.notConnected
    • Transceiver.acks
    • Transceiver.onodeID
    • Transceiver.messages
    • Transceiver.addPackets()
    • Transceiver.announce()
    • Transceiver.awaitFullResponse()
    • Transceiver.awaitResponse()
    • Transceiver.initNodeDH()
    • Transceiver.onConnection()
    • Transceiver.onReceive()
    • Transceiver.progressCheck()
    • Transceiver.responseCheck()
    • Transceiver.send()
    • Transceiver.sendAnnounce()
• serve: Web UI server
  • Server
    • Server.cLog
    • Server.transmitter
    • Server.network
    • Server.nodeID
    • Server.peerIDs
    • Server.app
    • Server.catch
    • Server.sendToPeer()

System Overview

PierMesh has two main events loops to learn about: the TUI and the service.

TUI

🔗 Docs

🔗 Source

The TUI is provided via Textual's library. It's a relatively simple application that gives us a quick overview of system statistics in the way of memory and cpu usage as well as scrollable logs. You can toggle full screen logs with f and close the TUI and service with q.

The Service

PierMesh runs a number of loops under the hood. These are primarily initialized in the main loop of run with a special logging loop outside of that.

Note that we make heavy use of Meshtastic's Python API.

run

🔗 run.py docs

🔗 run.py source

run.main

In run.main we (in order)

1. Initialize the Node class to a global nodeOb.
2. Wait for x seconds determined by command line arguments (see the falin and marcille scripts in scripts)
3. Initialize our Transceiver class (and the corresponding hardware)
4. Initialize our Server class
5. Create an async task running the Node.spongeListen method which looks for updates from the filter system
6. Kick the loop on with await asyncio.sleep(1) (we do this to kick on all async loops so I will omit this step going forward)
7. Create an async task running the Transceiver.checkProgress method which checks for packet reception progress and resends packets if necessary
8. Create an async task running Node.monitor which checks and reports system usage
9. Create an async task running the Transceiver.announce method which broadcasts a Packets.Message containing network mapping information
10. Last we start the Microdot server loop

..

Travelling out of the run.main thread to the primary main code we see the other two threads: one running the logPassLoop loop which is created in the same way as the main thread: lplThread = threading.Thread(target=asyncio.run, args=(logPassLoop(),)) and one running the TUI loop which has it's own system we just kick on by instantiating the TUI class and calling .run() on.

Packet lifecycle

Prior to this we've been talking relatively high level but if you've read this far you probably want more low level details. Let's talk about packets.

Packets.Packet

This is a base singular packet. At the very least a packet will contain: a packets ID which determines what set of packets (message) a singular packet belongs to if any, a packetNumber which determines which order the data will be put in, a packetCount which is, well a count of all the packets in a set of packets and in most cases data which is an lzma compressed msgpack encoded chunk of data to be fused together when the full set of packets is received.

Packets.HeaderPacket

This is the metadata packet for a set of packets (message) which handles the details necessary for routing and action triggering

Packets.Message

A set of packets instantiated primarily from lzma compressed msgpack encoded bytes

You keep saying msgpack what is that?

msgpack is what I landed on for bundling metadata and arbitrary bytes after initially using bson as msgpack is leaner, simpler and better adopted for cross platform data interchange

Transmitting

Once we have our msgpack encoded lzma compressed data we can send it with Transceiver.addPackets this method creates a Message and sends each Packet (dumped to binary data) over LoRa using Meshtastic's Python interface via the Transceiver.send method.

Receiving

On the other end when we receive a Packet we pass it directly into Sponge.base.sieve which checks: if it's msgpack encoded and if it's a packet sent by the same node. In both cases it skips the packet, otherwise we sieve!

Sieving

Primarily in this stage we are gathering packets and adding them to an object containing all of the packet info and data we've received for that Message.

Once a Message is completed we pass it on to the appropriate protocol from Sponge.Protocols. These will pass the Message wherever it needs to go from there which can be any number of places.

To avoid unnecessary manual documentation labor for future devs I will not list everything that could happen here but here are some examples:

• Updating a Catch
• Sending a message up to a peer on the node
• Sending a Diffie Hellman handshake

Fin

Well not really, there's more to learn but these are the basics. There's more in the docs above and you can always submit an issue or reach me at info@piermesh.net or on Tumblr at utopicwork.