Shreds
Shred is a data structure which can be referred to as the smallest unit of a block that are sent across the network in the form of UDP packets using Turbine Block Propagation. Shredding is the process done after creating Entries into the ledger. Shredding is done for two reasons, one being that sending an entire entry at once over UDP is not possible given the IPV4 MTU of 1500 bytes and two shreds are erasure coded so they also help with packet loss and malicious dropping of packets.
Each shred has a sub-type - coding shred and data shred. Data shreds con- tain the ledger entries. Coding shreds are required to provide redundancy to protect against the network dropping packets. Since there is a chance for UDP networks to drop packets as in contrast to TCP where dropped packets are re- transmitted, the need for having erasure coded shreds encoding the data shreds itself is critical due to packet loss. The coding shreds also prevent any malicious node from dropping packets while propagating the shreds.
Each Shred also has the following data layout:
- Common Header
- Data or Coding Header
- Payload
Common Header
The common header contains the signature of the leader which is a signed message originated from the merkle root of the shreds. This is used to validate that the shreds are received from the leader and to slash the leader if the proofs don’t compute to the root. It also contains the slot number, the shred version and the FEC index which helps with creating the coding shreds.
Data Shred
Data shreds are the individually transmitted units of the ledger entries. The payload here contains the path of the shred to the merkle root i.e the proof and the actual merkle root. This allows for doing equality checks without having to reconstruct the whole block. It is important to note that this a newly imple- mented method and the legacy method didn’t have merkle proofs in shreds.
Coding Shred
Coding shreds store the erasure-coded data. Solana uses Reed-Solomon erasure
coding, a polynomial encoding algorithm. Given n shreds and k data shreds we
get the number of coding shreds t = n - k. The coded shreds are generated
using a generator polynomial g(x) = (x-a)(x-a2)(x-a3). . . . . . (x-a2t)