• Formats for Encoding Data:
  
  • Overview: Programs usually work with data in (at least) two different representations:
    
    • In memory, data is kept in objects, structs, lists, arrays, hash tables, trees, and so on. These data structures are optimized for efficient access and manipulation by the CPU (typically using pointers).
    • When you want to write data to a file or send it over the network, you have to encode it as some kind of self-contained sequence of bytes (for example, a JSON document). Since a pointer wouldn’t make sense to any other process, this sequence-of-bytes representation looks quite different from the data structures that are normally used in memory.
    • Thus, we need some kind of translation between the two representations. The translation from the in-memory representation to a byte sequence is called encoding (also known as serialization or marshalling), and the reverse is called decoding (parsing, deserialization, unmarshalling).
      
  • Language-Specific Formats
    
    • Java - java.io.Serializable
    • Ruby - Marshal
    • Python - pickle
    • etc.
      
  • JSON, XML, and Binary Variants
  • Thrift and Protocol Buffers
    
    • Field tags and schema evolution
    • Datatypes and schema evolution
      
  • Avro
    
    • The writer’s schema and the reader’s schema
    • Schema evolution rules
    • Dynamically generated schemas
    • Code generation and dynamically typed languages
      
  • The Merits of Schemas
    
• Modes of Dataflow
  
  • Dataflow Through Databases
  • Dataflow Through Services: REST and RPC
  • Message-Passing Dataflow