Multi-media message save format and parsing

I have yet to have a phone that can save multi-media messages, so I took it upon myself to manually save them.

I use a very simple text based key-value format to save them. The keys are free text, and the values may contain arbitrary text, including multiple lines.

small
 value
multi-line
 line 1
 line 2
 line 3

Using this format I just choose some field names for the parts of the message, and separated messages with a field name of "done". Field names may be freely repeated.

type
 incoming
subject
 A Picture/Video Message!
date
 2012-11-27 23:38
from
 +15555555555
 Sender Name
to
 +15656565656
 My Name
text
 the text of the message goes here...
image
 imagejpeg_3.jpg
 5e35003a0e65df1065982c07587cf147ccb76b5d.jpg
done

the first line of a from or to is the number, the second line is the name of the contact. the first line of an image is the file name in the message, the second is the file name of the image in the directory. The other fields names are type, subject, date, text, and slide.

I broke the parsing of the records into usable data structures in Google Apps Script in two phases, the first one extracted the blocks of fields, and the other turned the fields into a data structure.

function forEachRecord(blob, body) {
  var fields = new Array();
  var field = { name: "", value: new Array() };
  
  var lines = blob.getDataAsString().split(/\r?\n/, -1);
  for(var lineIndex = 0; lineIndex < lines.length; lineIndex++) {
    var line = lines[lineIndex];

    // indented lines are the field value
    if(line.substr(0, 1) == " ") {
      field.value.push(line.substr(1));
      continue;
    }
    
    // blank lines are completely ignored
    if(line == "") {
      continue;
    }
    
    // starting a new field
    field = { name: line, value: new Array() };
    
    // records end when a field name of "done" is found
    if(line == "done") {
      body(fields);
      fields = new Array();
      continue;
    }

    // this field is part of the record
    fields.push(field);
  }
  
  if(fields.length) {
    body(fields);
  }
}

function forEachMessage(blob, body) {
  forEachRecord(blob, function(fields) {
    var m = {
      type: "",
      subject: "",
      date: "",
      from: new Array(),
      to: new Array(),
      body: new Array(),
    };
    
    for(var fieldIndex = 0; fieldIndex < fields.length; fieldIndex++) {
      var f = fields[fieldIndex];
      
      if(f.name == "type") {
        m.type = f.value[0];
        continue;
      }
      
      if(f.name == "to") {
        m.to.push({ address: f.value[0], name: f.value[1] });
        continue;
      }
      
      if(f.name == "from") {
        m.from.push({ address: f.value[0], name: f.value[1] });
        continue;
      }
      
      if(f.name == "date") {
        m.date = f.value[0];
        continue;
      }
      
      if(f.name == "image") {
        m.body.push({ type: "image", name: f.value[0], file: f.value[1] });
        continue;
      }
      
      if(f.name == "text") {
        m.body.push({ type: "text", value: f.value.join("\n") });
        continue;
      }
      
      if(f.name == "slide") {
        m.body.push({ type: "slide" });
        continue;
      }
      
      // if we get here we found an unknown field
    }
    
    body(m);
  });
}

This looks similar to a loop in the function that uses it.

forEachMessage(file.getBlob(), function(message) {
  messages.push(message);
});

Turning the parsed messages into an easily human readable output is not difficult from here, and that was my overall objective.

LG Neon message parsing

Sometimes I enjoy looking back over very old text messages from my phone. My current phone is an old LG Neon. It can save text messages to a text file on the SD card so I don't loose them when the text message memory on the phone fills up, but can not save the multimedia messages. I desire to keep all of these and view them later. I also decided that I would like to keep the data on Google Drive, and to get some experience with Google Apps Script. This all leads to taking the save files from my phone and making them useful with a Google Apps Script.

The LG Neon saves text messages as UTF-16LE text, however the leading character of the text indicates that they are saved as UTF-16BE. The only character set that is documented to be supported in apps script is UTF-8, so I did the UTF-16 decoding manually in the script.

function decodeLgText(bytes) {
  // The LG Neon saves text messages in UTF-16LE, with the header bytes for UTF-16BE
  
  var str = "";
  for (var i = 2; i < bytes.length; i += 2) {
    var charcode = bytes[i] & 0xff | ((bytes[i+1] & 0xff) << 8);
    if (charcode < 0xd800 || charcode >= 0xe000) {
      str += String.fromCharCode(charcode);
    } else {
      i += 2;
      var charcode1 = bytes[i] & 0xff | ((bytes[i + 1] & 0xff) << 8);
      charcode = 0x10000 + ( ((charcode & 0x3ff) << 10) | (charcode1 & 0x3ff) );
      str += String.fromCharCode(charcode);
    }
  }
  return str;
}

Two blocks of text from the export file looks like this:

1) From : +1555555555(Sample Name)
   Sent : 2013/04/11 17:58
   Contents :
   See you

227) To : +15555555555(Dear Friend)
   Sent : 2013/04/03 20:37
   Contents :
   Back in my very warm fun fur hammock tonight, ther
   e was some pretty snow today, and i get to sleep i
   n a bit tomorrow :)
Sweet dreams! <3

The contents section of the records is interesting, the raw text message is broken up with "\r\n" line breaks, and if there was a line break in the text message it only has a "\n". The record ends with a double "\r\n" line break. The solution to parsing this that I finally settled on is two parts, the first decodes each record into an array of lines, with the indentation and message number stripped off, and the second turns the array of lines into a useful data structure.

function forEachLgBlock(blob, body) {
  var blocks = decodeLgText(blob.getBytes());
  blocks = blocks.split("\r\n\r\n");
  
  for(var blockIndex = 0; blockIndex < blocks.length; blockIndex++) {
    var block = blocks[blockIndex];
    if(block == "") {
      continue;
    }
    
    var lines = block.split("\r\n", -1);
    lines[0] = lines[0].replace(/^\d+\)/, "  ");
    for(var lineIndex = 0; lineIndex < lines.length; lineIndex++) {
      lines[lineIndex] = lines[lineIndex].replace(/^   /, "");
    }
    
    body(lines);
  }
}

function forEachLgSms(blob, body) {
  forEachLgBlock(blob, function(lines) {
    var m = {
      type: "",
      subject: "",
      date: "",
      from: new Array(),
      to: new Array(),
      body: [ { type: "text", value: "" } ]
    };
    
    var lineIndex;
    for(lineIndex = 0; lineIndex < lines.length; lineIndex++) {
      var line = lines[lineIndex];
      var match;

      match = line.match(/^To : (.*?)\((.*?)\)$/);
      if(match) {
        m.to.push([ { address: match[1], name: match[2] } ]);
        m.type = "outgoing";
        continue;
      }
      
      match = line.match(/^To : (.*)$/);
      if(match) {
        m.to.push([ { address: match[1] } ]);
        m.type = "outgoing";
        continue;
      }
      
      match = line.match(/^From : (.*?)\((.*?)\)$/);
      if(match) {
        m.from.push([ { address: match[1], name: match[2] } ]);
        m.type = "incoming";
        continue;
      }
      
      match = line.match(/^From : (.*)$/);
      if(match) {
        m.from.push([ { address: match[1] } ]);
        m.type = "incoming";
        continue;
      }
      
      match = line.match(/^Sent : (\d\d\d\d)\/(\d\d)\/(\d\d) (\d\d:\d\d)$/);
      if(match) {
        m.date = match[1] + "-" + match[2] + "-" + match[3] + " " + match[4];
        continue;
      }
      
      match = line.match(/^Contents :$/);
      if(match) {
        break;
      }
    }

    for(lineIndex++; lineIndex < lines.length; lineIndex++) {
      m.body[0].value = m.body[0].value + lines[lineIndex];
    }
    m.body[0].value.replace(/\n\r/g, "\n");
    
    body(m);
  });
}

In the main program this parser is called with a callback, making it appear similar to a loop.

var messages = new Array();

  // sms messages
  var folder = DriveApp.getFolderById(...);
  var files = folder.getFiles();
  while(files.hasNext()) {
    var file = files.next();
    forEachLgSms(file.getBlob(), function(message) {
      messages.push(message);
    });
  }

The two examples above would parse into this:

[
  {
    type: "incoming",
    subject: "",
    date: "2013-04-11 17:58",
    from: [ { address: "+1555555555", name: "Sample Name" } ],
    to: [ ],
    body: [ { type: "text", value: "See you" } ]
  },
  {
    type: "outgoing",
    subject: "",
    date: "2013-04-03 20:37",
    from: [ ],
    to: [ { address: "+1555555555", name: "Dear Friend" } ],
    body: [ { type: "text", value: "Back in my very warm fun fur hammock tonight, there was some pretty snow today, and i get to sleep in a bit tomorrow :)\nSweet dreams! <3" } ]
  }
]

What I did with the multi-media messages will be covered another time.

rainbow water bottle

Just before I left to hitchhike across the country to deliver one of my rainbow lighters to a close friend in a campground in BC, I painted myself a rainbow swirl metal water bottle.

The bottle has served me very well and even has a tendency of finding me when I leave it by itself for a while outside of where I am residing at the moment.

Wool lined hammock

I just finished my wool lined hammock for traveling Sunday...

It feels like like a tent on the inside when zipped up, though the similarity ends at there being a dark floor and light walls and ceiling.

The outside is nice and plain, all windbreaker nylon. That is my tarp tucked to the side.

The brown part is the wool, the rest is light nylon.

The hammock does not take up much room in my bag, and I have taken it down to zero with my heavy clothes on for a night, before the zippers were added, I expect I can sleep in this just past zero now.

Dehydrated orange cookies

I dehydrated some oranges, but instead of just slicing them thinly, I peeled them and sliced them in four, making round slices.  They took a long time to dry, but they look like and feel like cookies, but taste like orange.  Two cookies makes half an orange.  I also dried the peeling since it is edible.

So yummy, the oranges don't last long like this.

funky food jar

I had another empty jar taped and ready to paint... This time I decided to put a drop of paint on it, spread it using a brush without overlapping paint that was already on it, change colors, and repeat until it was covered.

updated sunrise jar

I was not feeling pleased with my sunset jar, so I got out the paint and turned it into a nice sunrise jar.

I like this more, but it still feels like something is missing, when I figure out what I will updated it again.