Tuesday, December 27, 2011

Recover text messages from nandroid backup

The objective is to restore the text messages that were on my Android phone before it got re-flashed, with an end result of all the old and current text messages being on the phone.

I have a nandroid backup image at ~/work/backup/, and the phone attached by USB with debugging enabled.

Previously I tried to install the yaffs file system on a Linux box and mount the image over loop back, however that did not work, the images failed to mount.

The solution turns out to be simple...

download the source for unyaffs, a program which extracts all the files form a yaffs image.

download unyaffs.c and unyaffs.h into ~/work/.

compile the source.
cd ~/work
gcc -o unyaffs unyaffs.c

make a target directory and run the program from the target directory.
mkdir data.img
cd data.img
../unyaffs ~/work/backup/data.img

after some quick reading I find that the text message database are at data/com.android.providers.telephony/databases/mmssms.db.

On the phone use SMS Backup & Restore to save the current text messages.

Make a backup of the message database on the phone, just in case.
adb pull /data/data/com.android.providers.telephony/databases/mmssms.db ~/work/mmssms.db-current

Push the pre-flash message database with the old messages onto the phone
adb push ~/work/data.img/data/com.android.providers.telephony/databases/mmssms.db /data/data/com.android.providers.telephony/databases/mmssms.db

Reboot the phone.

Only the old messages should be visible on the phone now.

On the phone use SMS Backup & Restore to restore the text messages that were just saved, there should be no need to check for duplicate messages.

All the messages should be visible now. All done.

Saturday, December 24, 2011

Minimal Swing GUI example

A well behaved graphical application will place windows like other applications on the platform place them, and will exit when the last window is closed, and also all GUI work will happen on the event thread, this example does all of that.

package org.yi.happy.teaching.gui;

import javax.swing.JFrame;
import javax.swing.SwingUtilities;
import javax.swing.WindowConstants;

public class SwingLaunch {
    public static void main(String[] args) {

        /* ALL GUI work should be done on the event thread */
        SwingUtilities.invokeLater(new Runnable() {
            public void run() {

                /* make a window */
                JFrame frame = new JFrame(SwingLaunch.class.getName());
                frame.pack();

                /*
                 * when all the windows are disposed the application will exit,
                 * make the window automatically dispose when it is closed
                 */
                frame.setDefaultCloseOperation(WindowConstants.DISPOSE_ON_CLOSE);

                /* put the window where the platform would put it */
                frame.setLocationByPlatform(true);

                /* show the window on the screen */
                frame.setVisible(true);
            }
        });
    }
}

For Day Job I have also written a GUI Launcher class that takes a swing component or a class and launches a window containing it with these settings.

Wednesday, August 3, 2011

Brownies

My friend Josie found this really simple and good brownie recipe, and I made them again recently so I am posting it here now.
Josie and I made brownies a while ago, and the recipe she had was very quick and easy, the next day the page of the book she wrote it on got soaked and she couldn't read it any more, so here is a copy transcribed from pictures.
Vegan Brownies
2 cups unbleached all purpose flour
2 cups white sugar (or other)
3/4 cup cocoa
1 teaspoon baking powder
1 teaspoon salt
1 cup water
1 cup vegetable oil
1 teaspoon vanilla extract
1. Preheat oven to 350 f
2. Stir together flour, sugar, cocoa powder, and salt. Pour in water, oil, and vanilla, blend well, spread evenly in a 9 x 13 inch baking pan.
3. Bake for 25 to 30 minutes in preheated oven, until top is no longer shiny, let cool about 10 minutes before cutting into squares.

YAY
BROWNIES!

Saturday, March 19, 2011

video to mp3

I had a desire to take a video file and extract the audio track into an mp3 file, so I ended up using a variant of the mp4 converter script to do the job.

#!/bin/bash
[ "$1" ] || {
  echo use: $0 file...
  exit 0
}

for file in "$@"; do
  ffmpeg -i "$file" -acodec libmp3lame -aq 100 "$file.mp3"
done

This script should work on most audio or video formats and makes variable bit rate high quality mp3 files.

Wednesday, March 2, 2011

mount on plug of external disk

For the home server we got an external USB powered hard disk for the public file share. Because this disk may not be present when the computer is started up, and may change device name since it is USB attached, I desired a way to mount that specific hard disk when it was plugged in.

The answer was a fairly long but simple udev rule, and a script to do the mounting.

/etc/udev/rules.d/storage.rules
ATTRS{idVendor}=="0bc2", ATTRS{idProduct}=="5031", ATTRS{serial}=="NA0B3DKV", ENV{DEVTYPE}=="partition", ACTION=="add", RUN+="/usr/local/sbin/mount-storage"

/usr/local/sbin/mount-storage
#!/bin/sh
mount -t vfat -o uid=nobody,gid=nogroup,dmask=000,fmask=111 \
 "$DEVNAME" /mnt/store

I found how to write the rule after referring to Writing udev rules and an example of a rule that matches the type of device.

It is still required to unmount the disk before unplugging it, as is the case with any operating system at the moment.

Monday, February 28, 2011

pancake syrup

When I am making pancakes, sometimes I like to have brown sugar syrup with them, it is also much cheaper to make pancake syrup from scratch than to buy it from the grocery store.

mix 1 cup brown sugar
and 2 cups water

boil for 25 minutes.

add vanilla.

That's all there is to that one.

Monday, February 21, 2011

console input in java

I wrote up a sample of nice and clean console input with validation, and repeated prompting on invalid inputs.

package org.yi.happy.console;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

public class DemoInput {
    /**
     * A demo of prompting for and reading validated data from standard input.
     */
    public static void main(String[] args) {
        try {
            /*
             * get a usable line reader.
             */
            BufferedReader in = new BufferedReader(new InputStreamReader(
                    System.in));

            /*
             * the result of the read loop
             */
            String word = null;
            /*
             * keep going until we get what we want
             */
            while (true) {
                /*
                 * ask the question
                 */
                System.out.print("enter a word: ");
                /*
                 * get the answer
                 */
                String line = in.readLine();
                /*
                 * is the answer what we want?
                 */
                if (line.matches("\\w+")) {
                    /*
                     * save the answer and done
                     */
                    word = line;
                    break;
                } else {
                    /*
                     * report a bad answer and repeat
                     */
                    System.out.println("that does not look like a word: "
                            + line);
                    continue;
                }
            }

            /*
             * the result of the read loop
             */
            double number = 0;
            /*
             * keep going until we get what we want
             */
            while (true) {
                /*
                 * ask the question
                 */
                System.out.print("enter a number: ");
                /*
                 * get the answer
                 */
                String line = in.readLine();
                /*
                 * parse the answer
                 */
                try {
                    number = Double.parseDouble(line);

                    /*
                     * range checks can go here (on failure do a continue, on
                     * accept do a break)
                     */

                    /*
                     * it was good, done
                     */
                    break;
                } catch (NumberFormatException e) {
                    /*
                     * it was not good, report the bad answer and repeat
                     */
                    System.out.println("that does not look like a number: "
                            + line);
                    continue;
                }
            }

            /*
             * if we get here we have two good answers.
             */

            /*
             * do something using the input
             */
            System.out.println("the word was " + word + " and the number was "
                    + number);

        } catch (IOException e) {
            /*
             * there was an error, so display it and give up
             */
            e.printStackTrace();
        }

    }
}

I don't think I can make it any simpler and still repeat the prompting for input until valid input is given. If there were many inputs being done I could break the logic out into an instance of the template pattern.

Sunday, February 20, 2011

all sub-sets of a set in java

I was offered the challenge to find all the sub-sets of a set, represented as an ordered list, using java.

I know a list isn't really a set, but it is the same idea in the end, since the elements are not repeated, also the non-recursive implementation I came up with would work equally well with sets as lists.

I started with a recursive implementation, and later found a non-recursive implementation with the same output.

First I made some tests to show what was going to happen.

package org.yi.happy.subset;

import static org.junit.Assert.assertEquals;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import org.junit.Test;

public class FindSubsetsTest {
    @Test
    public void testFindAll() {
        List<Integer> in = Arrays.asList(0,1,2,3);
        
        List<List<Integer>> have = FindSubsets.findAll(in);
        
        List<List<Integer>> want = new ArrayList<List<Integer>>();
        want.add(new ArrayList<Integer>());
        want.add(Arrays.asList(3));
        want.add(Arrays.asList(2));
        want.add(Arrays.asList(2, 3));
        want.add(Arrays.asList(1));
        want.add(Arrays.asList(1, 3));
        want.add(Arrays.asList(1, 2));
        want.add(Arrays.asList(1, 2, 3));
        want.add(Arrays.asList(0));
        want.add(Arrays.asList(0, 3));
        want.add(Arrays.asList(0, 2));
        want.add(Arrays.asList(0, 2, 3));
        want.add(Arrays.asList(0, 1));
        want.add(Arrays.asList(0, 1, 3));
        want.add(Arrays.asList(0, 1, 2));
        want.add(Arrays.asList(0, 1, 2, 3));

        assertEquals(want, have);
    }

    @Test
    public void testFindAllFlat() {
        List<Integer> in = Arrays.asList(0, 1, 2, 3);

        List<List<Integer>> have = FindSubsets.findAllFlat(in);

        List<List<Integer>> want = new ArrayList<List<Integer>>();
        want.add(new ArrayList<Integer>());
        want.add(Arrays.asList(3));
        want.add(Arrays.asList(2));
        want.add(Arrays.asList(2, 3));
        want.add(Arrays.asList(1));
        want.add(Arrays.asList(1, 3));
        want.add(Arrays.asList(1, 2));
        want.add(Arrays.asList(1, 2, 3));
        want.add(Arrays.asList(0));
        want.add(Arrays.asList(0, 3));
        want.add(Arrays.asList(0, 2));
        want.add(Arrays.asList(0, 2, 3));
        want.add(Arrays.asList(0, 1));
        want.add(Arrays.asList(0, 1, 3));
        want.add(Arrays.asList(0, 1, 2));
        want.add(Arrays.asList(0, 1, 2, 3));

        assertEquals(want, have);
    }
}

And the implementation, with implementation notes in it.

package org.yi.happy.subset;

import java.util.ArrayList;
import java.util.List;


public class FindSubsets {

    /**
     * Find all the subsets of in, recursively.
     * 
     * @param <T>
     *            the type of item items in the set.
     * @param in
     *            the set to get subsets off.
     * @return the list of subsets of in. The order is the same as doing binary
     *         counting with the least significant digit on the right.
     */
    public static <T> List<List<T>> findAll(List<T> in) {
        /*
         * we can do this recursively, we either exclude or include the first
         * item, and join all the deeper subsets to it.
         */

        if (in.size() == 0) {
            /*
             * the base case is just the empty set.
             */
            List<List<T>> out = new ArrayList<List<T>>();
            out.add(new ArrayList<T>());
            return out;
        }

        List<List<T>> out = new ArrayList<List<T>>();

        T first = in.get(0);
        List<List<T>> rest = findAll(in.subList(1, in.size()));

        /*
         * first all the sets excluding the first item
         */
        for (List<T> r : rest) {
            out.add(r);
        }

        /*
         * next all the sets including the first item
         */
        for (List<T> r : rest) {
            List<T> s = new ArrayList<T>();
            s.add(first);
            s.addAll(r);
            out.add(s);
        }

        return out;
    }

    /**
     * Find all the subsets of in, non-recursively.
     * 
     * @param <T>
     *            the type of item items in the set.
     * @param in
     *            the set to get subsets off.
     * @return the list of subsets of in.
     */
    public static <T> List<List<T>> findAllFlat(List<T> in) {
        List<List<T>> out = new ArrayList<List<T>>();
        out.add(new ArrayList<T>());

        for (T i : in) {
            List<List<T>> next = new ArrayList<List<T>>();
            for (List<T> j : out) {
                next.add(j);

                List<T> k = new ArrayList<T>(j);
                k.add(i);
                next.add(k);
            }
            out = next;
        }

        return out;
    }

}

It was not much harder than counting binary to make this list, I gave up on adding the restriction where the resulting lists of lists should be sorted.

Sunday, February 6, 2011

HTTP subversion access

The objective is restore the subversion repository URL space from my old web site. This will require making some subversion repositories available over HTTP using Apache 2.2. Authentication is required only for updates. Access to the repositories should be configured in the virtual host, and not globally.

The environment for the directions is Ubuntu 10.10 (Maverick Meerkat) Desktop freshly installed and updates done.

This example is for exposing multiple repositories under /svn in the URL space of a virtual host.

Install apache with the subversion module, and subversion:
sudo apt-get install apache2 libapache2-svn subversion

Make a home for the physical subversion repositories.
sudo mkdir /var/svn

And create a few repositories accessable to the web server:
sudo svnadmin create /var/svn/repos
sudo chown -R www-data:www-data /var/svn/repos
sudo svnadmin create /var/svn/repos2
sudo chown -R www-data:www-data /var/svn/repos2

create some users for authenticaton
sudo touch /var/svn/passwd
sudo htpasswd -bm /var/svn/passwd user1 pass1
sudo htpasswd -bm /var/svn/passwd user2 pass2

create the access control file /var/svn/access
[repos:/]
user1 = rw
* = r

[repos2:/]
user1 = rw
user2 = r
* =

Create a virtual host by creating /etc/apache2/sites-available/demo.local:
<VirtualHost *:80>
ServerName demo.local
ErrorLog ${APACHE_LOG_DIR}/demo.local-error.log
CustomLog ${APACHE_LOG_DIR}/demo.local-access.log combined

<Location /svn>
DAV svn
SVNParentPath /var/svn
SVNAutoVersioning On
AuthzSVNAccessFile /var/svn/access
Satisfy Any
Require valid-user
AuthType Basic
AuthName "Subversion repository"
AuthUserFile /var/svn/passwd
</Location>

</VirtualHost>

enable the virtual host
sudo a2ensite demo.local
sudo /etc/init.d/apache2 reload

At this point the repositories are available on the web server, and can be mounted using WebDAV. Extra details are in the SVN red book.

better mp4 conversion

Since I am using a Mac, it is sometimes good to convert videos that the stock player can not play into a format that it can play, for instance wmv to mp4, since wmv files can not be played by quicktime or the finder preview, VLC plays them instead.

Today I was looking at quality settings and found a tip on setting the output quality in ffmpeg.

My new script that converts all the files given on the command line to mp4 format is
#!/bin/bash
[ "$1" ] || {
  echo use: $0 file...
  exit 0
}

for file in "$@"; do
  ffmpeg -i "$file" -acodec libfaac -vcodec mpeg4 -qmax 8 "$file.mp4"
done

Lower numbers for -qmax are higher quality and larger file sizes.

Friday, February 4, 2011

minimal asterisk install

The objective is to do as little as possible to setup an asterisk server running and be able to accept a call from a sip client. The sip client is running on the same network as the server.

The environment is Ubuntu 10.10 (Maverick Meerkat) Desktop freshly installed and updates done.

Firstly, install asterisk:
sudo apt-get install asterisk
being in Canada my telephone country code is 1

Second, add a demo sip device by adding these lines to /etc/asterisk/sip.conf:
[user-01]
type=friend
context=demo
secret=test
qualify=yes
host=dynamic

Finally, reload the sip configuration
sudo asterisk -rx ‘sip reload’

Now you can connect with a sip client, such as Telephone for Mac

the settings in this example are:
Domain: test-server.local
User Name: user-01
Password: test

Dial s or 1000 to activate the demo.

Monday, January 24, 2011

old web site migration

I have a very old web site which has not been maintained for years on the server that is being replaced. It was mostly an experimentation space, and I doubt it was used. I decided that the migration strategy I would use with it is to just copy over bits, or mark them to return error 410 gone, as I find 404 errors in the logs.

A quick perl script that returns errors from the access log follows.

#!/usr/bin/perl -w
use strict;

while(<>) {
    my($status) = m{^\S+ \S+ \S+ \[.*?\] ".*?" (\d+) \d+ ".*?" ".*?"};
    
    if(not defined $status) {
        print "failed: $_";
        next;
    }
    
    if($status =~ m{^[45]}) {
        print $_;
    }
}

I probably haven't updated that site in over two years. I have also considered dropping the web site completely, which may still happen after watching the access logs for a while.

Sunday, January 16, 2011

Backup MySQL to a file per database

The objective is to make a backup of a MySQL database server, and end up with a file for each database, named based on the database.

This is part of a server migration where not all of the databases will be created on the target host, and some will be renamed as they are moved to the new host.

It turned out to be a single line of shell to do the task:
mysql --user=root --password=password --batch --skip-column-names --execute 'show databases' |
while read x ; do
 echo "dumping $x..."
 mysqldump --user=root --password=password --all "$x" > "$x".sql
done
The script was run in the target directory for the backup files.

It was also revealed that one of the databases had some corruption during the backup run, which was fixed with a quick invocation of the mysqlrepair command.

DNS Server basic setup

The objective is to set up a name server on a Ubuntu server that serves some domains to the Internet. The domains being served are too complex to be managed by the provider of the domain name, as a result they are being hosted on a home server on a dynamic IP.

The environment is Ubuntu 10.10 (Maverick Meerkat) Desktop freshly installed and updates done.

In the process of offloading as much DNS responsibility as possible to external services I found that MX and CNAME records clash, so if there are MX records for a domain, then the top of the domain should not have a CNAME record, so I used an A record that points at the yi.org url redirector server, in the future I may actually update the A record dynamically.

Given that, the following is the minimal steps required to configure the name server.

install the name server software:
sudo apt-get install bind9

Set up the zone file, the top level records should include the top level records that are also provided by the external services.
/etc/bind/db.happy.yi.org:
$TTL 604800
@ 3600 IN SOA happy.yi.org. happy.happy.yi.org. (
 2011011601 ; serial
 604800 ; refresh
 86400 ; retry
 2419200 ; expire
 3600 ) ; default ttl
@ 86400 IN NS sunriseyoga.dyndns.org.
@ 3600 IN A 173.203.238.64
@ 86400 IN MX 10 ASPMX.L.GOOGLE.COM.
@ 86400 IN MX 20 ALT1.ASPMX.L.GOOGLE.COM.
www 86400 IN CNAME sunriseyoga.dyndns.org.
vnc 86400 IN CNAME sunriseyoga.dyndns.org.
mail 86400 CNAME ghs.google.com.
pages 86400 CNAME ghs.google.com.
docs 86400 CNAME ghs.google.com.
sites 86400 CNAME ghs.google.com.
site 86400 CNAME ghs.google.com.
app 86400 CNAME ghs.google.com.
blog 86400 CNAME ghs.google.com.
feather-wiki 86400 CNAME ghs.google.com.

tell the name server to load the zone by adding the following line to /etc/bind/named.conf.local:
zone "happy.yi.org" { type master; file "/etc/bind/db.happy.yi.org"; };

reload the name server
sudo /etc/init.d/bind9 reload

now the domain is being served, and will be accessible from the Internet if the NS records point at the server.

If you want to know more, read the Ubuntu BIND9 Server HOWTO.

Wednesday, January 12, 2011

encoding mono video for windows

I was attempting to figure out why one of the computers at school would not play a media file with sound, the media file played fine on my laptop, and the instructor’s office computer, but not on the classroom computer.

After trying various players, and converting the media file to various formats, it continued to not play with sound. I continued by trying other media files that I had on my laptop, they appeared to play fine. Eventually I noticed a light crackle after a fairly low quality conversion, which was a hint that is might have to do with audio channels. At this point I tried using mono output in VLC and the sound came out clear.

After making a tree of files that would not play properly in the same way, I converted the whole batch of videos using the following bit of shell code. The codecs I used are ones that stock windows understands.

# low quality mono (256kb)
#ABR=64kb
#VBR=192kb
# medium quality mono (512kb)
#ABR=64kb
#VBR=448kb
# high quality mono (1024kb)
ABR=64kb
VBR=960kb

find . -type d -print |
while read x ; do
  mkdir -p ../out/"$x"
done

find . -type f |
while read file ; do
  ffmpeg -i "$file" -acodec wmav2 -vcodec msmpeg4v2 -ab $ABR -b $VBR -ac 1 "../out/$file.avi" < /dev/null
done

Friday, January 7, 2011

Public file share

The objective is some basic network attached storage (NAS), or public file share, where any attached computer can create, read, update, and delete any file without authentication. I would not consider this secure, as anyone who can attach to the network can do whatever they want to the file space, however secure is not the objective at this time. Also if a user is accessing the same shared file space locally on the server it should behave the same as if it was being accessed over the network.

The environment is Ubuntu Desktop 10.10 (Maverick Meerkat) freshly installed and updates done.

Given that, the following are the minimal steps and configuration required to achieve the objective.

Make a public file space based on "HowTo: Create shared directory for local users (with bindfs)", this works much better than access control lists can.

install bindfs
sudo apt-get install bindfs

configure the public space to be set up on startup.
/etc/init/bind-public.conf:
description "Remount public with different permissions"

start on stopped mountall

pre-start exec install --owner=nobody --group=nogroup --mode=0777 \
--directory /export/public

exec bindfs -f --owner=nobody --group=nogroup --perms=a=rwD \
--create-for-user=nobody --create-for-group=nogroup \
--create-with-perms=a=rwD --chown-ignore --chgrp-ignore --chmod-ignore \
/export/public /export/public

and make the public space active
sudo initctl start bind-public

Now to make the space available over the network using Samba.

Install samba
sudo apt-get install samba

And here is a minimal Samba configuration to do the job.
/etc/samba/smb.conf:
[global]
       map to guest = Bad User

[public]
       path = /export/public
       guest ok = yes
       read only = no

It is not necessary to restart samba for the changes to take effect.

At this point the objective is achieved for remote connections, and any local methods for accessing the directory.

For restricted access, configure Samba to require a log-on, or only allow particular users to access the public share.