May 2011

A client recently supplied some old data from an archive cd produced several years back

The data had been edited on a Mac system prior to OSX

Commodore 64There are convenient Gnu / Linux tools for converting CR+LF  to Line Feed (U+000A), however lone CR (Mac OS 9 and Commodore 8 bit) had me running for emacs and sed.

Ctrl-X and Equals (=) in Emacs will tell you about the document generally, your position, and the hex for the character under the cursor.

x0d or more correctly x0D is the lone CR

sed 's/\x0D/\n/g' < loneHatM.txt > fixed.txt

…is useful.

However if you already have dos2unix and tofrodos installed on Debian, then these two commands, in the sequence shown, will work also:

  1. dos2unix -l
  2. fromdos -a

Quoting directly from the dos2unix manpage about -l switch:

-l –newline
Add additional newline.  Only DOS line endings are changed to two Unix line endings.  In Mac mode only Mac line endings are changed to two              Unix line endings.

Quoting directly from the fromdos manpage about -a switch:

-a     Always convert. If converting from DOS to Unix, this option will cause the program to remove ALL  carriage  returns.  The  default  is  to
remove  carriage  returns  only if they are followed by line feeds.  If converting from Unix to DOS, this option will cause the program to
convert ALL linefeeds to carriage return pairs. The default is to convert linefeeds only if they are not already preceded  by  a  carriage

On this occasion I was lucky in that there were some lookup tables in addition to the main load to MySQL, and these tables were small enough to open in an editor (emacs)

For larger files either take the first few hundred characters (use cut command) or inspect with a hex editor without opening the entire file.

Max OSX now uses the Unix / Linux convention, so thankfully these conversions are now rare.

Postscript: Turns out that dos2unix utils has a convenient mac2unix command, which does exactly what I require for this client work.

( Given the mac change OS9 to OSX, i can see the need to create a backup before you try out mac2unix the first time )

( The image of the commodore 64 is courtesy of Wikipedia and is located at this page )


In a couple of years there will be whole teams who have never run a benchmark on bare metal.

With that in mind, I thought it worth doing some personal and standard benchmarks, so that future cloud provision can be judged against.

Here is my table for vps, dedicated, desktop, laptop which gives me somewhere to slot future cloud instances.

benchmark table

If you have access to bare metal today, then it might be worth spending an hour now doing a table.

Some of my benchmarks are very particular to Python and/or Mathematics using Gmp library, however unixbench score (highlighted) is a standard score, that you can use in your own table as a general guide.

System Administrators sometimes share their timesaving tips, here I share two.

alias baksysconfig="cd /bak/this/sysconfig/; \
/bin/echo '*shadow* *secret* etc/ssh/ssh*_key etc/.*secret* etc/*/*secret* ' | \
/usr/bin/tr ' ' '\n' | \
tar --wildcards --checkpoint=100 --exclude '*shadow*' -X - -cf etc__`/bin/date +%Y%m%dT%H%M`U.tar /etc; \
cd -"

… and …

alias infolist="(cd /var/lib/dpkg/info/;find . -type f -name '*.list' -printf '%f\n' | awk -F'.list' '{print \$1}')"

You probably already have a local shell startup script, that sets your aliases, and includes your own handy entries.

The –checkpoint flag for tar is a great manual check that:

  • Something actually gets put in the tar file
  • Your whole filesystem is not getting shoved into the tar file
Copying and pasting Linux command lines from the web, it is easy to miss a bit of the command.
Those two points I listed might sound really obvious, but when working in space limited VPS environments, a typo that results in the whole filesystem being tarred, is not a good situation 😐