Releasing mmullis-flay with Fix for Flay on 1.9.2

There’s a bug in Flay under 1.9.2 which throws the following

lib/flay.rb:277:in `+': can't convert Fixnum into String (TypeError)


To help everyone out, I forked Flay on github and fixed the problem as well as adding a unit test for the condition.
Unfortunately, not having received a response to my pull request, I’m compelled to release it.

Until the primary version is fixed, my forked version is available on rubygems as “mmullis-flay”
See: http://rubygems.org/gems/mmullis-flay

Cheers,
michael.

Released Ruby ffi-uuid gem

A few days ago, I released my weekend project called ffi-uuid, via gemcutter to rubygems.org. It uses the Ruby FFI (Foreign Function Interface) gem to wrap libuuid calls. It required learning basic FFI and reading examples and only just enough to get this working.
I knew that libuuid is what I wanted to use but didn’t want to write/maintain a bunch of C just to talk to it. Writing an extension, testing it on 1.8, 1.9 and on multiple platforms made it very unappealing. The biggest issues with uuidgen are that it doesn’t support getting a list of UUIDs and it’s not always the same on each platform. OS X uuidgen doesn’t support the same options as linux.

Shortly before giving up and hacking an extention, I came across FFI and it’s perfect for this situation. Read more about FFI here: http://github.com/ffi/ffi.

The ffi-uuid gem homepage is http://rubygems.org/gems/ffi-uuid (source is on github)
Hopefully, some other smart folks will make it better.

It also required learning jeweler and gemcutter to get the gem publishing process going but it’s all pretty easy and certainly better than the manual process.

The original approach for getting UUID’s in Ruby used some other uuid gems which were cumbersome. These were replaced with a call to the e2fsprogs command: uuidgen to make life easier.
The method looked like this:

def new_uuid
`uuidgen -t`.strip
end

Nothing fancy and does the job. The ‘-t’ means time based whereas ‘-r’ is random. The result is something like this “84e15bb0-d248-11df-9307-00237dd0b9fa”

The problem was that doing this a lot is very slow. I added some caching around a bash ‘for’ loop to avoid the system command overhead which hit uuidgen a bunch and then parsed the result into a stack and it’s a lot faster but nowhere close to the FFI approach.
Here’s why switching again is worthwhile.

1,000,000 uuids:
bash/for/uuidgen   16 minutes (miserably slow)
ffi-uuid           20 seconds (wicked fast)

That’s minutes vs seconds!
The bash approach is really bad (but still faster than the prior `uuidgen` call). Bash/for/uuidgen has a ridiculous amount of overhead because I had to run it in a forked process because it chews through file handles trying to work with uuidgen and they don’t get reclaimed fast enough.

Here’s the timing for all 3 approaches as a reference.

10,000 uuids: (that's only 10k - very small number for my usage)
`uuidgen -t`.strip     77.42 seconds (unbearably slow)
bash/for/uuidgen       9.70 seconds (noticeably slow)
ffi-uuid               0.17 seconds (not a hint of slow)


To run it

• gem install ffi
• gem install ffi-uuid

Here’s a simple test program to show the usage.

require 'rubygems'
require 'ffi-uuid'


def new_uuid
FFI::UUID.generate # get just one
end


10.times { puts new_uuid }

It ended up being so fast, that I don’t need any caching and get to delete a bunch of code – always a good thing.

From this experience, I learned that FFI is much easier to deal with than writing extensions and it’s nothing to fear. You may still have to write C or find a library that does the primary function but it not nearly as complicated as writing an extension and testing the library code first should be more straightforward.

cheers
michael

Creating Revision Specific Subversion Branches/Tags with git-svn

Since git is my vcs tool of choice, I’m trying very hard to avoid using svn but in my work, it’s often a requirement to create branches and tags in subversion off a specific revision instead of randomly assuming that the current trunk is the revision that QA wants. It’s annoying to have to switch from git over to subversion on a remote server to handle this one task but there hasn’t been a process described for me on how to accomplish this. I’ve found that the git-svn branch mechanism is a little lacking in the area and couldn’t find anything regarding documentation or tutorials on how to do this

The goal: Pick a revision and base a branch or tag on it.
This process should work but I’m still testing it out so be forewarned and always use -n or –dry-run when running commands that might affect the svn repo.

To see the current revision that master points to
git svn info

Dry run the branch creation to see what it’s going to do.

$> git svn branch -n -m"test branch via git-svn" TEST_BR_01
Copying svn://subversion/data/foo/trunk at r17089 to svn://subversion/data/foo/branches/TEST_BR_01...

You can add “-t” to make that a tag instead. Usually, my process when creating release branches is to create a pre-release tag
and then based the branch off that tag. It’s not quite as simple with git-svn and we’re just going to have to focus on the revision number for correlation.

To force the branch creation at a specific revision, we have to reset the master to point to the revision we want.

To get the correct revision, we check git svn log and look for the appropriate commit it.

$> git svn log
commit a7b9cf2d8c9d7d1621739d441b8f77927b5a8c90
Author: mmullis
Date: Wed Apr 14 00:32:45 2010 +0000

refactor: eliminate duplicate code

git-svn-id: svn://subversion/data/foo/trunk@17089 22e23496-6b6f-0410-a477-83c9a211b16a

…….

commit 29ff96bc08869dd1770ccea3e68ffec2689d9483
Author: mmullis
Date: Wed Apr 14 00:32:03 2010 +0000

drop support for unused features

git-svn-id: svn://subversion/data/foo/trunk@17085 22e23496-6b6f-0410-a477-83c

What QA asked for was to wait on the refactoring so we’re going to base the release on
commit “29ff96bc08869dd1770ccea3e68ffec2689d9483″.

Now for the tricky part which took me a good while to figure out – no help from google.
We’re going to reset master to point to a different commit point.

$> git reset --hard 29ff96bc08869dd1770ccea3e68ffec2689d9483
HEAD is now at 29ff96b drop support for unused features


With that in place, let’s test the branch create again.

$> git svn branch -n -m"test branch via git-svn" TEST_BR_01
Copying svn://subversion/data/foo/trunk at r17085 to svn://subversion/data/foo/branches/TEST_BR_01...

Notice that it’s using r17085 instead of the previous r17089.
Now I can run for real by dropping the -n and we’ll have a new branch created at a specific revision point.

Don’t forget to reset the master back to remotes/trunk or you’ll screw yourself up pretty good.
This one is easy and doesn’t need the commit id.

$> git reset --hard remotes/trunk


This put me much closer to eliminating svn from my daily process.

Dialyzer Warning: Function has no local return

This one is tricky. I’ve been working with dializer and typer on coverize and hit the following warning from dialyzer:
coverize.erl:13: Function run/2 has no local return

Seems like it would have something to do with the return types. Right? Certainly seems like it would be. After searching, experimenting and trying everything I could think of with setting the return, luck bestowed upon me the following link: http://zerthurd.blogspot.com/2009/05/dialyzer-warning-no-local-return.html .

After about 3 seconds of staring at the following code, it hit me.

-spec(run/2 :: (list(string()), atom()) -> ok).
run(SourceDirs,TestSuiteModule) ->
  CompileOptions = [ debug_info,
                     { i, "./include" },
                     { outdir, "./ebin" }],
    run(SourceDirs,CompileOptions,TestSuiteModule,"./coverage").

-spec(run/4 :: (list(string()), list(atom()|tuple()), atom(), string()) -> ok).

Look closely at the values in CompileOptions. Tuples -yes, then AHA! – debug_info is an atom().
The correct spec should have atom() as an alternative in the second parameter.
Here’s the corrected version
-spec(run/4 :: (list(string()), list(atom()|tuple()), atom(), string()) -> ok).

Dialyzer is now happy!

What a huge waste of time to figure that one out. Dialyzer needs better messages for this

Bash Prompt with Git/Svn info

While looking for emacs stuff, I stumbled on this post about integrating scm info into the bash prompt.
I knew I was irritated by always typing “git status” in svn dirs or “svn st” in git dirs because I have so many of both but never got motivated to do anything about it. Thanks to Christopher Sexton for this one.
What a timesaver! Super simple to install. Here’s the link – http://www.codeography.com/2009/05/26/speedy-bash-prompt-git-and-subversion-integration.html

If on linux, just stick the first code fragment in ~/.bashrc and you’re good to go.
It shows both the git or svn revision and the current branch. Pure Awesomeness!

Accessing Erlang Records In List Comprehensions

Accessing Erlang records in List Comprehensions can be really useful when you want to keep the amount of typing down to a minimum yet retain readability in a compact format.
In my rather feeble attempt to participate in the 2009 GitHub Contest, I’m accessing and manipulating records quite often and using list comprehensions is a natural approach.

Let’s jump to some code to see something interesting. This is a little test program to see two different ways to access record elements in a list comprehension which I tend to use a lot.


module(test).
-compile([export_all]).

-record(detail, {id, name}).

run() ->
DetailList = [
#detail{id=1},
#detail{id=2}
],
% 2 different ways to access an element of record in a list of records
Result1 = [ ID || _X = #detail{id=ID} <- DetailList],
Result2 = [ X#detail.id || X <- DetailList],
io:format("Result1 ~p~n",[Result1]),
io:format("Result2 ~p~n",[Result2]).


Result1 and Result2 are [1,2] which is just the id values from Detail List. The list comprehensions are equivalent but there’s a trade-off that I consider when using them.

The second method is more obvious and shorter when it’s just one value to deal with. When there are several values that you want to pull from the
list, using one pattern is cleaner for understanding the values going into the result.

For example:

Result1 = [ {ID, Name} || _X = #detail{id=ID, name=Name} <- DetailList],
vs.
Result2 = [ {X#detail.id, X#detail.name} || X <- DetailList],

In this example, the values extracted in Result2 become obscured by the record extraction noise(X#detail).
and it takes more effort to read than simply {ID,Name}.

I’ve not benchmarked it but I believe that if there are several values to extract, the
single pattern match is faster than accessing each one separately with the dot (“.’) notation.

erlang-cheat Released

I really like the Cheat Sheets on http://cheat.errtheblog.com but I wanted to access the sheets from inside the erlang shell.  The basic code for erlang-cheat was written last year but I finally got around to cleaning it up for relase on github.  It works equally well when running an erlang shell within Emacs and makes grabbing cheat snippets a breeze.

erlang-cheat is available with my other
releases at http://github.com/mmullis

It’s just the client side since I had no intention of hosting a separate cheats site. The current code will share the same cache as the cheat rubygem so it doesn’t impose any extra burden on the source site.

The structure is there for adding eunit tests but since it’s mostly for developer use, there’s nothing of substance yet.

Happy Cheating!
michael

Softlinks and Erlang Libs on Windows

Erlang + cygwin on Windows does NOT like softlinks for libraries.

On linux, I use softlinks in erlang/lib for projects I’m working on and tracking in another location.
This practice seems to be problematic on Windows so just dont do it.

It took a bit to figure out so I hope it helps someone else save the time I lost.

Macaddr For Erlang Released

Ah. My first real Erlang effort presented for all to see – http://github.com/mmullis/erlang-macaddr.

It started with the goal of porting the macaddr gem for ruby. This is all on R12B-4 so keep that in mind. Macaddr basically runs an os specific command like ifconfig or ipconfig and attempts to extract out the mac address information from the resulting spewage using a regular expression. There’s quite a few posts about how this is not available in Erlang and that the undocumented inet:ifget and inet:getiflist are not really portable. I hope someone finds it useful.

Ok, so it’s nothing hard in Ruby but Erlang ended up being more difficult. Mostly due to the new re module not being quite the same. I also had to break the code up a bit more in order to isolate different aspects. I’m not an Erlang guru either so there’s probably some dumb code in there and optimizations needed.

Two things I learned in the process of dealing with the re module.  The first is that it is based on PCRE underneath. To help in debugging, I built the PCRE packakge using the source which is tar.bz2 in the OTP release. PCRE includes a program called pcretest that will let you run regexes with different data. It’s easy enough to figure out. What you do is validate that PCRE is handling a regex the way you want, and then back up into Erlang.

The second thing I learned is that you have to escape ALL backslashed expressions/assertions so that they pass through Erlang unmodified and get to PCRE. I was trying to use “\b” which is word boundary but it never made it down to PCRE correctly untill it hit me and I tried “\\b”. Eureka! (yes, one of my favorite shows).

The doc for re has a really small blurb in the replace function doc. Too hidden for me so I sent a request to the  erlang-bugs mailing list to make it really clear.

The real time consumer was that the regex that worked fine on windows and linux broke down on macos where there’s a much longer sequence of “nn:”.  This resulted in the exporting of an internal function so I could test it more readily.

Overall, I think it took longer than I would like a small project to take but I’m picky and wanted to make it reasonably complete with Makefile, eunit tests,  git hosting on github, Readme, edoc, clean dialyzer, testing on 3 different platforms (Linux, Windows Vista, and Mac OSX). It should be a decent structure for my next project.

mnesia and eunit

I’ve been working my way through Kevin Smith’s Erlang In Practice screencast series and decided to write some eunit tests for episode 6 (covering Mochiweb).  The plan was to do a basic test which would register a nickname, then register the recipient which would also send back the waiting message and give me a chance to compare it with the original message.

Well, it took a LOT longer than it should have. The tests were inconsistent and would fail often but not predictably. Reworking the tests and underlying code didn’t yield much help. Debugging and tracing were not helpful either. There’s some timeouts in mochiweb or the http handling that are triggered when stepping through the code in the debugger. The get on the right track, I created a more specific test for the message_store.

Eventually it became clear that mnesia:start has some delay and that the unit tests run so fast that accessing the table becomes a race. If mnesia cannot see the table yet when we try to read the table, there’s an error that looks like this

{badmatch,{aborted,{no_exists, chat_message}}}

I wrote some code to wait for the table to become available and then I came across this post – http://rakuto.blogspot.com/2008/07/erlang-tips-sometimes-to-call.html.

Apparently, there is already a function in mnesia to handle this. So, all we need is

mnesia:wait_for_tables([chat_message], 3000),

There’s a couple of places it can be used.  The first is in each mnesia transaction but this would be overkill. The problem only surfaces when mnesia:start is called but the tables are not ready for use so placing it right after mnesia:start seems reasonable.

In message_store.erl:init_store add the following line right after mnesia:start() and all should be good.

mnesia:wait_for_tables([person], 3000),

Hooray! Now the message_store tests run consistently and I can move on with the overall tests.