I know it’s a little late but the usual drive of new year innovation and releases within the Open Source world have been keeping me busy recently.

New years resolutions

1. Package AuthStor
2. Learn another RDBMS – Oracle / MSSQL?
3. Upgrade my local servers to FreeBSD 7.1
4. Pick up the guitar at least once a week
5. Lay down my first audio track in Ableton Live

The year started with my attempt at some resolutions (you know the ones you never keep) of which I am already making good progress on.

First on my list was AuthStor which is working on a basic level at the moment but there is so much more that can be done to make it easier to use. Why has it taken so long you may ask?

Well apart from being the only developer last year, I made several attempts to re-write the app in something like JSP or Zend Framework, partly because there is a lot of activity in those communities but mostly due to the complications faced with packaging around 50 Perl modules that each have their own compile time dependencies – some of which change frequently often breaking the application (no strict API/Interfaces to adhere to).

So after several prototypes why did I decide to stick with Perl?
Well for starters the changes that did break the application often forced me to change the app in a way that was more logical or introduced a more structured design. I also found that JSP and Zend although cool frameworks tended to feel incomplete and introduced complexities elsewhere e.g. containers / authentication / roles. Catalyst is simple and extremely powerful providing both a development environment and production framework that takes little effort to introduce new features quickly. I can start 2009 in the knowledge that Perl and Catalyst are the right language / framework for the job. I just need to work on the packaging now!

The next item on my list is to learn yet another RDBMS. Why? Enlightenment first and foremost, plus I am a big believer in gaining as much knowledge as possible in my chosen field. You’ll never complete the jigsaw if you have several pieces missing, and seeing as I spent a good part of last year getting to know DB2 I decided this year that my focus would by Oracle and MSSQL, thus completing the picture.

With the release of FreeBSD 7.1 there are now so many cool features that I decided to upgrade some of my older servers that have been sitting in the corner running nicely on FreeBSD 6 (they never fail). DTrace and ZFS put a stop to FreeBSD slipping behind OpenSolaris and the ULE scheduler helps improve performance over your average Linux distro, and with ULE 3 due for release in FreeBSD 8.0 (June) you can expect to see further dramatic performance improvements.

Last on my list will be to pick the Les Paul at least once a week!. One of my prezzies in 2008 was a new orange amp! If that does not encourage me to pick up the guitar I don’t know what will!. You never know I might even mix something up with the copy of Live I got for Christmas

It’s been a while since I had chance to work on Informeer as my itch was one of multi-user web based password management (AuthStor). Oh and moving house.

Now that things are settling down again (Servers back up and running) I decided to take a break from AuthStor and focus on something new – Informeer.

The concept is simple, Centralised Notification.

I am forever configuring notifications from several sources, be it backup alerts, host monitoring notification and even simple applications that send mail via SMTP. When living in a world of change, both software and business, having to visit every application to change an e-mail address or add a new user to a notification schedule can be quite time consuming. Add to that the effort of having to modify firewalls, SMTP servers and XMPP settings etc etc. The idea of a single web interface where all your notifications can be configured is quite appealing. Informeer aims to make that a reality (one day).

While I take my time with the implementation I thought I would post a basic intro to RPC-XML using Perl. The RPC-XML Perl module makes writing your own client/server application a piece of cake, and while not quite point and click, it will give you enough flexibility to centralise your own notifications (or anything else for that matter). It’s not rocket science, but it works!

If you have been looking for a flexible quick and easy fix to the centralised notification problem, or even if you are just looking to push some data from a firewalled site to one of your servers This Tutorial should help get you started.

In between moving house and playing with notifications I found time to upgrade to Wordpress 2.5 and MoinMoin 1.6, and wow what an improvement in both camps.
The Wordpress dashboard is the most visible improvement – it’s amazingly clean! If you have been waiting for 2.5 to settle down or just cautious about upgrading, I would say take the plunge, you won’t regret it. As for MoinMoin 1.6, there really is no better Wiki engine out there (my opinion) and it just keeps getting better with every release!.

That just leaves me to post a quick MySQL tip that may well be obvious, but happens to catch me out from time to time.

MySQL Tip of the month

If you ever get the urge to convert a 1Gig MyISAM table (with 34 million rows) to InnoDB within MySQL, try to avoid the simple ALTER TABLE tablename ENGINE=INNODB; method.
I made the mistake of running that command on a fairly decent server with bags of space and memory only to find the command running over 5 hours later with an InnoDB tablespace at around 4Gig (ok a bit of tuning might have helped)

Your best bet is to dump the table using mysqldump, drop the existing table and re-import the table after modifying the definition to be InnoDB. That worked for me in less than 10 minutes. Alternatively you can create a new InnoDB table and insert directly from the existing table as per MySQL documentation e.g INSERT INTO newtable SELECT * FROM oldtable.

I’m sure I am not the first to make this mistake, and may not be the last…..

AuthStor came to life back in late 2007 after the Password Safe I had been using became corrupt. Not through any failure in Password Safe’s functionality but really to the way it was being used – Multi-User.

I had been seeking a decent Web Based Password Manager (Open Source of course) for ages but had not found anything that I liked. All the semi-decent software out there seemed to be Closed Source with license fees attached and to be honest I would feel more comfortable if I understood the inner workings, especially if this software is storing the key to my empire (mwuhahahaha).

So after a long history of Spreadsheets, Password Safe, Keychain and various other methods of storing passwords I finally decided to scratch the itch and roll my own password manager.

Now AuthStor is not my first attempt at a password manager, in fact I originally started to write an extension to MoinMoin that provided a method of storing passwords directly within the Wiki. This worked great for a while but was not the most secure method of managing passwords, especially as the secret key was embedded within the Python code!. I learned a lot from mmPasswords and hope to improve upon that greatly through the development of AuthStor.

So what is in store for AuthStor? (Get it? ) Well key features will include Multi-User capability, Full auditing, GnuPG encryption, REST like interface and that is just the start. Future plans include interfaces like SMS, VoIP and anything else I can think up.

Here is a sneak peak of the progress made so far:

The application itself is written in Perl and runs on top of the Catalyst Framework. Why Catalyst? Flexibility, Performance, Scalability, the list goes on… Catalyst’s plugin architecture provides multiple views (among many other things) so I can focus on the functionality and not worry about the presentation. Speaking of presentation, AuthStor makes extensive use of the Yahoo! User Interface Library to provide a rich interface with dynamic elements such as DataTable’s, Toolbars and Treeviews and all wrapped in a BSD License!!. Nice work folks!.

Expect a commit in the next few weeks…

2007 is almost over and what a year this has been for Wave2!

My programming preferences have gone full circle, from beginning the year playing snakes and ladders (Python), I spent the summer drinking coffee (Java) and finally settled down surrounded by Camel’s, Lama’s and Parrots (Perl). Bizarre but never dull!

Steady progress has been made all round and as always the Open Source community has proven a valuable resource. Not a day went by where I did not discover something new or think of a new idea that could be built using Open Source Software. I only wish I had more time in the day!.

My Christmas present came early this year in the form of a MySQL 5.1 Cluster DBA Certification Study Guide. I have been putting off the final MySQL certification while waiting for the book to arrive, and now it is in my hands the only thing preventing me from taking the exam is the lack of free space over the Christmas period. Hehe

I also ordered a Nokia N810 (My Techie Present). My laptop is not small enough to carry around with me and I am hoping to capture more development time on this device. Either that or feed my mind.

Anyway I’m off to hit the WarHawk servers before getting ready to see the New Year in Amsterdam – home of the Blender Institute. With projects like AuthStor making great progress and ideas such as Unix Architect and BinaryStor slowly maturing, I cannot wait to see what 2008 holds for Wave2!

Before I head off I have to mention the Catalyst Advent Calendar. Every year they pack 24 days of tutorials and tips into the month of December. If you are looking to get the most of the Catalyst Framework you really must take a look at this resource. I was able to prototype AuthStor in under 2 weeks and enjoyed every minute!

Wave2 Wishes You a Very Merry Christmas and an Open Source New Year!!!

It’s been a few weeks since I last posted here or even found time to develop any of the ideas that continually spring to mind. In fact this is the first weekend in several where my head is not stuck in a book about Perl, Solaris or UML. Why you may ask? (or not).

Well after finally getting round to taking the Solaris Network Administrator exam, I thought it only right that I complete the set and get to know Solaris Security a bit better by taking the SCSECA.

After achieving the SCSECA, I was just about getting ready to start writing code again when I stumbled upon the UML Certification Program.
I have been a fan of UML for a long time but never found a tool that I felt comfortable using, or more to the point a tool worth the money!. That was until on the very same day I decided to trial Visual Paradigm, could this be the UML tool I have been seeking?

Visual Paradigm for UML is not overly expensive (many price levels to cater for all requirements), it runs on Windows / Linux / OSX and it’s easy to use – I like it!.

Armed with the UML tool I have been looking for and a copy of the UML 2 Certification Guide I spent another week brushing up on my UML skills while studying for the first of 3 exams offered by the OMG, UML Certified Professional – Fundamental.
I passed the exam last week and learned a lot more UML along the way!
So finally, with all that studying out of the way I can finally get back to the fun stuff – Code.

But where does Perl fall into this jacked-in period? Well I have been waiting for Perl 6 for a long time now and plan to use Catalyst as the platform for BinaryStor and AuthStor. So what better way to get up to speed than a Perl readathon of as many Perl books as I can get my hands on. From Learning to Mastering and eventually Best Practices all over again, you never know I might just finish reading before the release of Perl 6.

As for the Code I cannot make my mind up as to where to focus my energy and so opted for a working NetUnit (I did think of porting it to Perl ). It’s far from complete of course and suffers from the constraint of root privileges, but I’m hoping a future Java release will fix that….

While working on MySQLDump.java this weekend I decided to look into the new features that are heading our way in MySQL 5.1. It cannot be long before we see a general release and I needed to start thinking of ways to backup items like Events. This also made a good opportunity to start afresh with my dev server and try out the latest Solaris 10 8/07 release with GlassFish v2, the releases just keep coming! (SXDE 9/07 was just released).

The Solaris 10 install was as smooth as previous releases (Using Text Based) and once started took at least an hour or so to finish, I cannot remember the exact time as after checking the status a couple of times I forgot all about it. The general feel was much improved however, and the overall speed increase made the upgrade worthwhile.

GlassFish installed and ran as expected – easy. I would be very interested to see the difference in http performance when compared to Apache. Something for another day maybe?

Next on my list was MySQL 5.1.21-beta.
I originally downloaded the pkgadd version but failed to start the server due to a an [ERROR] Can't find messagefile. It was only after receiving the same message with the Tar distribution that I headed over to the MySQL Bug System where it seemed the issue was a known one – BugID 30678.

Not a major issue as the solution was to simply add --language=./share/mysql/english/ to server install / startup commands. Something that I am sure would have been fixed in the release candidate. I would like to see a more complete installer for Solaris, something similar to the FreeBSD port. The pkgadd install does not really offer much customisation and both the Tar file / pkgadd lack any sample service manifest files.

Once installed everything worked as expected and my first task was to install the Sakila test database. I intend to use this database to validate the operation of the MySQLDump.java app. It may not be the biggest database to work with but it does support a wide variety of data types, views, triggers etc. I am currently using the MySQL Toolkit to verify the backup and restore. If you have not used the Toolkit before (it’s written in Perl) it includes a nice utility that can test if 2 tables are identical – handy for validating a backup.

I envisage using unit tests for this purpose in future, but need to find a good book on TestNG first. A quick dump of the film_text table worked as expected and the data was identical to that supplied by MySQL, I must be on the right track! – If only I had more time…

I did however find time to attend the MySQL Backup and Recovery webinar this morning and was impressed with the new Backup API planned for MySQL 5.2 onward. I just booked my place at the European MySQL Conference and hope to get some more insight into future developments, or at least a copy of the MySQL 5.1 Cluster Certification Study Guide.

I put the Asterisk Widget on the backburner this week to try and make way for another one of my many on-going projects – Informeer.

The concept is simple – a notification service – the implementation on the other hand will take some thinking

I originally planned to develop Informeer in Java and provide both client and server components to handle the notifications, however since I discovered Catalyst and started to read RESTful Web Services things have changed for the better.

Informeer in its essence is a framework that aims to simplify the distribution and management of notification messages. Huh?
Ok lets start at the beginning…..Question – how can I notify a group of individuals from OpenNMS? Easy – just create a notification command. Ok but what if the individuals want to be notified by SMS or Jabber? Just as easy – create multiple notifications per user.
Right – now what about all my other OpenNMS instances, IP monitor servers, Nagios servers and that cool script I wrote last summer?.

Enter Informeer

Informeer aims to be a single point of administration and management for notification messages and destination paths (SMS, Jabber, E-mail etc).
The initial API will be HTTP based so you can be pretty sure the mesasges will pass through your firewall, XMLRPC anyone?. There will also be a web interface for administration and form based notification submission.

Sounds cool huh? Well I better get back to it then….

For quite a while I have been playing with Web Frameworks in the hope that I would find a more “Large Scale” method of writing Web Applications.

Since being hooked on Python at the time I started with Zope, reading the programmers guide from front to back I ended up believing that Zope was indeed a great Framework, It made sense why it was rebuilt from scratch, but I was no closer to being able to use it.
Then came Django and TurboGears. Once again I skimmed the manuals and even gave Django an install but they still felt alien to me and incomplete (Maybe things have changed now – that was over a year ago:)).

So I lay down my Python tools and picked up my Java book, after all Java 6 was much faster now and many Java based Web Frameworks were out there and driving some of the largest sites in the world. My only problem now was choice.
Choosing a decent Framework using Java is tough – not only do you need to choose the Framework you need to choose a container to host your application. Do you use JBoss,Geronimo (My Fave), GlassFish or any of the other App Servers out there? and do they support your chosen OS? Java was written by Sun and always runs faster on Solaris – but I use FreeBSD – something that Sun don’t even offer a download of the JDK for!. In the end I got Geronimo working fine on FreeBSD – but it still did not feel right.

That’s when I found Catalyst

Described as an Elegant MVC Framework, Catalyst felt right, looked right and worked perfectly on FreeBSD.
Written in Perl and designed for Perl developers my enthusiasm for Perl 6 drove me to give it a whirl and am I glad I did!
The documentation is excellent, the built in debugging actually assists the developer and the Framework is so flexible you can write your App pretty much any way you like.

I plan to use the Catalyst Framework for the notification application Wave2 Infomeer and will let you know how things work out, but in the meantime if like me you have tried several Web Frameworks and never felt comfortable I’d suggest you give Catalyst an hour of your time – you’ll be impressed!.