When they announced the venue for EuroBSDCon 2009 I was pretty excited to hear that it was going to be held in Cambridge, just a few minutes down the road from where I work. I was even more excited however when I heard that Dr. Marshall Kirk McKusick was going to be giving a 1 day tutorial on the inner workings of the FreeBSD kernel!

I have been working with FreeBSD for well over 12 years now and have read many a book on the subject, including a few written by Kirk McKusick, one of which was the subject of this tutorial ‘The Design and Implementation of the FreeBSD Operating System’. It’s been a while since I last read the book and as FreeBSD is so reliable I have yet to find an itch within the Kernel that has not been scratched by some other developer.
That does not mean that I am not curious about the inner workings, after all if you do not fully understand something how can you maximise it’s potential.

Justifying the spend in the current economic climate was difficult, but there was no way I was going to miss the tutorial even at the expense of missing the weekend talks which I later found out contained another session by Kirk on SuperPages in FreeBSD 8.0.

The tutorial and conference were being held at Robinson College, not too far from the centre and a peaceful walk past the playing fields. Upon arriving I was surprised to see so many beastie T-shirts and many famous faces from the world of Unix!

The tutorial was one of 3 being given that day and saw around 20 delegates squeeze into the Music Room, a small yet comfortable room with brick walls and a couple of oil paintings, one of which had what looked like a tiny Beastie painted in a field (check out the photo)….weird huh….

The tutorial began with an explanation that the course had been condensed over time due to the request of conference organisers and was usually taught over several 1hr 30 sessions. This was the first attempt at squeezing the content into a 1 day tutorial and so might be a little unpredictable.

Having read the book previously I could immediately relate to the slides, and as Kirk continued to enlighten us with his graphical view into the Kernel, many areas of the book that I had read but not necessarily understood, slowly began to make sense. In fact after the first session it became apparent that I had a better idea of how the Kernel worked than I first imagined.

The first section flew by real fast and I got the impression that Kirk might have been regretting the decision to allow questions to be thrown throughout the tutorial. For the delegates however this freeflow questioning was priceless as it was clear that there were a few who had come into the tutorial with no prior knowledge of how Unix/Linux Kernels operate. Even for the initiated, just being able to ask a question and get a detailed response was worth the tutorial fee.

After a brief coffee break it was full speed ahead for next hour or so until lunch, which was surprisingly tasty, and managed to curb the headache I was feeling, along with other cold-like symptoms I had been fighting. Back at the Music Room Randall Stewart popped in to say hello and joke about his session on SCTP where he was having to throw questions out to the delegates to keep them awake (I’m sure it was a joke!)

As everyone returned from lunch, Kirk explained that we had only managed to cover 1 out of 4 of the sessions and asked if we would like to vote for which of the remaining 3 sessions to focus on. All 3 sessions got equal votes, so it was decided that questions be kept to a minimum and Kirk would talk less but still cover all the topics.
By the time we got around to the next coffee break we were back on track and ready for a sprint through Networking and IPC. Speaking of networking, Eric Allman (Sendmail – get it ) dropped in, just in time for the tutorial to finish, and with only 1 minute left for questions!

After the tutorial we were all invited to join the developers for a punt along the river cam joined by food and beer in town, but as I my head was throbbing, I decided to give that a miss on this occasion and head home in search of a dark room to browse the FreeBSD source – or just some sleep.

The tutorial was excellent and if you happen to be curious about the inner workings of FreeBSD, or like me had read the book but needed some more clarity, you should try to catch the next one or buy the course on DVD and learn at your own leisure!

A big shout out to Kirk and all those who made EuroBSDCon happen! Thanks folks!!

After a well needed break, spent mostly on a sun lounger reading Masterminds of Programming (Fantastic book if you truly enjoy your code!), The Art of Lean Software Development (A real eye opener and highly recommended) and The Productive Programmer (Common sense for experienced developers, but an easy enough read to help refresh your memory), I arrived home a few pounds heavier than when I left, eager to release the creativity from my well rested brain.

While on holiday I managed to pay a visit to the gym (one day it rained ), where I stumbled across an ergometer that is quite simply an impressive piece of engineering!. At first glance the rower looks solid and unobtrusive, however the minute you start to row, you can tell that a lot of work had gone into designing this machine. From the super smooth whir of the flywheel, to the instant gratification from the console, I was hooked and just had to have one!.

It took me a while to finally track down the model (I couldn’t find it in the high street), but after trawling Google images for rowers, I finally found myself at the Concept2 website. It came as no surprise that the Concept2 is regarded as the world’s best rowing machine, and after buying mine from their site I have to agree with that statement.

So what’s rowing got to do with Open Source anyway?, well one of the cool things about owning a Concept2 rower is the ability to log and validate your rowing time online. Doing so allows you to take part in the many challenges offered (I’m looking forward to Skeleton Crew) and compare your ranking with other rowers around the world – rewards help motivate you and keep you rowing!.

It was while I was updating my Online Logbook that I began to think about adding my current progress to this site and wondering what the best method of presenting that information would be. Concept2 are working toward a web based API to access stats, so if I wanted to embed my progress in the meantime I would need to parse the current ranking page.

I started off looking at Javascript, as I knew that most browsers support the language and it would not require any server side code. Finding a decent progressbar that had a nice API and worked well in most browsers, however, proved to be tricky.
I remembered that the Open Source Adobe® Flex® Framework has an easy to use progressbar, and with Adobe Air targeting the desktop I could run the widget on the desktop if needed.

It’s been a while since I last played with Flex, but I am still impressed with the simplicity of MXML and how easy it is to get an working app in no time. Within minutes I had a working widget, my only issue was that when I uploaded to the website, the actionscript that parsed the logbook page failed due to a sandbox violation. A common issue that is easily fixed by uploading a cross-domain policy file to the website you want to access. Not something I wanted to request, so I just resorted to mirroring the page on my own site for now – good old Wget and Cron.

Adding the app to the site was no trouble either, with the help of SWFObject only a few lines of html are required to embed the app and handle any cross browser Flash issues. If you want to see just how easy Flex is to work with, check out the tutorial on the wiki for details on how the widget was built.

Of course that’s just a start, there’s loads more features that could be added to the widget and if parameterised / themed it could also be shared with other sites – watch this space!.

While working with Flex / Flash I also stumbled across Zero Clipboard, a cool library that allows you to access the clipboard via Javascript – AuthStor now has 1 click copy, a feature that cut out several steps required to copy a password and ensures that nobody can snoop over your shoulder.

Right back to rowing, I have 976433 meters to go before I make it into the Million Meter Club!.

I know it’s a little late but the usual drive of new year innovation and releases within the Open Source world have been keeping me busy recently.

New years resolutions

1. Package AuthStor
2. Learn another RDBMS – Oracle / MSSQL?
3. Upgrade my local servers to FreeBSD 7.1
4. Pick up the guitar at least once a week
5. Lay down my first audio track in Ableton Live

The year started with my attempt at some resolutions (you know the ones you never keep) of which I am already making good progress on.

First on my list was AuthStor which is working on a basic level at the moment but there is so much more that can be done to make it easier to use. Why has it taken so long you may ask?

Well apart from being the only developer last year, I made several attempts to re-write the app in something like JSP or Zend Framework, partly because there is a lot of activity in those communities but mostly due to the complications faced with packaging around 50 Perl modules that each have their own compile time dependencies – some of which change frequently often breaking the application (no strict API/Interfaces to adhere to).

So after several prototypes why did I decide to stick with Perl?
Well for starters the changes that did break the application often forced me to change the app in a way that was more logical or introduced a more structured design. I also found that JSP and Zend although cool frameworks tended to feel incomplete and introduced complexities elsewhere e.g. containers / authentication / roles. Catalyst is simple and extremely powerful providing both a development environment and production framework that takes little effort to introduce new features quickly. I can start 2009 in the knowledge that Perl and Catalyst are the right language / framework for the job. I just need to work on the packaging now!

The next item on my list is to learn yet another RDBMS. Why? Enlightenment first and foremost, plus I am a big believer in gaining as much knowledge as possible in my chosen field. You’ll never complete the jigsaw if you have several pieces missing, and seeing as I spent a good part of last year getting to know DB2 I decided this year that my focus would by Oracle and MSSQL, thus completing the picture.

With the release of FreeBSD 7.1 there are now so many cool features that I decided to upgrade some of my older servers that have been sitting in the corner running nicely on FreeBSD 6 (they never fail). DTrace and ZFS put a stop to FreeBSD slipping behind OpenSolaris and the ULE scheduler helps improve performance over your average Linux distro, and with ULE 3 due for release in FreeBSD 8.0 (June) you can expect to see further dramatic performance improvements.

Last on my list will be to pick the Les Paul at least once a week!. One of my prezzies in 2008 was a new orange amp! If that does not encourage me to pick up the guitar I don’t know what will!. You never know I might even mix something up with the copy of Live I got for Christmas

I spent the last month taking a break from Open Source, partly due to FreeBSD 7.1 failing to hit the FTP servers as scheduled (any excuse), but mostly due to Santa’s early delivery of Ableton Live. Not really Open Source I may add but extremely cool nonetheless and a much needed break from coding!.

I find that staring too much at code can hamper the creative process (hehe), and as you know Creativity and Innovation go hand in hand with Open Source Software!.

Speaking of which, while I have been busy playing with waveforms and elastic audio, the Open Source world has been all guns blazing with cool releases like MySQL 5.1 GA, Open Solaris 2008.11, VirtualBox 2.1 and WordPress 2.7 to name a few.
You really cannot take a break these days without something big in the world of Open Source happening in your absence.

So the year is nearly at an end which got me thinking, what was the highlight of 2008 in the world of Open Source?.
I have to confess to spending more time than I would have liked this year working with proprietary software, always in combination with Open Source tools of course!. So if I had to highlight anything from 2008 it would be that Open Source is now more pervasive than ever. From your OSX laptop at home to the Linux servers at work, right the way up to the Internet’s biggest sites and services (running FreeBSD ), you’ll find Open Source Software behind every successful venture. In fact it is now so popular even Microsoft could not resist the urge to get in on the game!.

In fact there are so many Companies using Open Source now that proprietary software may actually be the minority! We all know everyone loves an underdog, so maybe propreitary software will be the hit in 2009? Then again with the state of the economy maybe not!

The biggest influence for me this year (both good and bad) had to be Sun Microsystems who played a large part in Open Source adoption in the enterprise and continue to lead the way with their open business model (GlassFish, MySQL, Netbeans, OpenJDK, OpenSolaris and VirtualBox to name just a few of their many contributions).

Although most of the time it seems like they cannot do anything right, one thing is for sure, without the corporate weight of Sun behind the Open Source ethos we may not have made it here so quickly. So while your saving big bucks with your ZFS storage and processing millions of transactions with MySQL, don’t forget to say thank you to Sun next time you are complaining about JavaFX or MySQL 5.1.
You may even want to swap those complaints for contributions (I know complaining is often easier). Thanks Sun!

So if you are one of the few companies yet to Open your code to the world, what are you waiting for? This Christmas give the gift of knowledge and share your achievements instead of fearing the competition and wrapping your code in a black box (don’t worry we won’t laugh at it). You’ll feel much better knowing you helped make the world a better place!

Until next year I wish you all a Very Open Source Christmas!! and as always Happy Coding!

If you are security conscious and nowadays who isn’t?, you may have pondered over ways to make your external SSH connection safer.
Of course there’s always One-time Passwords and Public Keys, but if like me you watch the daily barrage of brute force attacks you may be thinking it’s only a matter of time…

That’s what I was thinking at about the same time an SMS arrived on my mobile….then it hit me….I take my mobile everywhere, if only my server could call me and ask if it is really me trying to log in.

My first instinct was to try and insert a Perl or Python script into the login process. After all Clickatell have several libraries that you can use to send SMS messages. How hard could it be?
Well for starters it didn’t feel right relying on a script that runs within an interpreter that is running within a shell, but how else could it be done? Pluggable Authentication sprang to mind but that’s hard right? Wrong!.

Hitting the FreeBSD handbook I was able to gain a decent understanding of PAM functionality and even managed to find some sample code. Shame the programming section had not yet been written. O’Reilly to the rescue. After reading a few articles on PAM it wasn’t long before I had a working prototype, in fact most of my time was spent deciding upon the right library/s to use.
The end result is a pam module that can be dropped into the authentication chain for any PAM aware application/service – think SSH, FTP, POP3 etc.

Upon successfully entering your Username/Password the pam_codemein module uses libcurl to send an HTTPS request via Clickatell, my preferred SMS gateway, which then forwards the random Code to your mobile before prompting you to enter the same Code at the prompt to gain access. Of course there are issues with using SMS, such as mobile reception and timeouts, but in testing I didn’t notice any.

If you are concerned over the cost of SMS messages, Clickatell are a bulk SMS provider so rates are good, and if you prefer to be contacted via some other method the code can be adapted to any other service e.g. XMPP via TLS, you could even GPG encrypt the Code in an email if you prefer.

It was only after I had the fun of writing this module that I stumbled across this blog post. Shame as it may have saved me a lot of time, but hey where’s the fun in that? I’ll just have to get cracking on the XMPP method – either that or search deeper within Google.

Have you seen RT 3.8!

It’s been a long time coming but after 2 years in the making the enterprise-grade ticketing system that is RT has had a well earned facelift, and what a difference!!.

Upgrading from 3.6.3 would have been straightforward if it was not for the fact that I also decided to upgrade to MySQL 5.1 at the same time. The final product made the pitfalls worthwhile however and RT is now slicker than ever. Plus I got know RT better along the way – bonus!.

If your thinking of upgrading to RT 3.8 / MySQL 5.1 just remember to use the MyISAM engine for the Attachments table (It’s worked so far for me with around 84k tickets). Failing to do so will result in slow response from the UI on certain actions due to MySQL failing to select the correct index on certain queries:

ALTER TABLE Attachments ENGINE=MyISAM;

It looks like the optimiser is broken for the InnoDB plugin, however if you check out this bug it may have been fixed by now (I was using FreeBSD port: mysql-server-5.1.26).

If only your database would just tell you that replication had failed or that the disk was full…
Ok some database servers do but MySQL doesn’t (yet). Another excuse to write a script

In my pursuit for total database visibility I have been searching for a tool that would tell me when something went wrong, that would simulate regular usage, and let me know if anything failed. After all SNMP can only probe so far and if your database is secure the last thing you want to do is open up another port on your server.

Don’t get me wrong there are tools out there that do a decent job of monitoring MySQL. MySQL Entperprise Monitor is one such tool. But if you want to look under the hood or add some functionality specific to your environment things start to get tricky.

So what’s this dbAlerter?

dbAlerter is the name of a project (Script) that I started developing to provide notification of key server events. It’s written in Python; Why?, I wanted something lightweight. Java would have been my first choice but the JVM eats too much ram and felt too heavy for such simple functionality.
Python on the other hand feels lighter and when running a script as a daemon it seems to take up less resources allowing the DB to use more memory and continue doing it’s job. Plus it’s been a while since I wrote anything with Python, variety is the spice of life!

Speaking of Java, if you have ever used tools like JConsole and jmap (and if not why not!) you may not have realised that since JDK 6 Update 7 there’s now one tool to rule them all – VisualVM.

VisualVM pulls together several commandline tools like JConsole, jstat, jinfo, jstack, and jmap to make monitoring and profiling a more visual experience.

Now that the changes from BSD Java have been merged into the OpenJDK and the FreeBSD Foundation have updated the JDK to 1.6.0-7 I’m pretty sure the original concept of write once run anywhere is closer than ever.

Databases are used by many different disciplines, from your DBA to your Developer, right down to the End User. They can also be viewed from many different perspectives…

DBA’s are usually interested in the server configuration, index usage etc. Developers however are often concerned with retrieval methods, datatypes and occasionally stored procedures. One thing is for sure, your database if used effectively will be changing frequently, and with so many people accessing your database do you know what the last change was? Would you know if someone added an index to a table? (would you care… )

Revision Control is often used to track changes to source code and other documents, so why not your database?.

When I first started to think about change control for my database servers one tool sprung to mind, Rancid. If you manage more than one network device you may already be familiar with the Really Awesome New Cisco confIg Differ.

Using a combination of scripts Rancid will connect to your network devices, check for changes by comparing the config with a VCS managed copy and notify you with details of the changes. Why not do the same for your database? Enter dbRecorder.

Rather than extend Rancid I decided to prove the concept by writing a quick Java application that utilised Open Source libraries like SVNKit and jYAML. The features continued to creep and before I knew it I was recording more than just table definitions. Server configuration and grant tables all found their way into my Subversion repository coupled with e-mail notification whenever anything changed – nice huh!.

The aptly named dbRecorder is currently targeted at my database of choice, MySQL. It’s far from a finished product (then again so are most of my ideas) but it’s working to a degree and can only improve over time….

One thing I never got around to was making this site work well with IE6 (or even just splash the screen with some silly message when someone using IE6 tries to connect) and with Firefox 3 soon to be released I am hoping I won’t have to. If you have not tried Firefox 3 already you can visit the site today and pledge your download to help break a world record as soon as it is released!.

It’s been a while since I had chance to work on Informeer as my itch was one of multi-user web based password management (AuthStor). Oh and moving house.

Now that things are settling down again (Servers back up and running) I decided to take a break from AuthStor and focus on something new – Informeer.

The concept is simple, Centralised Notification.

I am forever configuring notifications from several sources, be it backup alerts, host monitoring notification and even simple applications that send mail via SMTP. When living in a world of change, both software and business, having to visit every application to change an e-mail address or add a new user to a notification schedule can be quite time consuming. Add to that the effort of having to modify firewalls, SMTP servers and XMPP settings etc etc. The idea of a single web interface where all your notifications can be configured is quite appealing. Informeer aims to make that a reality (one day).

While I take my time with the implementation I thought I would post a basic intro to RPC-XML using Perl. The RPC-XML Perl module makes writing your own client/server application a piece of cake, and while not quite point and click, it will give you enough flexibility to centralise your own notifications (or anything else for that matter). It’s not rocket science, but it works!

If you have been looking for a flexible quick and easy fix to the centralised notification problem, or even if you are just looking to push some data from a firewalled site to one of your servers This Tutorial should help get you started.

In between moving house and playing with notifications I found time to upgrade to WordPress 2.5 and MoinMoin 1.6, and wow what an improvement in both camps.
The WordPress dashboard is the most visible improvement – it’s amazingly clean! If you have been waiting for 2.5 to settle down or just cautious about upgrading, I would say take the plunge, you won’t regret it. As for MoinMoin 1.6, there really is no better Wiki engine out there (my opinion) and it just keeps getting better with every release!.

That just leaves me to post a quick MySQL tip that may well be obvious, but happens to catch me out from time to time.

MySQL Tip of the month

If you ever get the urge to convert a 1Gig MyISAM table (with 34 million rows) to InnoDB within MySQL, try to avoid the simple ALTER TABLE tablename ENGINE=INNODB; method.
I made the mistake of running that command on a fairly decent server with bags of space and memory only to find the command running over 5 hours later with an InnoDB tablespace at around 4Gig (ok a bit of tuning might have helped)

Your best bet is to dump the table using mysqldump, drop the existing table and re-import the table after modifying the definition to be InnoDB. That worked for me in less than 10 minutes. Alternatively you can create a new InnoDB table and insert directly from the existing table as per MySQL documentation e.g INSERT INTO newtable SELECT * FROM oldtable.

I’m sure I am not the first to make this mistake, and may not be the last…..

AuthStor came to life back in late 2007 after the Password Safe I had been using became corrupt. Not through any failure in Password Safe’s functionality but really to the way it was being used – Multi-User.

I had been seeking a decent Web Based Password Manager (Open Source of course) for ages but had not found anything that I liked. All the semi-decent software out there seemed to be Closed Source with license fees attached and to be honest I would feel more comfortable if I understood the inner workings, especially if this software is storing the key to my empire (mwuhahahaha).

So after a long history of Spreadsheets, Password Safe, Keychain and various other methods of storing passwords I finally decided to scratch the itch and roll my own password manager.

Now AuthStor is not my first attempt at a password manager, in fact I originally started to write an extension to MoinMoin that provided a method of storing passwords directly within the Wiki. This worked great for a while but was not the most secure method of managing passwords, especially as the secret key was embedded within the Python code!. I learned a lot from mmPasswords and hope to improve upon that greatly through the development of AuthStor.

So what is in store for AuthStor? (Get it? ) Well key features will include Multi-User capability, Full auditing, GnuPG encryption, REST like interface and that is just the start. Future plans include interfaces like SMS, VoIP and anything else I can think up.

Here is a sneak peak of the progress made so far:

The application itself is written in Perl and runs on top of the Catalyst Framework. Why Catalyst? Flexibility, Performance, Scalability, the list goes on… Catalyst’s plugin architecture provides multiple views (among many other things) so I can focus on the functionality and not worry about the presentation. Speaking of presentation, AuthStor makes extensive use of the Yahoo! User Interface Library to provide a rich interface with dynamic elements such as DataTable’s, Toolbars and Treeviews and all wrapped in a BSD License!!. Nice work folks!.

Expect a commit in the next few weeks…

It’s been a few weeks since I last posted here or even found time to develop any of the ideas that continually spring to mind. In fact this is the first weekend in several where my head is not stuck in a book about Perl, Solaris or UML. Why you may ask? (or not).

Well after finally getting round to taking the Solaris Network Administrator exam, I thought it only right that I complete the set and get to know Solaris Security a bit better by taking the SCSECA.

After achieving the SCSECA, I was just about getting ready to start writing code again when I stumbled upon the UML Certification Program.
I have been a fan of UML for a long time but never found a tool that I felt comfortable using, or more to the point a tool worth the money!. That was until on the very same day I decided to trial Visual Paradigm, could this be the UML tool I have been seeking?

Visual Paradigm for UML is not overly expensive (many price levels to cater for all requirements), it runs on Windows / Linux / OSX and it’s easy to use – I like it!.

Armed with the UML tool I have been looking for and a copy of the UML 2 Certification Guide I spent another week brushing up on my UML skills while studying for the first of 3 exams offered by the OMG, UML Certified Professional – Fundamental.
I passed the exam last week and learned a lot more UML along the way!
So finally, with all that studying out of the way I can finally get back to the fun stuff – Code.

But where does Perl fall into this jacked-in period? Well I have been waiting for Perl 6 for a long time now and plan to use Catalyst as the platform for BinaryStor and AuthStor. So what better way to get up to speed than a Perl readathon of as many Perl books as I can get my hands on. From Learning to Mastering and eventually Best Practices all over again, you never know I might just finish reading before the release of Perl 6.

As for the Code I cannot make my mind up as to where to focus my energy and so opted for a working NetUnit (I did think of porting it to Perl ). It’s far from complete of course and suffers from the constraint of root privileges, but I’m hoping a future Java release will fix that….