Ever since Sun Microsystems agreed to acquire MySQL back in 2008, there has been a fair bit of uncertainty and chaos surrounding the world’s most popular Open Source database. With many big names in the MySQL community pulling in different directions and the recent Oracle / Sun acquisition, the choice of which Open Source database to use is now easier than ever - PostgreSQL.

One of MySQL’s key strengths has always been its simplicity to both install, manage and code against, resulting in mass adoption over time, which in turn forced the once web bound database to evolve into a near fully fledged enterprise RDBMS.

PostgreSQL on the other hand was born an RDBMS and instead worked silently in the background improving the performance, reliability and feature list of the world’s most advanced Open Source database, and all under the lesser restrictive BSD license - nice work!.

Getting up to speed with PostgreSQL can take a little longer for MySQL veterans, mostly due to the many subtle differences, but more importantly the extensive feature set (one of the reasons it is taking longer to add PostgreSQL support to dbRecorder ) but the rewards are worth it, as companies such as Skype can attest to.

It was installing OpenNMS several years ago that convinced me to get to grips with PostgreSQL, and although there is a distinct lack of new reading material out there the manual covers everything you need to start building your next web app.

The GUI tool pgAdmin has also come a long way and provides a great interface to PostgreSQL, with excellent instructional prompts and support for all major platforms, there really is no better tool to help you work with PostgreSQL while getting to grips with the vast feature set. Check out the latest beta over at the pgAdmin site.

If you’re not yet excited about PostgreSQL, maybe you should check out the feature list planned for release 8.4 (currently at beta2), with cool features such as Column Level Permissions and SSL Connections you can feel less stressed about the security of your data, oh and the performance improvements won’t hurt either!.

While keeping with the BSD license, another notable release last month was FreeBSD 7.2, a great platform to complement the reliability of PostgreSQL and another way to gain some performance improvements.
Speaking of performance I was pleased to read that Jails are much improved in the latest release, and that it seems I am not the only one that prefers Jails / Zones over VM solutions!.

I’m off to make some progress with PostgreSQLDump.java, the worker class that will allow me to add PostgreSQL support to dbRecorder.
Watch this space…..

There was a time when Novell ruled the server…a time when CNE was a mark of technical excellence and server uptime was measured in years not weeks…a time and when Novell Directory Services (NDS) provided a single point of administration for every aspect of the enterprise environment, including the then evolving Active Directory!.

Yet just when everything was coming together for Novell the attraction of owning the desktop led them to acquire Ximian, the coolest Linux desktop on the block, before joining forces with IBM and acquiring SuSE - possibly the biggest competitor to RedHat at the time.

It was not long after the SuSE acquisition that Novell announced their intention to phase out NetWare, a move that saw many IT Managers faced with the decision of migrating their servers to the then largely untested OES (Open Enterprise Server) or crumbling under the ever increasing pressure of Active Directory and Exchange. Needless to say not many survived to log in via the Novell Client…

I personally focused my energy on FreeBSD and other OpenSource solutions, occasionally rolling out OES to companies that required enterprise support, secure file synchronisation (iFolder) or a rock solid collaboration platform (GroupWise). Novell have always had sensible license fees that were often based on the user model and not server providing massive cost savings.

It was cost savings that I was looking for recently while searching for a cross platform LDAP solution that was easy to manage, secure, reliable and had enterprise support to boot. It was exactly those requirements that led me back to Novell, back to eDirectory, which after all this time continue to offer the Full Service Directory promise that ruled the enterprise several years ago.

Installation of eDirectory on SuSE is seamless, RedHat however required a few dependencies (yum install) but still took under 10 minutes, and that included bolting on iManager the secure web interface to many Novell products (think Console One). Configuring the RedHat clients was just as easy, in fact using the authconfig tool you can configure the client to talk to eDirectory with a single command:

authconfig --enableldap --enableldapauth --enableldaptls --ldapserver=myserver --ldapbasedn=o=myorganisation --enablemkhomedir --updateall

As for adding a replica for resilience, well you only need to install eDirectory onto another server on your network (yes it runs on Windows too!), and as long as you select your existing tree during the install process you have a secure read/write replica - simple eh!.

So it seems Novell never gave up. They focused their energy into migrating all the great tools we took for granted with NetWare while combining the best of breed Linux tools into SUSE Linux Enterprise 11 - Novell’s latest enterprise server platform.

With features such as .NET support for System z mainframes, OCFS2 / cLVM2 and new fine-grained management of CPU, memory, storage and networking resources, it looks like Novell have a worthy contender to the enterprise Linux throne!.
Could Novell be ready to pick up where they left off with NetWare?, only time will tell….now if only eDirectory ran on FreeBSD.

I know it’s a little late but the usual drive of new year innovation and releases within the Open Source world have been keeping me busy recently.

New years resolutions

1. Package AuthStor
2. Learn another RDBMS - Oracle / MSSQL?
3. Upgrade my local servers to FreeBSD 7.1
4. Pick up the guitar at least once a week
5. Lay down my first audio track in Ableton Live

The year started with my attempt at some resolutions (you know the ones you never keep) of which I am already making good progress on.

First on my list was AuthStor which is working on a basic level at the moment but there is so much more that can be done to make it easier to use. Why has it taken so long you may ask?

Well apart from being the only developer last year, I made several attempts to re-write the app in something like JSP or Zend Framework, partly because there is a lot of activity in those communities but mostly due to the complications faced with packaging around 50 Perl modules that each have their own compile time dependencies - some of which change frequently often breaking the application (no strict API/Interfaces to adhere to).

So after several prototypes why did I decide to stick with Perl?
Well for starters the changes that did break the application often forced me to change the app in a way that was more logical or introduced a more structured design. I also found that JSP and Zend although cool frameworks tended to feel incomplete and introduced complexities elsewhere e.g. containers / authentication / roles. Catalyst is simple and extremely powerful providing both a development environment and production framework that takes little effort to introduce new features quickly. I can start 2009 in the knowledge that Perl and Catalyst are the right language / framework for the job. I just need to work on the packaging now!

The next item on my list is to learn yet another RDBMS. Why? Enlightenment first and foremost, plus I am a big believer in gaining as much knowledge as possible in my chosen field. You’ll never complete the jigsaw if you have several pieces missing, and seeing as I spent a good part of last year getting to know DB2 I decided this year that my focus would by Oracle and MSSQL, thus completing the picture.

With the release of FreeBSD 7.1 there are now so many cool features that I decided to upgrade some of my older servers that have been sitting in the corner running nicely on FreeBSD 6 (they never fail). DTrace and ZFS put a stop to FreeBSD slipping behind OpenSolaris and the ULE scheduler helps improve performance over your average Linux distro, and with ULE 3 due for release in FreeBSD 8.0 (June) you can expect to see further dramatic performance improvements.

Last on my list will be to pick the Les Paul at least once a week!. One of my prezzies in 2008 was a new orange amp! If that does not encourage me to pick up the guitar I don’t know what will!. You never know I might even mix something up with the copy of Live I got for Christmas

I spent the last month taking a break from Open Source, partly due to FreeBSD 7.1 failing to hit the FTP servers as scheduled (any excuse), but mostly due to Santa’s early delivery of Ableton Live. Not really Open Source I may add but extremely cool nonetheless and a much needed break from coding!.

I find that staring too much at code can hamper the creative process (hehe), and as you know Creativity and Innovation go hand in hand with Open Source Software!.

Speaking of which, while I have been busy playing with waveforms and elastic audio, the Open Source world has been all guns blazing with cool releases like MySQL 5.1 GA, Open Solaris 2008.11, VirtualBox 2.1 and Wordpress 2.7 to name a few.
You really cannot take a break these days without something big in the world of Open Source happening in your absence.

So the year is nearly at an end which got me thinking, what was the highlight of 2008 in the world of Open Source?.
I have to confess to spending more time than I would have liked this year working with proprietary software, always in combination with Open Source tools of course!. So if I had to highlight anything from 2008 it would be that Open Source is now more pervasive than ever. From your OSX laptop at home to the Linux servers at work, right the way up to the Internet’s biggest sites and services (running FreeBSD ), you’ll find Open Source Software behind every successful venture. In fact it is now so popular even Microsoft could not resist the urge to get in on the game!.

In fact there are so many Companies using Open Source now that proprietary software may actually be the minority! We all know everyone loves an underdog, so maybe propreitary software will be the hit in 2009? Then again with the state of the economy maybe not!

The biggest influence for me this year (both good and bad) had to be Sun Microsystems who played a large part in Open Source adoption in the enterprise and continue to lead the way with their open business model (GlassFish, MySQL, Netbeans, OpenJDK, OpenSolaris and VirtualBox to name just a few of their many contributions).

Although most of the time it seems like they cannot do anything right, one thing is for sure, without the corporate weight of Sun behind the Open Source ethos we may not have made it here so quickly. So while your saving big bucks with your ZFS storage and processing millions of transactions with MySQL, don’t forget to say thank you to Sun next time you are complaining about JavaFX or MySQL 5.1.
You may even want to swap those complaints for contributions (I know complaining is often easier). Thanks Sun!

So if you are one of the few companies yet to Open your code to the world, what are you waiting for? This Christmas give the gift of knowledge and share your achievements instead of fearing the competition and wrapping your code in a black box (don’t worry we won’t laugh at it). You’ll feel much better knowing you helped make the world a better place!

Until next year I wish you all a Very Open Source Christmas!! and as always Happy Coding!

Happy Halloween! - well by the time you read this it will be more - Remember, remember the 5th of November!. So what’s been happening?

Well I spent the last month getting to grips with DB2 - why I hear you ask?.
You shouldn’t keep all your eggs in one basket, and there were rumors that DB2 might become Open Source at some point (or not). Anyway even if that never happens there is a lot to learn from the original DBMS and with DB2 Express-C available for free there’s nothing to stop your Open Source app taking advantage of that fact. Besides competition is a good thing.

Anyway not forgetting my favorite DBMS, I managed to find time to spend a morning at the 4th MySQL Conference in London where the keynote opened to - here comes the sun - cute. There was a lot of emphasis on the fact that MySQL was not being written to run faster on Solaris (even though I got the impression that it does, with features like ZFS and snappy 64bit libraries outlined during the second session).

It was a relief to hear that MySQL 5.1 will finally go GA during the first week in December along with the new query analyzer (aka proxy). I couldn’t help get the feeling of déjà vu out of my head, after all it was only last October that I remember seeing some very similar slides - anyway fingers crossed for a December release!.

My only disappointment at the conference was hearing that the auditing / security features would not make an appearance until after 2010? (TBD), frustrating especially with items like PCI compliance high on everyones agenda. Still with a little ingenuity and help from the MySQL Proxy Server you can roll your own auditing solution and customise it to your own needs (is there anything proxy can’t do ).

Speaking of which it looks like the Proxy Server will finally hit GA early next year in the form of the MySQL Load Balancer. In fact it is planned to make an appearance in December in the form of the Query Analyzer - yes that’s based on the Proxy too!. Time to brush up on your Lua and check out the samples over at the Forge.

The big news this October/November goes to the G1, the first phone to take advantage of Android: a complete, open, and free mobile platform.

My own G1 adventure started last Thursday with a call to T-Mobile who were keen to dispatch my phone next day, but not so keen to deliver it to my workplace. Not a problem, I could wait till the weekend even if it meant paying £6 for delivery.

Saturday morning came and went with no sign of a Royal Mail van and after a 20 minute call to T-Mobile it seemed I would not be getting my G1 anytime soon. I listened to several explanations ranging from the order not going through to high demand exhausting stock. No offer of apology or even a discount of any kind!.

Not one to accept lame excuses I asked for the order to be cancelled and fought for another 15 minutes with a customer service rep who claimed the order could not be cancelled over the phone. His manager however had no problem canceling the order - funny that eh. A quick Google search confirmed my thoughts!.

My local T-Mobile shop was not much help either claiming that the plan I had accepted on Thursday no longer existed on Saturday due to it now being November. They did however have the G1 in stock, well they did until I decided to purchase it, then it seemed Monday was the earliest I could get one. I picked mine up on Sunday from Bluewater and even got the plan that no longer existed this month…

The fun didn’t end there though as it was about 5 minutes after opening the phone that I managed to lock the SIM while trying to set-up the password protection - PUK locked please contact customer services - do I have to I thought!. 30 minutes later the customer services rep had no idea how we could unlock the phone and instead decided to give me my PUK code along with the phone number for HTC’s customer services (it’s their problem not T-Mobile’s) - which as you may have guessed already happened to be closed at the weekend. Maybe Google could help!.

Another quick search and it seemed I was not the only one with this problem and the only answer so far was to place the sim in another phone to enter the PUK code and unlock the SIM. Simple when you know how! (and you have a spare mobile to hand). Anyway my phone is now unlocked and I’m ready to start exploring Android. I did try logging into the My T-Mobile site to register my phone etc however the link failed - no surprises there.

As with most Open Source projects it is the community that provides the support and with the G1 / Android this is no exception. If you want to learn more about Android on the G1 don’t waste your time contacting T-Mobile, instead head on over to the website or forum and marvel at what is no doubt going to be THE mobile platform of 2009. All I need to do now is think up an idea for an app…..

If you are security conscious and nowadays who isn’t?, you may have pondered over ways to make your external SSH connection safer.
Of course there’s always One-time Passwords and Public Keys, but if like me you watch the daily barrage of brute force attacks you may be thinking it’s only a matter of time…

That’s what I was thinking at about the same time an SMS arrived on my mobile….then it hit me….I take my mobile everywhere, if only my server could call me and ask if it is really me trying to log in.

My first instinct was to try and insert a Perl or Python script into the login process. After all Clickatell have several libraries that you can use to send SMS messages. How hard could it be?
Well for starters it didn’t feel right relying on a script that runs within an interpreter that is running within a shell, but how else could it be done? Pluggable Authentication sprang to mind but that’s hard right? Wrong!.

Hitting the FreeBSD handbook I was able to gain a decent understanding of PAM functionality and even managed to find some sample code. Shame the programming section had not yet been written. O’Reilly to the rescue. After reading a few articles on PAM it wasn’t long before I had a working prototype, in fact most of my time was spent deciding upon the right library/s to use.
The end result is a pam module that can be dropped into the authentication chain for any PAM aware application/service - think SSH, FTP, POP3 etc.

Upon successfully entering your Username/Password the pam_codemein module uses libcurl to send an HTTPS request via Clickatell, my preferred SMS gateway, which then forwards the random Code to your mobile before prompting you to enter the same Code at the prompt to gain access. Of course there are issues with using SMS, such as mobile reception and timeouts, but in testing I didn’t notice any.

If you are concerned over the cost of SMS messages, Clickatell are a bulk SMS provider so rates are good, and if you prefer to be contacted via some other method the code can be adapted to any other service e.g. XMPP via TLS, you could even GPG encrypt the Code in an email if you prefer.

It was only after I had the fun of writing this module that I stumbled across this blog post. Shame as it may have saved me a lot of time, but hey where’s the fun in that? I’ll just have to get cracking on the XMPP method - either that or search deeper within Google.

Have you seen RT 3.8!

It’s been a long time coming but after 2 years in the making the enterprise-grade ticketing system that is RT has had a well earned facelift, and what a difference!!.

Upgrading from 3.6.3 would have been straightforward if it was not for the fact that I also decided to upgrade to MySQL 5.1 at the same time. The final product made the pitfalls worthwhile however and RT is now slicker than ever. Plus I got know RT better along the way - bonus!.

If your thinking of upgrading to RT 3.8 / MySQL 5.1 just remember to use the MyISAM engine for the Attachments table (It’s worked so far for me with around 84k tickets). Failing to do so will result in slow response from the UI on certain actions due to MySQL failing to select the correct index on certain queries:

ALTER TABLE Attachments ENGINE=MyISAM;

It looks like the optimiser is broken for the InnoDB plugin, however if you check out this bug it may have been fixed by now (I was using FreeBSD port: mysql-server-5.1.26).

If only your database would just tell you that replication had failed or that the disk was full…
Ok some database servers do but MySQL doesn’t (yet). Another excuse to write a script

In my pursuit for total database visibility I have been searching for a tool that would tell me when something went wrong, that would simulate regular usage, and let me know if anything failed. After all SNMP can only probe so far and if your database is secure the last thing you want to do is open up another port on your server.

Don’t get me wrong there are tools out there that do a decent job of monitoring MySQL. MySQL Entperprise Monitor is one such tool. But if you want to look under the hood or add some functionality specific to your environment things start to get tricky.

So what’s this dbAlerter?

dbAlerter is the name of a project (Script) that I started developing to provide notification of key server events. It’s written in Python; Why?, I wanted something lightweight. Java would have been my first choice but the JVM eats too much ram and felt too heavy for such simple functionality.
Python on the other hand feels lighter and when running a script as a daemon it seems to take up less resources allowing the DB to use more memory and continue doing it’s job. Plus it’s been a while since I wrote anything with Python, variety is the spice of life!

Speaking of Java, if you have ever used tools like JConsole and jmap (and if not why not!) you may not have realised that since JDK 6 Update 7 there’s now one tool to rule them all - VisualVM.

VisualVM pulls together several commandline tools like JConsole, jstat, jinfo, jstack, and jmap to make monitoring and profiling a more visual experience.

Now that the changes from BSD Java have been merged into the OpenJDK and the FreeBSD Foundation have updated the JDK to 1.6.0-7 I’m pretty sure the original concept of write once run anywhere is closer than ever.

Databases are used by many different disciplines, from your DBA to your Developer, right down to the End User. They can also be viewed from many different perspectives…

DBA’s are usually interested in the server configuration, index usage etc. Developers however are often concerned with retrieval methods, datatypes and occasionally stored procedures. One thing is for sure, your database if used effectively will be changing frequently, and with so many people accessing your database do you know what the last change was? Would you know if someone added an index to a table? (would you care… )

Revision Control is often used to track changes to source code and other documents, so why not your database?.

When I first started to think about change control for my database servers one tool sprung to mind, Rancid. If you manage more than one network device you may already be familiar with the Really Awesome New Cisco confIg Differ.

Using a combination of scripts Rancid will connect to your network devices, check for changes by comparing the config with a VCS managed copy and notify you with details of the changes. Why not do the same for your database? Enter dbRecorder.

Rather than extend Rancid I decided to prove the concept by writing a quick Java application that utilised Open Source libraries like SVNKit and jYAML. The features continued to creep and before I knew it I was recording more than just table definitions. Server configuration and grant tables all found their way into my Subversion repository coupled with e-mail notification whenever anything changed - nice huh!.

The aptly named dbRecorder is currently targeted at my database of choice, MySQL. It’s far from a finished product (then again so are most of my ideas) but it’s working to a degree and can only improve over time….

One thing I never got around to was making this site work well with IE6 (or even just splash the screen with some silly message when someone using IE6 tries to connect) and with Firefox 3 soon to be released I am hoping I won’t have to. If you have not tried Firefox 3 already you can visit the site today and pledge your download to help break a world record as soon as it is released!.

It’s been a busy month so far and with Easter now upon us the year is showing no sign of deceleration.

I spent the last few weeks working hard to recover from my initial attempt at passing the Zend Certified Engineer exam. Having used PHP and other languages for several years I grabbed a copy of the Php|architect’s Zend PHP 5 Certification Study Guide, bought a few online practice tests from Zend and spent a few days scoring Excellent before heading down to the test centre confident I would be leaving with a Pass.
Forty five minutes into the exam I was racing through the questions thinking I had it in the bag only to be presented with a Fail upon clicking the End button…..Bummer

The score report gave no real indication of how close I came to passing, so wondering where I went wrong I scoured the net for any information that could help me prepare for my next attempt (which you get a 10% discount on if you ask Zend nicely). During my search I found the Zend Certification Forum, however it was broken and I had to ask for it to be fixed (which took a couple of days) before I could read others views on the exam. The Zend forum and site I have to say are not good adverts for PHP. The site often takes ages to load (is this down to PHP? people will think) and the forum feels pre web 2.0.

After a few hours of searching I finally found this PDF, the PHP 5 course material that outlines everything you need to focus on for the exam and provides enough examples to to get your head around the “gotchas” that are hidden in many of the questions. Second time round I passed, although many questions were similar to my first attempt and I answered them no different, I can only put my initial fail down to the PHP “gotchas” like:

What is the output of echo 09;

Post certification I had to ask for the Yellow Pages entry to be added (a common problem it seems), and after finally getting the page updated it seems to have reverted to its former state…..If only it was written in Java I keep thinking to myself.
Don’t get me wrong I am a big fan of PHP but feel its place is within the presentation layer (this site is written using PHP). When it comes down to middle tier processing I am yet to be convinced…

Speaking of Java, I decided that along with moving house this Easter (I need a server room ), I would open my mind to the world of software that is Oracle. The Oracle Database is hardly Open Source, however Oracle as a company do support the Open Source Community and to be honest I’m a knowledge junkie and need something fresh to feed my mind with (fear not MySQL).
Besides at the end of the day everything I learn from taking apart Oracle can be directly applied to the Open Source world, that is if it has not been applied already.

You may have noticed the Easter Bunny pic included with this post and if you are into Open Source you would have recognised him as Big Buck Bunny the star of the upcoming open movie Peach. That’s right there are only a few more weeks to the release of the movie and you can now take a look at the trailer before downloading your copy of Blender and having a go yourself - I can’t wait!

I spend the majority of my days working with OpenSource Unix derivatives and have to say I enjoy every minute of it! Not a day goes by where I do not learn something new or discover a new way to solve a problem all thanks to Unics (Everything has a beginning).

If you are passionate about GNU / Unix / Linux / OpenSource I apologise for the generalization “Unix” but wanted to focus on the OS and tools that make moving between one Unix variant and another as easy as pie.
Putting aside all the nuances that distinguish the many flavors of Unix, once you gain a core understanding of the Community, Tools and Kernels, you find that moving from one OS flavor to another is as simple as uname and man.

Take for instance my latest branch off the Unix tree - HP-UX. Having minimal exposure to HP-UX I wanted to get up to speed quick as possible, so I grabbed a copy of the HP-UX Study Guide and booked the CSA exam knowing that if I had a goal I would not just move onto the next flavor.

The book was a great resource and I only wished I had read it sooner as it covered many tools and concepts that are commonplace in SVR4 Unix derivatives such as Solaris. I felt like I was stepping back 10 years to when I first started to discover basic utilities such as cat, sed and awk but explained in so much detail that I found myself learning some new tricks!. Even if you are not interested in HP-UX this book is so well written and easy to condense that it is worth a place on your Unix bookshelf!.

As for HP-UX, well I felt right at home and only had to spend time learning the key differences such as Software Management and HP’s Partitioning Continuum technology (Virtualization to you and me) - very cool!.

Would I run HP-UX on my own servers? Probably not, it’s not OpenSource for starters (plus I don’t have the hardware), and to be honest I have been running FreeBSD for over 10 years now and no matter how many Unix/Linux derivatives I encounter not one has been able to surpass FreeBSD’s community, stability and manageability. Plus FreeBSD is Free in every sense!.

Speaking of FreeBSD you might have noticed the image in the upper left corner of this post - yes you guessed it; FreeBSD 7.0 has been released!.
I have been running a release candidate for a while now and have to say that the one thing you will notice is how fast it runs. Don’t take my word for that either, check out the Benchmarks and read through the new features explained in detail over at the BSD Devcenter. With performance like that why would you want to run any other OS?

Download your copy from here and never look back.