If only your database would just tell you that replication had failed or that the disk was full…
Ok some database servers do but MySQL doesn’t (yet). Another excuse to write a script

In my pursuit for total database visibility I have been searching for a tool that would tell me when something went wrong, that would simulate regular usage, and let me know if anything failed. After all SNMP can only probe so far and if your database is secure the last thing you want to do is open up another port on your server.

Don’t get me wrong there are tools out there that do a decent job of monitoring MySQL. MySQL Entperprise Monitor is one such tool. But if you want to look under the hood or add some functionality specific to your environment things start to get tricky.

So what’s this dbAlerter?

dbAlerter is the name of a project (Script) that I started developing to provide notification of key server events. It’s written in Python; Why?, I wanted something lightweight. Java would have been my first choice but the JVM eats too much ram and felt too heavy for such simple functionality.
Python on the other hand feels lighter and when running a script as a daemon it seems to take up less resources allowing the DB to use more memory and continue doing it’s job. Plus it’s been a while since I wrote anything with Python, variety is the spice of life!

Speaking of Java, if you have ever used tools like JConsole and jmap (and if not why not!) you may not have realised that since JDK 6 Update 7 there’s now one tool to rule them all - VisualVM.

VisualVM pulls together several commandline tools like JConsole, jstat, jinfo, jstack, and jmap to make monitoring and profiling a more visual experience.

Now that the changes from BSD Java have been merged into the OpenJDK and the FreeBSD Foundation have updated the JDK to 1.6.0-7 I’m pretty sure the original concept of write once run anywhere is closer than ever.

Databases are used by many different disciplines, from your DBA to your Developer, right down to the End User. They can also be viewed from many different perspectives…

DBA’s are usually interested in the server configuration, index usage etc. Developers however are often concerned with retrieval methods, datatypes and occasionally stored procedures. One thing is for sure, your database if used effectively will be changing frequently, and with so many people accessing your database do you know what the last change was? Would you know if someone added an index to a table? (would you care… )

Revision Control is often used to track changes to source code and other documents, so why not your database?.

When I first started to think about change control for my database servers one tool sprung to mind, Rancid. If you manage more than one network device you may already be familiar with the Really Awesome New Cisco confIg Differ.

Using a combination of scripts Rancid will connect to your network devices, check for changes by comparing the config with a VCS managed copy and notify you with details of the changes. Why not do the same for your database? Enter dbRecorder.

Rather than extend Rancid I decided to prove the concept by writing a quick Java application that utilised Open Source libraries like SVNKit and jYAML. The features continued to creep and before I knew it I was recording more than just table definitions. Server configuration and grant tables all found their way into my Subversion repository coupled with e-mail notification whenever anything changed - nice huh!.

The aptly named dbRecorder is currently targeted at my database of choice, MySQL. It’s far from a finished product (then again so are most of my ideas) but it’s working to a degree and can only improve over time….

One thing I never got around to was making this site work well with IE6 (or even just splash the screen with some silly message when someone using IE6 tries to connect) and with Firefox 3 soon to be released I am hoping I won’t have to. If you have not tried Firefox 3 already you can visit the site today and pledge your download to help break a world record as soon as it is released!.

It’s been a while since I had chance to work on Informeer as my itch was one of multi-user web based password management (AuthStor). Oh and moving house.

Now that things are settling down again (Servers back up and running) I decided to take a break from AuthStor and focus on something new - Informeer.

The concept is simple, Centralised Notification.

I am forever configuring notifications from several sources, be it backup alerts, host monitoring notification and even simple applications that send mail via SMTP. When living in a world of change, both software and business, having to visit every application to change an e-mail address or add a new user to a notification schedule can be quite time consuming. Add to that the effort of having to modify firewalls, SMTP servers and XMPP settings etc etc. The idea of a single web interface where all your notifications can be configured is quite appealing. Informeer aims to make that a reality (one day).

While I take my time with the implementation I thought I would post a basic intro to RPC-XML using Perl. The RPC-XML Perl module makes writing your own client/server application a piece of cake, and while not quite point and click, it will give you enough flexibility to centralise your own notifications (or anything else for that matter). It’s not rocket science, but it works!

If you have been looking for a flexible quick and easy fix to the centralised notification problem, or even if you are just looking to push some data from a firewalled site to one of your servers This Tutorial should help get you started.

In between moving house and playing with notifications I found time to upgrade to Wordpress 2.5 and MoinMoin 1.6, and wow what an improvement in both camps.
The Wordpress dashboard is the most visible improvement - it’s amazingly clean! If you have been waiting for 2.5 to settle down or just cautious about upgrading, I would say take the plunge, you won’t regret it. As for MoinMoin 1.6, there really is no better Wiki engine out there (my opinion) and it just keeps getting better with every release!.

That just leaves me to post a quick MySQL tip that may well be obvious, but happens to catch me out from time to time.

MySQL Tip of the month

If you ever get the urge to convert a 1Gig MyISAM table (with 34 million rows) to InnoDB within MySQL, try to avoid the simple ALTER TABLE tablename ENGINE=INNODB; method.
I made the mistake of running that command on a fairly decent server with bags of space and memory only to find the command running over 5 hours later with an InnoDB tablespace at around 4Gig (ok a bit of tuning might have helped)

Your best bet is to dump the table using mysqldump, drop the existing table and re-import the table after modifying the definition to be InnoDB. That worked for me in less than 10 minutes. Alternatively you can create a new InnoDB table and insert directly from the existing table as per MySQL documentation e.g INSERT INTO newtable SELECT * FROM oldtable.

I’m sure I am not the first to make this mistake, and may not be the last…..

I’m sure by now you would have read the great news that is Sun Microsystems acquisition of MySQL!

Sun have been forging ahead in the Open Source world with OpenSolaris, an Operating System that scales to Enterprise proportions and is an ideal host for the number one Open Source Database Server that is MySQL.

Whether this will have any impact on the MySQL Linux / FreeBSD / Windows offerings is something I am sure the community will be watching closely, but in my experience Java runs much faster on Solaris as it can take advantage of the MultiThreading libraries that help Solaris set those performance records.
Don’t forget, it was not that long ago that combined with Sun hardware MySQL was able to set it’s own world records!.

With that in mind it should only be a matter of time before we will begin to see the fruits of Sun / MySQL’s labour and I don’t know about you but I can’t wait!.

How this move will affect the future of other database servers such as PostgreSQL and Oracle I cannot begin to imagine.
Personally I never understood how Oracle devoted so much attention to Linux when the Oracle server (and App servers) have such a dependency on Java (which as I say always works better running under Solaris )?
As for PostgreSQL, Sun have always endorsed PostgreSQL on Solaris, will we see this focus switch to MySQL in the near future?.

One thing is for certain, MySQL combined with Solaris makes for one world class package that is going to be hard to beat!!.

I know it’s a little late, but as this is my first post of 2008 I wanted to start on a high!

After receiving my copy of the MySQL 5.1 Cluster DBA book at Christmas, I spent an hour or so each day with my head engrossed in MySQL Cluster technology, so much so that my goal of reading every Perl book I own (cover to cover) has been somewhat put on hold…

The book was much smaller than I expected, in both physical size and pages (266) which was not a bad thing as I could carry it around if needed, and each chapter could be read within 15-20 minutes, just about right for my attention span

I booked the exam a couple of weeks ago thinking I might need some incentive to fully digest MySQL Cluster details. I need not have worried, the book provides a great introduction to MySQL Cluster in easy-to-read chapters bringing you up to speed in no time, much more entertaining than the reference manual!

The exam itself consisted of 70 questions and on this occasion ended with a prompt asking to print to a file……argghhh……I suddenly got worried that the exam had crashed and that all my answers were lost?!, after all you normally get an instant result on screen and a printed record. I knew I should have used my regular testing centre Print -> File -> USB Stick -> another PC -> Adobe Distiller -> Print and I finally got the result - Passed - Phew!.

So the holidays are over and I have more time to focus on the fun stuff, like upgrading this site to the latest version of WordPress or adding GnuPG based encryption to AuthStor. If history is anything to go by the next few months are bound to see a spate of version bumps. I am personally looking forward to FreeBSD 7.0!

You never know, I might even find time to finish reading those Perl books before Perl 6 is complete

2007 is almost over and what a year this has been for Wave2!

My programming preferences have gone full circle, from beginning the year playing snakes and ladders (Python), I spent the summer drinking coffee (Java) and finally settled down surrounded by Camel’s, Lama’s and Parrots (Perl). Bizarre but never dull!

Steady progress has been made all round and as always the Open Source community has proven a valuable resource. Not a day went by where I did not discover something new or think of a new idea that could be built using Open Source Software. I only wish I had more time in the day!.

My Christmas present came early this year in the form of a MySQL 5.1 Cluster DBA Certification Study Guide. I have been putting off the final MySQL certification while waiting for the book to arrive, and now it is in my hands the only thing preventing me from taking the exam is the lack of free space over the Christmas period. Hehe

I also ordered a Nokia N810 (My Techie Present). My laptop is not small enough to carry around with me and I am hoping to capture more development time on this device. Either that or feed my mind.

Anyway I’m off to hit the WarHawk servers before getting ready to see the New Year in Amsterdam - home of the Blender Institute. With projects like AuthStor making great progress and ideas such as Unix Architect and BinaryStor slowly maturing, I cannot wait to see what 2008 holds for Wave2!

Before I head off I have to mention the Catalyst Advent Calendar. Every year they pack 24 days of tutorials and tips into the month of December. If you are looking to get the most of the Catalyst Framework you really must take a look at this resource. I was able to prototype AuthStor in under 2 weeks and enjoyed every minute!

Wave2 Wishes You a Very Merry Christmas and an Open Source New Year!!!

I read a lot about MySQL backups using LVM Snapshots on Linux, WAFL Snapshots on NetApp and more recently ZFS Snapshots. But did you know you can do the same under FreeBSD?

FreeBSD has had snapshot capability since around 2001 allowing administrators to take a frozen image of a filesystem at a given instant in time with minimal impact on the server / filesystem. So how does this help with MySQL Backups?

If like me you have loads of space on your database volume and often find yourself making frequent risky modifications to your databases, a snapshot can save you loads of time and headaches by creating a point in time marker that you can fallback on if needed.

Snapshots provide the following benefits:

1. Efficient - Only as blocks in the active filesystem are modified and written to new locations on disk does the snapshot begin to consume extra space.
2. Fast - It takes about 30 seconds to create a snapshot of an 8Gb filesystem. Of that time 25 seconds is spent in preparation; filesystem activity is only suspended for the final 5 seconds of that period.
3. Reliable - FreeBSD ensures that no disk write activity can take place during the setup of the snapshot (typically less than a second). The result is a consistent replica.

If you are running MySQL on FreeBSD, snapshots can certainly help improve your backup strategy. So how do you get started?
Well if you head over to the Wiki you’ll find a detailed guide to creating your first snapshot, but before you click that link there is another new FreeBSD feature that you may not be aware of - Security Event Auditing.

Protecting your data requires more than just regular backups and the security auditing feature in FreeBSD 6.2+ can help assist with the logging of activity on your MySQL server.

Based on Sun’s published Basic Security Module (BSM), the de facto industry standard for auditing, FreeBSD Security Event Auditing provides reliable, fine-grained, and configurable logging of a variety of security-relevant system events, including logins, configuration changes, and file and network access. These log records can be invaluable for live system monitoring, intrusion detection, and postmortem analysis.

At the moment the stable release of FreeBSD will require a Kernel compile to enable audit capability, but future releases should have audit enabled in the generic kernel. You never know by the time FreeBSD 7.0 is available I may have figured out a way to audit MySQL logins using auditd.

If your feeling brave you can download the latest monthly snapshot of the upcoming FreeBSD 7.0 release and put auditing to the test.

Right - I’m off to get a working NetUnit jar ready for the end of the week - I hope

While working on MySQLDump.java this weekend I decided to look into the new features that are heading our way in MySQL 5.1. It cannot be long before we see a general release and I needed to start thinking of ways to backup items like Events. This also made a good opportunity to start afresh with my dev server and try out the latest Solaris 10 8/07 release with GlassFish v2, the releases just keep coming! (SXDE 9/07 was just released).

The Solaris 10 install was as smooth as previous releases (Using Text Based) and once started took at least an hour or so to finish, I cannot remember the exact time as after checking the status a couple of times I forgot all about it. The general feel was much improved however, and the overall speed increase made the upgrade worthwhile.

GlassFish installed and ran as expected - easy. I would be very interested to see the difference in http performance when compared to Apache. Something for another day maybe?

Next on my list was MySQL 5.1.21-beta.
I originally downloaded the pkgadd version but failed to start the server due to a an [ERROR] Can’t find messagefile. It was only after receiving the same message with the Tar distribution that I headed over to the MySQL Bug System where it seemed the issue was a known one - BugID 30678.

Not a major issue as the solution was to simply add --language=./share/mysql/english/ to server install / startup commands. Something that I am sure would have been fixed in the release candidate. I would like to see a more complete installer for Solaris, something similar to the FreeBSD port. The pkgadd install does not really offer much customisation and both the Tar file / pkgadd lack any sample service manifest files.

Once installed everything worked as expected and my first task was to install the Sakila test database. I intend to use this database to validate the operation of the MySQLDump.java app. It may not be the biggest database to work with but it does support a wide variety of data types, views, triggers etc. I am currently using the MySQL Toolkit to verify the backup and restore. If you have not used the Toolkit before (it’s written in Perl) it includes a nice utility that can test if 2 tables are identical - handy for validating a backup.

I envisage using unit tests for this purpose in future, but need to find a good book on TestNG first. A quick dump of the film_text table worked as expected and the data was identical to that supplied by MySQL, I must be on the right track! - If only I had more time…

I did however find time to attend the MySQL Backup and Recovery webinar this morning and was impressed with the new Backup API planned for MySQL 5.2 onward. I just booked my place at the European MySQL Conference and hope to get some more insight into future developments, or at least a copy of the MySQL 5.1 Cluster Certification Study Guide.

Whenever I decide to take a holiday I can usually guarantee that something cool will occur in the IT industry in my absence.

Now this holiday has been slightly different in many ways. I had to cancel my flight due to a sudden onset of Vertigo 1 day before I was supposed to fly. If like me you have never experienced Vertigo before, thank yourself lucky!. Having no sense of balance and intense room spin whenever you open your eyes is no fun at all, it took at least 2 days for me to stop being sick.

Having been able to actually use my laptop in the last couple of days I notice that the “cool thing” I usually miss on holiday turns out to be big trouble all over OpenSourceVille, the majority of which seems to revolve around licensing and intellectual property.

Who could miss the BSD vs GPL debate that I’m sure will rage for a long time after the ath5k driver had the BSD license removed in favour of the GPL. Stealing code was suddenly the headline for this action and the rift between the “Linux” developers and “BSD” developers increased once more.
Personally I feel the license should be as “Open” as possible and always apply the BSD license where I can. I want to be recognised as the author of any work I produce and protected against any misuse of such work (liability). With OpenSource development taking place in clear view nowadays, such visibility can only enforce your ideas and intellectual property. Why complicate things with over restrictive licenses…

While still on the topic of licensing it seems John Birrell is having to start all over again with a new port of DTrace on FreeBSD. The CDDL this time preventing kernel hooks from being added to the FreeBSD kernel. This seems such a shame as DTrace technology provides a massive benefit for software developers and will starting afresh really change the situation?, If not through licensing I am sure there must be patents on the intellectual property that Sun could enforce on any DTrace port? Just look at what is going on between NetApp and Sun at the moment.

So from licensing issues to patent infringement where it seems ZFS may not be as revolutionary as Sun claim and in fact NetApp actually came up with some of the functionality back in 1998. Dave’s Blog outlines all the details in what I can only describe as an excellent explanation as to how NetApp have had to file a lawsuit against Sun. Strangely enough I originally heard about this from the MySQL Planet! and not from any of the Solaris / OpenSolaris Planets each of which have around 45 Sun employee feeds?

We develop software to improve our way of life. Licensing and Patents although often necessary seem only to be hindering progress… Still it keeps us busy re-inventing the wheel.

On a good note, I was around to see the launch of the iPod Touch - if you can call that good…I might just wait until the drive space increases before I buy one, 16Gig seems tiny these days but then again the Touch would come in handy for on the go web browsing.
I also noticed Oracle 11g had been released (I obviously missed that one), although only for Linux at the moment?. I’m no Oracle expert but seeing as Oracle provide Java based development tools and my previous installs had a lot of Java involved surely Solaris would make a better fit?

Image combined using artistry from Project TIMEMACHINE - http://www.timemachine.ne.jp/

I started work on a Java class that mimics the functionality of the mysqldump command today.
My goal is to create a standalone application to begin with, but design the class such that it can be plugged into any other Java application requiring MySQL backup capability.

After completing the initial prototype of the dump_table method (minus blob support), I began to wonder what the performance would be like compared to the bundled mysqldump application (written in C).

As this was a prototype with no optimisations, the results were not that bad. The following figures were obtained from running both the C-based mysqldump and the MySQLDump.java app against a fairly large innodb attachments table (about 260mb).

Now this was in no way a strict test environment. I ran the test on my laptop running OSX 10.4.10 connecting to an OpenSolaris server running MySQL 4.0.24 (out of the box) over Wifi. I had to increase the max_allowed_packet and import the large table, but apart from that the server was a clean install.

The mysqldump client was from the MySQL 5.0 packaged install for OSX and the Java VM was version 1.5.0. I did try running the test against the beta JVM 1.6.0 for OSX, but that was 30 seconds slower. I’m sure that will improve when the final version is released and debugging code removed.

I tried a final test by running both commands directly on the OpenSolaris server still connecting via IP.

As you can see there is a slight improvement, probably due to the mysqldump version being older out of the box on Solaris and Java apps running faster on that platform. Of course the final version will check if it is run locally on the server and issue a “SELECT /*!40001 SQL_NO_CACHE */ * INTO OUTFILE”. I just wanted to confirm my view of Java running faster under Solaris.

I wonder how fast this would run under Perl…OK, enough fun with timings, I best get back to writing the code before I head off on Holiday….