If only your database would just tell you that replication had failed or that the disk was full…
Ok some database servers do but MySQL doesn’t (yet). Another excuse to write a script

In my pursuit for total database visibility I have been searching for a tool that would tell me when something went wrong, that would simulate regular usage, and let me know if anything failed. After all SNMP can only probe so far and if your database is secure the last thing you want to do is open up another port on your server.

Don’t get me wrong there are tools out there that do a decent job of monitoring MySQL. MySQL Entperprise Monitor is one such tool. But if you want to look under the hood or add some functionality specific to your environment things start to get tricky.

So what’s this dbAlerter?

dbAlerter is the name of a project (Script) that I started developing to provide notification of key server events. It’s written in Python; Why?, I wanted something lightweight. Java would have been my first choice but the JVM eats too much ram and felt too heavy for such simple functionality.
Python on the other hand feels lighter and when running a script as a daemon it seems to take up less resources allowing the DB to use more memory and continue doing it’s job. Plus it’s been a while since I wrote anything with Python, variety is the spice of life!

Speaking of Java, if you have ever used tools like JConsole and jmap (and if not why not!) you may not have realised that since JDK 6 Update 7 there’s now one tool to rule them all - VisualVM.

VisualVM pulls together several commandline tools like JConsole, jstat, jinfo, jstack, and jmap to make monitoring and profiling a more visual experience.

Now that the changes from BSD Java have been merged into the OpenJDK and the FreeBSD Foundation have updated the JDK to 1.6.0-7 I’m pretty sure the original concept of write once run anywhere is closer than ever.

Databases are used by many different disciplines, from your DBA to your Developer, right down to the End User. They can also be viewed from many different perspectives…

DBA’s are usually interested in the server configuration, index usage etc. Developers however are often concerned with retrieval methods, datatypes and occasionally stored procedures. One thing is for sure, your database if used effectively will be changing frequently, and with so many people accessing your database do you know what the last change was? Would you know if someone added an index to a table? (would you care… )

Revision Control is often used to track changes to source code and other documents, so why not your database?.

When I first started to think about change control for my database servers one tool sprung to mind, Rancid. If you manage more than one network device you may already be familiar with the Really Awesome New Cisco confIg Differ.

Using a combination of scripts Rancid will connect to your network devices, check for changes by comparing the config with a VCS managed copy and notify you with details of the changes. Why not do the same for your database? Enter dbRecorder.

Rather than extend Rancid I decided to prove the concept by writing a quick Java application that utilised Open Source libraries like SVNKit and jYAML. The features continued to creep and before I knew it I was recording more than just table definitions. Server configuration and grant tables all found their way into my Subversion repository coupled with e-mail notification whenever anything changed - nice huh!.

The aptly named dbRecorder is currently targeted at my database of choice, MySQL. It’s far from a finished product (then again so are most of my ideas) but it’s working to a degree and can only improve over time….

One thing I never got around to was making this site work well with IE6 (or even just splash the screen with some silly message when someone using IE6 tries to connect) and with Firefox 3 soon to be released I am hoping I won’t have to. If you have not tried Firefox 3 already you can visit the site today and pledge your download to help break a world record as soon as it is released!.

2007 is almost over and what a year this has been for Wave2!

My programming preferences have gone full circle, from beginning the year playing snakes and ladders (Python), I spent the summer drinking coffee (Java) and finally settled down surrounded by Camel’s, Lama’s and Parrots (Perl). Bizarre but never dull!

Steady progress has been made all round and as always the Open Source community has proven a valuable resource. Not a day went by where I did not discover something new or think of a new idea that could be built using Open Source Software. I only wish I had more time in the day!.

My Christmas present came early this year in the form of a MySQL 5.1 Cluster DBA Certification Study Guide. I have been putting off the final MySQL certification while waiting for the book to arrive, and now it is in my hands the only thing preventing me from taking the exam is the lack of free space over the Christmas period. Hehe

I also ordered a Nokia N810 (My Techie Present). My laptop is not small enough to carry around with me and I am hoping to capture more development time on this device. Either that or feed my mind.

Anyway I’m off to hit the WarHawk servers before getting ready to see the New Year in Amsterdam - home of the Blender Institute. With projects like AuthStor making great progress and ideas such as Unix Architect and BinaryStor slowly maturing, I cannot wait to see what 2008 holds for Wave2!

Before I head off I have to mention the Catalyst Advent Calendar. Every year they pack 24 days of tutorials and tips into the month of December. If you are looking to get the most of the Catalyst Framework you really must take a look at this resource. I was able to prototype AuthStor in under 2 weeks and enjoyed every minute!

Wave2 Wishes You a Very Merry Christmas and an Open Source New Year!!!

It’s been a few weeks since I last posted here or even found time to develop any of the ideas that continually spring to mind. In fact this is the first weekend in several where my head is not stuck in a book about Perl, Solaris or UML. Why you may ask? (or not).

Well after finally getting round to taking the Solaris Network Administrator exam, I thought it only right that I complete the set and get to know Solaris Security a bit better by taking the SCSECA.

After achieving the SCSECA, I was just about getting ready to start writing code again when I stumbled upon the UML Certification Program.
I have been a fan of UML for a long time but never found a tool that I felt comfortable using, or more to the point a tool worth the money!. That was until on the very same day I decided to trial Visual Paradigm, could this be the UML tool I have been seeking?

Visual Paradigm for UML is not overly expensive (many price levels to cater for all requirements), it runs on Windows / Linux / OSX and it’s easy to use - I like it!.

Armed with the UML tool I have been looking for and a copy of the UML 2 Certification Guide I spent another week brushing up on my UML skills while studying for the first of 3 exams offered by the OMG, UML Certified Professional - Fundamental.
I passed the exam last week and learned a lot more UML along the way!
So finally, with all that studying out of the way I can finally get back to the fun stuff - Code.

But where does Perl fall into this jacked-in period? Well I have been waiting for Perl 6 for a long time now and plan to use Catalyst as the platform for BinaryStor and AuthStor. So what better way to get up to speed than a Perl readathon of as many Perl books as I can get my hands on. From Learning to Mastering and eventually Best Practices all over again, you never know I might just finish reading before the release of Perl 6.

As for the Code I cannot make my mind up as to where to focus my energy and so opted for a working NetUnit (I did think of porting it to Perl ). It’s far from complete of course and suffers from the constraint of root privileges, but I’m hoping a future Java release will fix that….

I had one of those lightbulb moments the other day while pondering a problem I recently encountered relating to firewall changes.
The problem came to light the morning after some firewall rules had been modified, nothing new there I hear you say.
Of course at the time everything looked to be working, that is from the perspective of the few people that were actually working that late in the night. It was only when the morning came and the rest of the business started to come online that the problems were apparent. It didn’t take long to fix the problem but it got me thinking, how could we prevent this in future?

I have been spending some time looking at Unit testing recently and began to wonder if there were any testing tools or frameworks that could be used to prevent this kind of problem in future. Unit testing for software is pretty much standard practice nowadays and works well during the development phase, but once the application is in situ how can you test the operation from a networking perspective?. Web testing tools are great for validating web application functionality and I’m pretty sure there are loads of other test frameworks out there, but are they easy to use? I wanted a way to describe and validate the problem before and after the work was undertaken, and so NetUnit was born…

NetUnit aims to be simple to use, something I could give to a non-techie and have them run a Test Suite to validate connectivity or application functionality. NetUnit mimics Apache Ant in its operation and design, in fact I was going to write some Ant tasks at first but kept thinking of new ways to extend the tool in future.

The Test Suite used to describe the Tests to perform is a simple XML file. NetUnit parses this file and outputs the result of each test along the way - simple huh!.

Another benefit of this tool that immediately sprung to mind was the ability to take a Test Suite file and generate documentation for your applications external network interface or firewall policies. In fact the ideas don’t stop there and I have quite a list of features that could be implemented in future. Time permitting as always….

While working on MySQLDump.java this weekend I decided to look into the new features that are heading our way in MySQL 5.1. It cannot be long before we see a general release and I needed to start thinking of ways to backup items like Events. This also made a good opportunity to start afresh with my dev server and try out the latest Solaris 10 8/07 release with GlassFish v2, the releases just keep coming! (SXDE 9/07 was just released).

The Solaris 10 install was as smooth as previous releases (Using Text Based) and once started took at least an hour or so to finish, I cannot remember the exact time as after checking the status a couple of times I forgot all about it. The general feel was much improved however, and the overall speed increase made the upgrade worthwhile.

GlassFish installed and ran as expected - easy. I would be very interested to see the difference in http performance when compared to Apache. Something for another day maybe?

Next on my list was MySQL 5.1.21-beta.
I originally downloaded the pkgadd version but failed to start the server due to a an [ERROR] Can’t find messagefile. It was only after receiving the same message with the Tar distribution that I headed over to the MySQL Bug System where it seemed the issue was a known one - BugID 30678.

Not a major issue as the solution was to simply add --language=./share/mysql/english/ to server install / startup commands. Something that I am sure would have been fixed in the release candidate. I would like to see a more complete installer for Solaris, something similar to the FreeBSD port. The pkgadd install does not really offer much customisation and both the Tar file / pkgadd lack any sample service manifest files.

Once installed everything worked as expected and my first task was to install the Sakila test database. I intend to use this database to validate the operation of the MySQLDump.java app. It may not be the biggest database to work with but it does support a wide variety of data types, views, triggers etc. I am currently using the MySQL Toolkit to verify the backup and restore. If you have not used the Toolkit before (it’s written in Perl) it includes a nice utility that can test if 2 tables are identical - handy for validating a backup.

I envisage using unit tests for this purpose in future, but need to find a good book on TestNG first. A quick dump of the film_text table worked as expected and the data was identical to that supplied by MySQL, I must be on the right track! - If only I had more time…

I did however find time to attend the MySQL Backup and Recovery webinar this morning and was impressed with the new Backup API planned for MySQL 5.2 onward. I just booked my place at the European MySQL Conference and hope to get some more insight into future developments, or at least a copy of the MySQL 5.1 Cluster Certification Study Guide.

It’s been quite a while since I last looked at Google Co-op, but after moving my site recently I realised my Subscribed Links were no longer up to date, let alone working!. I then discovered the Python script I wrote a while back was no longer functioning either, then again it’s no wonder seeing as I had not looked at it for over a year!. Time for an upgrade…

I have been on a Java kick recently, and when not suffering from Vertigo I found myself getting back up to speed with JDK 1.6 and the now final GlassFish V2 release. The difference in speed between JDK 1.5 and 1.6 is certainly noticeable, and as my Google Analytics stats displayed 99% of recent clients having Java support (not necessarily 1.6) I decided a re-write of DeliciousSubscribed was in order.

I started out on the del.icio.us site where there was not much in the way of Java utilities, there was however a well built library - del.icio.us Java API. I would have used that library if it were not for the dependencies on Apache Commons, not that there is anything wrong with Apache Commons I might add!, I just wanted to keep the dependencies to a minimum for size and simplicities sake. So back to drawing board it was, my goal was to simply reproduce the Python script in Java and create an executable jar that can be run from the command line or embedded in another Java app if needed.

The JRE/JDK contains enough libraries to make connecting to the del.icio.us HTTP API and parsing the resultant XML a breeze. I did select one dependency however and that is args4j. Args4j provides simplified command line parsing, comes in under 36k and happens to be licensed under the MIT License so no viral licensing there. It did not take long to get a valid Subscribed Links XML file.

There is still a lot more to do before the code is stable of course and I am sure there are many more features I can add, in fact while re-writing the app I noticed a few cool additions to the Subscribed Links service that I may make use of in a future release:

• Include images in your Subscribed Links.
• Include Google Gadgets in your Subscribed Links.

You never know I might even whip up a JNLP deployed GUI when I finally get around to reading Filthy Rich Clients.

Speaking of features MySQL 5.1 must be getting close to Release Candidate and includes some cool new features such as Partitioning, Row based replication, Table logs and my personal fave Events.
If you are looking for a easy read intro i’d suggest you wander over to the MySQL site where you can find a brief intro into the major new features and bug fixes found in 5.1.

Right I must get back to Warhawk on the PS3. It’s been a while since I got hooked on an online game but this one seems to be stealing every hour I have free!

I started work on a Java class that mimics the functionality of the mysqldump command today.
My goal is to create a standalone application to begin with, but design the class such that it can be plugged into any other Java application requiring MySQL backup capability.

After completing the initial prototype of the dump_table method (minus blob support), I began to wonder what the performance would be like compared to the bundled mysqldump application (written in C).

As this was a prototype with no optimisations, the results were not that bad. The following figures were obtained from running both the C-based mysqldump and the MySQLDump.java app against a fairly large innodb attachments table (about 260mb).

Now this was in no way a strict test environment. I ran the test on my laptop running OSX 10.4.10 connecting to an OpenSolaris server running MySQL 4.0.24 (out of the box) over Wifi. I had to increase the max_allowed_packet and import the large table, but apart from that the server was a clean install.

The mysqldump client was from the MySQL 5.0 packaged install for OSX and the Java VM was version 1.5.0. I did try running the test against the beta JVM 1.6.0 for OSX, but that was 30 seconds slower. I’m sure that will improve when the final version is released and debugging code removed.

I tried a final test by running both commands directly on the OpenSolaris server still connecting via IP.

As you can see there is a slight improvement, probably due to the mysqldump version being older out of the box on Solaris and Java apps running faster on that platform. Of course the final version will check if it is run locally on the server and issue a “SELECT /*!40001 SQL_NO_CACHE */ * INTO OUTFILE”. I just wanted to confirm my view of Java running faster under Solaris.

I wonder how fast this would run under Perl…OK, enough fun with timings, I best get back to writing the code before I head off on Holiday….

Since writing the mmMySQL macro a few weeks ago I have been looking for a way to produce impressive Graphs that would compliment the textual output from the macro.

Creating good looking Graphs often means resorting to Matplotlib in Python, GD in Perl/PHP or RRDTool. The result of these tools is often a static image depicting an historical view of your data, something that does not lend well to a non-daemonized Wiki page. That got me thinking…..how could I embed a graph into the webpage and collect data at the same time?…..a Java Applet!

After a quick search for a MySQL Java Applet I came up short and began to wonder how easy it would be to write my own. MySQL already produce a connector for Java, Connector/J, so that’s the connectivity sorted, I just needed a graph library now…..enter JFreeChart.

Not necessarily lightweight at around 2Meg, the libraries make easy work of connecting to MySQL and populating live Graphs. In fact with today’s connectivity and the caching built into the JRE 1.6 the initial download is the only bottleneck, unless of course you are using a mobile device which may not have the capacity or even the capability.

Starting out in Eclipse with a simple Hello World applet I quickly got frustrated at the lack of GUI tools and the flaky applet viewer. A wander over to the Netbeans site soon got me back on track with the top notch Matisse GUI builder, add to that the wicked profiler (Nice Graphs) it didn’t take long to get a working live graph applet. In fact the hardest part in creating the concept was dealing with the JRE security model and having to manually sign the applet for it to work (the JRE protects the user and prevents any socket connections without user approval).

So I have a concept, something that works, a lot of potential, all I need to do now is pull out my Thinking in Java book and work on something more concrete. I’m thinking of creating wrapper classes that extend the JFreeChart classes to provide custom MySQL charts e.g. Traffic Chart and Connections Chart. If only I had more time…..