Lenz Grimmer's blog

Like in the years before, I (and some other colleagues from MySQL) will be attending FOSDEM 2008 in Brussels, Belgium on February 23rd and 24th.

The schedule is packed as usual and lists several MySQL-related topics and sessions:

• Main tracks: Unicoding With PHP 6 by Andrei Zmievski and the talk about CakePHP by Garrett J Woodworth are certainly relevant for web developers using PHP.
• Lightning talks: Jan Kneschke will speak about MySQL Proxy (not yet listed in the schedule, but likely between 15-16 o'clock on Sunday). Other sessions with MySQL relevance/relations are Kettle: extracting, transforming and loading data by Matt Casters and Alfresco: fresh document management approach by Philippe Dubois.
• DevRooms: The Drupal people have set up a DevRoom with a session track - Kris Buytaert will talk about Drupal and MySQL High Availability. Another DevRoom worth visiting will be the one organized by the Belgian Ruby User Group - they set up a Ruby on Rails Developer Room. My friends from the openSUSE project also have a great number of sessions lined up for their openSUSE Developer Room: several talks about the Build Service (proudly powered by MySQL) and a session about the openSUSE redirector (a tool to set up and manage a content delivery network with MySQL) by Peter Poeml.

If you also plan to attend, please ping me if you would like to meet!

 

 

 

When it comes to handling my work-related email, many people will probably conclude that I am an old fart in Linux terms: I still use the console-based Pine as my MUA of choice, as my fingers have been hardcoded to its key combinations over the years (my first encounter with Pine was around 1994 on a shared DEC Ultrix box in my university). So far, I have not found any other application that allows me to process email as quickly as by using this tool.

I admit that I do use Thunderbird for my personal email, though, to gather experience with it (and to toy around with the various extensions, especially Nostalgy is a gift from heaven for people like me!). And of course because the mail volume there is less critical to cope with! But I am not going to start a holy war here - this article is supposed to explain how I have configured my mail setup to rotate the log files that keep track of all my incoming mails.

Continue reading "Linux tip: Automatically rotate/archive your fetchmail/procmail log files"

We have just published an interview with our CEO Mårten Mickos on our Developer Zone, where he answers a number of questions from the Community Team about Sun's plans to acquire MySQL AB and what this means for the MySQL User Community and the direction of the product (thanks to Colin for compiling and editing the text). I hope that the comments help to calm the concerns of some people about the future of MySQL with regards to Open Source, support of platforms or language bindings:

CT: Is MySQL fully committed to free and open source software, and the GPL?
Mårten: Yes.
CT: What does the Sun acquisition mean for Linux support? What about Windows, Mac OS/X and other platforms?
Mårten: One of the reasons for us to be interested to discuss an acquisition with Sun was their new strategy to work with technologies and companies that previously had been seen as competitors. Our mandate within Sun is to grow our database business - on all operating systems and with all development and scripting languages and on all hardware platforms. Linux remains the most important operating system for our business. Windows remains the top operating system among our downloads. Mac OS X is a vital platform and many of our own engineers use Mac notebooks.
CT: Will all the language bindings still be supported? i.e Will PHP, Perl, Ruby, etc. be treated as well as Java?
Mårten: Absolutely. See my previous answer. Our goal and mandate is to grow our business on all fronts.

I personally am very excited about this change and expect many good things to come out of  it. Sun is really trying hard to "not screw this up" (quoting Jonathan Schwartz).

UPDATE: I just stumbled over another article about this topic, which I also consider recommended reading: Mårten Mickos: Why Did I Change My Mind?:

I may sound naively excited about Sun in my writing here, and perhaps I am. Perhaps I drank the Kool-Aid along with the traditional Swedish vodka shot when we closed the deal. But in my life and my professional career so far, I have never regretted using my gut instinct to choose between two rational alternatives. 

 

 

While helping a user with some questions about the geospatial extensions of MySQL on the #mysql-dev IRC channel on Freenode, I stumbled over this blog: How to use MySQL Spatial Extensions. There currently is just one post, but it was exactly what we were looking for: "Using Circular Area Selection". Nice work! I hope the author will soon provide more examples of how to make use of this functionality.

By the way, there is work in progress to improve the GIS functionality in MySQL - if you are looking for new GIS functions that do not use MBRs, take a look at our GIS feature preview on the MySQL Forge. We plan to publish a new set of preview binaries once the 5.1.23 release has been published and merged into this code base. 

 

Just a short reminder: the Hamburg MySQL Meetup Group will meet next week, on Monday the 4th of February! As usual, there will be a MySQL-related talk - this time I will use this opportunity to practice my talk about "Performing MySQL Backups using LVM Snapshots" that I will present at the upcoming MySQL Conference & Expo 2008 in Santa Clara, CA. In addition to that I assume that there will be some questions about the acquisition of MySQL AB by Sun - I will try to answer these as good as I can. Looking forward to meet you next week. Please make sure to RSVP either via Meetup.com or Xing.com, so I can keep track of the number of attendees! Thanks!

 

 

While browsing the many blog entries on blogs.sun.com about the MySQL Acquisition (thanks a lot for the very warm welcome!), I stumbled over this (Python-based) utility: SnapBack, a tool that uses ZFS snapshots to perform physical backups of MySQL databases on Solaris. Very cool! This is actually something I was wanting to add to the mylvmbackup script, too - I have to take a closer look at how this is done (I tried to install OpenSolaris on a VirtualBox instance, but it caused it to crash the emulator).
 

 

Since the very early days, the MySQL distribution packages contain a very useful commandline-tool named replace. As the name implies, it can be used to replace strings in text files. From the documentation:

The replace utility program changes strings in place in files or on the standard input. Invoke replace in one of the following ways:

shell> replace from to [from to] ... -- file [file] ...
shell> replace from to [from to] ... < file

from represents a string to look for and to represents its replacement. There can be one or more pairs of strings. Use the -- option to indicate where the string-replacement list ends and the filenames begin. In this case, any file named on the command line is modified in place, so you may want to make a copy of the original before converting it. replace prints a message indicating which of the input files it actually modifies. If the -- option is not given, replace reads the standard input and writes to the standard output. replace uses a finite state machine to match longer strings first. It can be used to swap strings. For example, the following command swaps a and b in the given files, file1 and file2:

shell> replace a b b a -- file1 file2 ...

This comes in very handy, especially if you need to replace a similar text in a large number of files, or if you need to replace multiple different strings in a file at the same time! So next time you need to replace strings in text files, take a look at this hidden gem!

Unfortunately the string matching does not support regular expressions - but the source code actually documents some additional options:

Special characters in from string:
  ^  Match start of line.
  \$  Match end of line.
  \b  Match space-character, start of line or end of line.
        For end \b the next replace starts locking at the end space-character.
        An \b alone or in a string matches only a space-character.
  \r, \t, \v as in C.

The programs make a DFA-state-machine of the strings and the speed isn't
dependent on the count of replace-strings (only of the number of replaces).
A line is assumed ending with \n or \0.
There are no limit exept memory on length of strings.

 

 

 

 

Like in the past two years, there will be a DotOrg-Pavilion as part of the MySQL Conference & Expo 2008 exhibition floor. We've now concluded the invitation process - the following 15 Open Source projects that support MySQL will be present to showcase their work:

• CMS made simple - a fast and easy way to create a web site and manage its contents.
• DbBlender - a project that implements SQL support in Blender to enable remote repository capabilities.
• Entrance - an easy-to use browser for MySQL databases.
• Deki Wiki - a free open source wiki and application platform for communities and enterprises.
• lbpool - a load balancing JDBC driver for use with DB connection pools.
• memcached - a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
• Mojo Portal - an open source web site framework and content management system written in C# that runs under ASP.NET on Windows or under Mono on Linux or Mac OS X.
• Open Medical Record System - a community-developed, open-source, enterprise electronic medical record system framework intended to aid resource-constrained healthcare environments.
• phpMyAdmin - a tool written in PHP intended to handle the administration of MySQL over the Web.
• PHP Women - a user group specifically targeted toward the women in the PHP world.
• Phorum - a modular and powerful Forum siste, written in PHP.
• SilverStripe CMS - an intuitive content management system atop a powerful programming framework.
• Sphinx FULLTEXT Search Engine - a standalone full-text search engine, meant to provide fast, size-efficient and relevant fulltext search functions to other applications
• Symfony - a web application framework for PHP5 projects.
• XAMPP - an easy to install Apache distribution containing MySQL, PHP and Perl.

I hope that the list above encourages you to strongly consider visiting the MySQL Conference! If not the list of projects above, I am sure that the session schedule will provide enough incentive for you to register now! See you there!

I must be awake for about 24 hours now, things are getting a bit blurry. But I have arrived at our MySQL Staff Meeting here in Orlando safely! My flight with Northwest airlines from Frankfurt via Detroit was uneventful - the plane was pretty empty so I was happy to have two seats for myself! The board entertainment program was neat, I really prefer video on demand over scheduled movie broadcasts (I watched "The Nanny Diaries", which was quite funny, and "The Fantastic Four", which had some nice CG effects).

While standing in the immigration line in Detroit somebody in the line next to us waved to me - it was Tobias "Flupps" Asplund, one of our trainers! Quite a funny coincidence, he just had arrived via Amsterdam. We actually were on the same flight from Detroit to Orlando afterwards. After checking into the Hotel I finally found my way to the evening reception - it was nice and overwhelming at the same time to see so many MySQLers in one place! I enjoyed talking and meeting old friends as well as getting to know new colleagues that joined the company since the last time we met. The following days will be exciting, interesting and exhausting... But for now I am off to bed.

 

I am happy to announce that I have been selected to speak at the MySQL Conference & Expo 2008 on April 14-17, 2008 in Santa Clara, CA. The topic of my talk will be "Performing MySQL Backups using LVM Snapshots" - and I will of course include a plug for the mylvmbackup tool

From the abstract:

Modern Linux distributions provide a very advanced technology to maintain and administer storage devices, called “Logical Volume Manager” (LVM). Among providing several benefits over using plain hard disk partitions, one of the unique features of LVM is the capability to create snapshots of file systems on the fly. This functionality comes in handy for performing consistent physical backups of MySQL tables without having to take down the MySQL server for the entire backup operation.

In this session, Lenz will give an introduction to the underlying technology, the actual steps that have to be performed and which caveats there may be. Additionally, he will introduce “mylvmbackup”, a script created to automate and ease the described procedure.

My talk has been scheduled as part of the "Security and Database Administration" track for 17:15 on Wednesday, April 16th in Ballroom B - the last slot before the infamous Quiz Show!

 

 

While some colleagues have already arrived in Orlando to meet fellow MySQLers, my plane from FRA to MCO (via DTW) does not leave until 10:20am tomorrow. Currently I am staying at my parents' place in Heidelberg, from which I will be picked up by an airport shuttle service early tomorrow morning (5:40am, ugh!).

I am looking forward to meeting with the other members of my team as well as old and new colleagues from the different departments. There seems to be some kind of outdoor team building event taking place on Wednesday afternoon - we have been split up into mixed groups and were asked to bring comfortable clothes and a light sweater or jacket. I hope I am not too jetlagged by the end of the day...

This will be my fourth MySQL staff meeting - when I joined the company in 2002, I was lucky to attend the third staff meeting in St. Petersburg, Russia right during the first week when I started my new job. Back then, we were around 40 people in total - the entire company still fitted in one coach that brought us from Helsinki, Finland to St. Petersburg. It's amazing to think that this staff meeting will host the tenfold amount of people!

 

Version 0.6 of mylvmbackup, a script to perform backups of a MySQL server using Linux LVM snapshots, has now been released.

In addition to various code cleanups and documentation improvements, many new features have been added to this version. I'd like to specially thank Robin H. Johnson from the Gentoo project for contributing many of the improvements to this release!

• Added a new rsync backup type. This is very useful if you want to use mylvmbackup to create the initial state for your slave servers. Instead of creating a .tar.gz archive, the data directory is copied into a timestamped archive directory. (Robin)
• Added support for a trailing argument to tar, which can be used for excluding files. (Robin)
• Separated out the suffix of the tarball (Preperation for rsync and users that want to use bzip2 or no compression on the tarball.) (Robin)
• While the backup is performed, a temporary suffix at the end of the tar backup file name (or the rsync target directory name) now indicates that it is incomplete. (Robin)
• The my.cnf configuration file is now included in the backup. (Robin)
• Added the ability to run an extra FLUSH TABLES on busy databases where lvcreate might take a long time (and may overrun the interactivity timeout on the connection, losing the lock). (Robin)
• Added option "--pidfile" to provide an alternative PID file location for the second server instance that is started to perform the InnoDB recovery on the snapshot prior to backing it up (Otherwise it may default to using the same pid file location that the running server uses and safe_mysqld will abort) - thanks to Kristian Köhntopp for making me aware of this problem.
• Before discarding the snapshot LV, the output of "lvs <snapshot>" is now printed out for diagnostics. It contains useful information like "how much percent of the backing store was used", which helps tuning the size of the snapshot LV.
• Added option "--skip-flush-tables" that performs the snapshot without flushing the tables to disk beforehand (which is not supported by InnoDB tables anyway) - this would save time, as the flushing can take a while, depending on the buffer sizes. (Thanks to Peter Zaitsev for the suggestion)

Version 0.6 is now available for download from http://www.lenzg.net/mylvmbackup/ (source tarball and RPM). I also provide RPM packagages for a number of additional platforms via my home:LenzGr repository on the fabolous openSUSE Build Service.

Enjoy! Feedback, patches and suggestions are welcome - please consider joining the mylvmbackup mailing list to discuss your experiences with this tool.

 

This certainly took longer than expected, but here are some pictures that I took during our MySQL Developer Meeting in Heidelberg, Germany last month. Enjoy!

For those of you who enjoy running exotic operating systems: I just stumbled over a port of MySQL 5.0.45 for OS/2 and eComStation. Thanks a lot to Paul Smedley for maintaining this version! He also maintains a large number of other Unix applications that he ported over to OS/2 - very impressive.

There is a lot of exciting stuff happening inside of MySQL AB. But due to the distributed nature of our company it's hardly possible to get a good overview about what the various teams of our development department are currently working on and what they have achieved since the last time we met.

So one cool new idea for our currently ongoing MySQL Developer Meeting in Heidelberg was to let developers show off their work to each other. They were encouraged to prepare demos, either in the form of slide shows or by running live demonstrations from their laptops. Last Thursday and Saturday we allocated time for these team exhibitions and the exhibitors set up tables in the meeting rooms for others to sit next to them, see the new and cool stuff and chat about it. The non-exhibiting attendees received a sheet of paper where they could collect signatures for each demo point they visited, the one that managed to see the most demos was eligible for winning a price. I managed to visit ~8 of them on Thursday - there was way more stuff to look at than what one could actually get done in the given time frame. Therefore we decided to repeat some of Thursday's exhibitions and included some new ones on Saturday.

Here is a recap of the exhibitions I saw on Thursday:

• Alexey 'Holyfoot' Botchkov demoed the new and improved GIS (geospacial information) functionality that I already blogged about some time ago. It provides a number of new GIS functions and most of the already existing ones no longer use MBRs. Holyfoot actually prepared a visual demo application that displayed a map of the earth by getting the polygons to draw out of the database. It was then highlighting the current country under the mouse cursor, displaying some additional information about it. Moving the mouse across country borders immediately switched the highlighting to the respective state. When using the old, MBR-based functions this would only happen when you crossed the borders of the bounding rectangle.
• My fellow colleague Giuseppe Maxia performed some MySQL Proxy Magic, misusing the MySQL Proxy for some fun stuff and also demonstrating some useful use cases (e.g. crosstabs). See his blog entry for some examples of what he was showing.
• Jay Pipes demoed his work on MySQL Forge 2.0, now with more Ajax magic, rating and commenting on projects and snippets as well as the obligatory rounded corners! There's still some work to be done, but I look forward to seeing it go live soon.
• Monty Tailor explained his work on NDB/Connectors, which provides bindings to many programming languages like Ruby, Perl, Lua, PHP and others. They make it possible to connect to a MySQL Cluster without having to go through the MySQL API, if you don't need to use SQL and just want to use MySQL Cluster as a highly available datastore (similar to memcached). It was interesting to compare the various test scripts for each language with each other - the Perl version looked quite ugly in comparison to the Ruby code...
• An alpha version of the MySQL Workbench was demoed by Johannes Taxacher from the GUI team. He demonstrated how one can visually create tables and set up the relations between them. The GUI is quite nice, but they support Windows only for now - Mac OS X and Linux are also planned, however.
• Kent Boortz from the build team showed off his work on creating a self-contained ODBC build environment. I was quite impressed by the complexity and steps involved in creating the binary packages of the ODBC driver we provide. As we support quite a number of platforms, Kent had to come up with all kinds of interesting tweaks for exotic architectures and strange compilers.
• Jess Balint from the Connectors team demoed the ODBC 5.1 driver by connecting to a MySQL Server using MS Access or with a MS Visual C++ application. It was nice to observe the functional UTF-8 support.
• Paul DuBois, long time member of our documentation team explained the build system they created in order to cope with the massive amount of build targets that are being created from the DocBook sources of the reference manual. In addition to various formats of the manual itself (e.g. PDF, HTML, Windows Help), they also build the manual pages that are included in our *nix distributions as well as some README files and the Changelogs. It is written in Ruby and is capable of distributing the build jobs to several machines.

That's all I was able to see on Thursday. I just wish I had some more time to see all of the other demos. On Saturday I saw the following demos:

• Marc Alff was demoing LTP-GCOV, a tool that visualizes the output of gcov as HTML pages and allows one to find out which code has been covered during the test suite run. It would be great if we could combine it with our internal pushbuild system, which automatically rebuilds the MySQL Server and runs the test suite on a number of platforms for every ChangeSet that is being pushed into a source tree. We already perform gcov build runs to make sure that new code is properly covered by the tests, but this tool would be a very helpful addition, allowing us to find code with low coverage much easier.
• Kristian Nielsen showed an early implementation of a multi-threaded ndbd, the process that runs on a MySQL Cluster data node. This should boost performance on multi-core systems (the current implementation is single-threaded). However, it's still in an early prototype phase, but looks interesting!
• Chad Miller demonstrated an alternative revision control system: bzr. Similar to BitKeeper, it is a fully distributed system which does not require a central server. Creating a local branch of an existing MySQL code tree was quite easy and it seems to be very similar in using it. What impressed me in particular was its capability of connecting to remote SVN repositories - it is actually possible to use bzr locally to work against a remote Subversion server! This should give us a good opportunity to start testing it against our existing SVN trees and gather some more practical experience with it.

All in all I was very impressed and excited to learn about all the cool things that are going on inside of the development team. Even though there is always a lot of work to be done and they're busy getting the MySQL 5.1 release out the door, they still find time to experiment and come up with great ideas and improvements in other areas as well. I got the impression that everybody really enjoyed being able to show off his work and also receiving good feedback from the others that were passing by. I hope we will repeat these team exhibitions at our next meeting!