MySQL: Partition-wise backups with mysqldump

To whom it may concern,

in response to a query from André Simões (also known as ITXpander), I slapped together a MySQL script that outputs mysqldump commands for backing up individual partitions of the tables in the current schema. The script is maintained as a snippet at MySQL Forge a gist on github.

How it works

The script works by querying the information_schema.PARTITIONS system view to generate an appropriate expression for mysqldump's --where option. The generated command also redirects the output to a file with this name pattern:

<schema>.<table>.<partition-name>.sql

For example, for this table (taken from the MySQL reference manual):

CREATE TABLE members (
    firstname VARCHAR(25) NOT NULL,
    lastname VARCHAR(25) NOT NULL,
    username VARCHAR(16) NOT NULL,
    email VARCHAR(35),
    joined DATE NOT NULL
)
PARTITION BY RANGE( YEAR(joined) ) (
    PARTITION p0 VALUES LESS THAN (1960),
    PARTITION p1 VALUES LESS THAN (1970),
    PARTITION p2 VALUES LESS THAN (1980),
    PARTITION p3 VALUES LESS THAN (1990),
    PARTITION p4 VALUES LESS THAN MAXVALUE
);

the script generates the following commands:

mysqldump --user=username --password=password --no-create-info --where=" YEAR(joined) < 1960" test members > test.members.p0.sql
mysqldump --user=username --password=password --no-create-info --where=" YEAR(joined) >= 1960 and  YEAR(joined) < 1970" test members > test.members.p1.sql
mysqldump --user=username --password=password --no-create-info --where=" YEAR(joined) >= 1970 and  YEAR(joined) < 1980" test members > test.members.p2.sql
mysqldump --user=username --password=password --no-create-info --where=" YEAR(joined) >= 1980 and  YEAR(joined) < 1990" test members > test.members.p3.sql
mysqldump --user=username --password=password --no-create-info --where=" YEAR(joined) >= 1990 and  YEAR(joined) < 18446744073709551615" test members > test.members.p4.sql

Tip: in order to obtain directly executable output from the mysql command line tool, run the script with the --skip-column-names (or -N) option.

Features

Currently, the script supports the following partitioning methods:

• HASH


• LIST


• RANGE

Limitations

The LINEAR HASH method is currently not supported, but I may implement that in the future.

Currently I do not have plans to implement the KEY and LINEAR KEY partitioning methods, but I may reconsider if and when I have more information about the storage-engine specific partitioning functions used by these methods.

Finally, I should point out that querying the information_schema.PARTITIONS table is dog-slow. This may not be too big of an issue, however it is pretty annoying. If anybody has some tips to increase performance, please let me know.

Acknowledgements

Thanks to André for posing the problem. I had a fun hour of procrastination to implement this, and it made me read part of the MySQL reference manual on partitioning.

I also would like to thank Giuseppe Maxia (the Datacharmer) for providing valuable feedback. If you're interested in either partitioning or the mysql command line, you should visit his tutorials at the MySQL conference, april 12-15, 2010.

"Starring Sakila": a Data Warehousing mini-tutorial at the MySQL UC 2009

Hi!

Recently I wrote about how glad I was to see two of my proposals for the upcoming MySQL Conference approved.

I am absolutely thrilled to see my third proposal has been approved as well: "Starring Sakila: data warehousing explained, illustrated and subtitled". I'm very proud to announce that I'm doing this talk together with Matt Casters, chief data integration of Pentaho and creator of the popular ETL/Data Integration suite Kettle (a.k.a Pentaho data integration).

During this 45-minute seminar, Matt and I will take the Sakila sample database and explain how to design and load a data warehouse for it, and how to use that to create those reports and charts that makes the managers smile ;-). Along the way, we'll explain often uttered terms and concepts such as dimensional model, denormalization, star schema, OLAP, data staging, business intelligence, ETL and data integration.

Note that this session is targeted at SQL developers, DBAs and tech-savvy analysts ('suits' are most welcome) that are new to business intelligence and datawarehousing. We noticed that although data warehousing and business intelligence are hardly new topics, there is still a lot of confusion surrounding them. The goal of this session is mainly to explain what it is all about, and to immediately make the concepts we cover tangible by illustrating it using a database schema which is easy to grasp, and which already familiar to most of the MySQL crowd.

Of course - the MySQL User's conference will feature more advanced data warehousing and business intelligence session too. I am pretty sure we will see a lot of Infobright and Kickfire. One session that I know will be very interesting is Matt's talk on utilizing computing clouds to power highly performing, scalable ETL processes.

Anyway, I'm looking forward to it. I will post a more detailed overview of the content of this and the other sessions I'm involved in when the conference date is nearing.

Video

A video recording of this session can be found here, courtesy of Sheeri.