Today I got to test the latest 64-bit ActiveState Perl distro for Windows, version 5.12.3.1204. I tested it on Windows 2008 R2 64-bit. I am happy to report that it works. I am not categorically recommend FOR the installation of 64-bit Perl on Windows, though.

Here are the steps:
1. Get the ActiveState Perl 64-bit package for Windows and install it, following all the default options;
2. On command prompt, do:
cd c:\perl64\bin
ppm install DBI
ppm install DBD-mysql

I then tested against both Oracle’s MySQL 5.5 Community Server and MariaDb’s 5.2.7 on Windows with MaatKit’s mk-table-checksum to confirm. And it worked fine:

C:\Users\Administrator\Downloads\maatkit-7540\maatkit-7540\bin>c:\Perl64\bin\perl.exe mk-table-checksum –databases mysql h=localhost,u=root,p=password

To get started, I thought it would be fun to rewrite a Perl utility I wrote before with Python. That script converts MySQL character sets to utf8, a very common task for wikis and blogs during an upgrade. This time, I did everything from scratch: firing up an Amazon EC2 Linux instance, hand install and configuring MySQL 5.1.50 (creating mysql user, group, wget tarball, setting directory ownership and permissions, creating symbolic to MySQL binaries, editing my.cnf, /etc/init.d/ and chkconfig automatic startup, environmental variables, the works), compiled and configured Python 2.7, compiled and configured Python easy_install, compiled and installed MySQLdb module for Python, and finally successfully rewrote and tested the utility in Python. It is listed at the bottom of this post.

The original Perl program used some SQL code generation technique based on metadata within information_schema. As is hopefully well-known by now, information_schema should be used with caution because it can leads to server lockup. In fact, I noticed it when I ran my Perl program on a shared web hosting server. See Baron’s post here for an alternative way of doing it. But my Python program listed below didn’t use it, just to be consistent with the one used in Perl. I had a lot of fun going through this exercise.

Here are some notes I took.
1. It would be really nice if there is a catalog of every publicly available AMIs that contains detailed metadata that we can query, such as:
a. Version and distro of *nix;
b. What has been installed? Apache, MySQL, Python, PHP, Nagios, etc., etc.
c. For the software that has been installed, what is the version? What version(s) of MySQL was installed? What patches have been applied? Is it Python 2.4, 2.5, 2.6, 2.7, 3.0? Things like that.
d. Hardware related stuff: memory, disk size, etc.

Granted, I used AWS Management Console web interface to fire up and terminate instances, and accessed the servers via Putty. I lost my Linux workstation and haven’t built a new one yet. I do recall there were some EC2 command line tools on Linux that can do some of that when I last tried it a few months back. So it could be that things have improved and/or I just didn’t know the right way to get that out. If that is the case, I’d really appreciate it if you can give me some pointers. As it stands, the xml manifest file is too vague and not detailed enough.

2. Trying to do Python upgrade, say from 2.4 to 2.6 or 2.7, is generally not a good idea. Having multiple versions of Python side by side is probably a better way to go. Yes, it can be confusing, but I think in most cases, the cost/benefit analysis weighs against an upgrade: broken libraries and modules, failed cron jobs, you name it.

When having multiple versions of Python side by side, you can define a default version by some creative use of symbolic link. For instance, I did
./configure && make && sudo make install
for Python 2.7 on a server that has 2.4 pre-installed. I then did:
ln -s /usr/local/bin/python2.7 /usr/bin/python
to make Python 2.7 the default version. If I need Python 2.4, I will just call python2.4 binary.

While on that note, I found it surprising that 2 AMI instances (ami-3e836657 and ami-01b75668) I used both had Python 2.4 installed, which in my mind was pretty old.

3. Installing MySQLdb Python module was a bit problematic. On one server,
sudo easy_install mysql-python
did not work for me. I didn’t spend time digging because the following worked:
sudo yum install MySQL-python
On another server, easy_install mysql-python did work, but when I did import MySQLdb, I got:
ImportError: libmysqlclient_r.so.16: cannot open shared object file: No such file or directory
To fix that, do:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/mysql/lib
Here is the Python code

import MySQLdb
db = MySQLdb.connect(host="localhost", user="root", passwd="", db="charset_test", unix_socket="/tmp/mysql.sock")
cursor = db.cursor()
sql = "SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', REPLACE(column_type, 'char', 'binary')) FROM information_schema.columns WHERE table_schema = 'charset_test' and data_type LIKE '%char%'"
cursor.execute(sql)
results = cursor.fetchall()
for row in results:
        """Convert char types to binary types first"""
        cursor.execute(row[0])
        """Now we convert binary type back to char with CHARACTER SET utf8 defined"""
        cursor.execute(row[0].rpartition(' ')[0] + ' ' + row[0].rpartition(' ')[2].replace('binary', 'char') + ' CHARACTER SET utf8')
sql = "SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', REPLACE(column_type, 'text', 'blob')) FROM information_schema.columns WHERE table_schema = 'MyDatabase' and data_type LIKE '%text%'"
cursor.execute(sql)
results = cursor.fetchall()
for row in results:
        """Convert text types to blob types first"""
        cursor.execute(row[0])
        """Now we convert blob type back to text with CHARACTER SET utf8 defined"""
        cursor.execute(row[0].rpartition(' ')[0] + ' ' + row[0].rpartition(' ')[2].replace('blob', 'text') + ' CHARACTER SET utf8')
db.close()

Notes:
1. It does put BEGIN and END before and after the procedure creation. Because some script has grant statements in it, and I want to put END right after the procedure creation, therefore there is a variable called $FirstGrant;
2. The new files created will be named like StoredProcedureName.sql.

#!/usr/bin/perl
use strict;
open(MYDATA, "c:\\junk\\MyDb\\AllProc.sql") or
  die("Error: cannot open file 'AllProc.sql'\n");
    my $FirstGrant = 0;
while( my $line = <mydata> ){
if ($line =~ /^create\s+proc/i) {
   close (FILE);
   $FirstGrant = 0;
   (my $filename) = $line =~ /create\s+proce?d?u?r?e?\s+\[?d?b?o?]?\.?\[?(\w+)/i;
   open (FILE, ">", "c:\\junk\\MyDb\\".$filename.".sql");
   print FILE "IF NOT EXISTS (SELECT * FROM dbo.sysobjects WHERE id = OBJECT_ID(N'[dbo].[$filename]') AND OBJECTPROPERTY(id,N'IsProcedure') = 1)\n";
   print FILE "BEGIN\n";
   print FILE $line;
}
else {
if (($line =~ /^grant\s+execute/i) && ($FirstGrant == 0)) {
   print FILE "END\n";
   $FirstGrant = 1;
}
   print FILE $line;
}
}
close MYDATA;

A few things worth mentioning here:
1. This script uses a text file, TableList, that contains a list of tables that you want insert statements created. Each line in the list contains the database name and table name, separated by space(s). This is a quick hack, one could easily pass the file name as an input;
2. Obviously the connection information needs to be modified for your own needs;
3. The script does escape single quote in character fields properly;
4. Use
perl ScriptName.pl > c:\MyFolder\MyTableInsert.sql
to collect the results into c:\MyFolder\MyTableInsert.sql file.
5. Thanks to Summit Amar and people who commented on this article for sql script ideas.
6. Update: modified the script so it handles identity field if it is present in a table.

By the way, Microsoft’s new scripting language, PowerShell, borrowed a lot of good stuff from Perl and Unix shell scripting, which is pretty powerful. My friends Yan Pan and Ananthakumar Muthusamy wrote a book, Microsoft SQL Server 2008 Administration with Windows PowerShell, which I helped to tech-edit. If you are looking to learn more automation and scripting tricks with PowerShell, definitely check out that book, which will come out next month.

#!/usr/bin/perl
use strict;
use Win32::SqlServer;
# Get server name from the command line
# my $ServerName = shift or die "Please enter server name as the first only parameter needed";
# my $TableName = "authors";
open(MYDATA, "RefList.txt") or
  die("Error: cannot open file 'RefList.txt'\n");
while( my $line = <mydata> ){
  $line =~ /(\w+)\s+(\w+)/;
  my $db;
  my $table;
  if ($db ne $1){
	print "-----Tables in $1-----\n";
  }
  $db = $1;
  my $sqlsrv = sql_init(".", "sa", "s3cr3t", $1);
  my $HasIdentityColumn = $sqlsrv->sql_one("select objectproperty(object_id('$2'), 'TableHasIdentity')");
  if ($HasIdentityColumn == 1) {
	print "SET IDENTITY_INSERT $2 ON\n";
  }
# Our SQL statement.
  my $stmnt = < <SQLEND;
--Declare a cursor to retrieve column specific information for the specified table
DECLARE cursCol CURSOR FAST_FORWARD FOR
SELECT column_name,data_type FROM information_schema.columns WHERE table_name = '$2'
OPEN cursCol
DECLARE \@string nvarchar(3000) --for storing the first half of INSERT statement
DECLARE \@stringData nvarchar(3000) --for storing the data (VALUES) related statement
DECLARE \@dataType nvarchar(1000) --data types returned for respective columns
SET \@string='INSERT '+ '$2' +'('
SET \@stringData=''
DECLARE \@colName nvarchar(50)
FETCH NEXT FROM cursCol INTO \@colName,\@dataType
IF \@\@fetch_status<>0
	begin
	--print 'Table ' + $2 + ' not found, processing skipped.'
	close curscol
	deallocate curscol
	return
END
WHILE \@\@FETCH_STATUS=0
BEGIN
IF \@dataType in ('varchar','char','nchar','nvarchar')
BEGIN
        SET \@stringData=\@stringData+''''+'''+isnull('''''+'''''+REPLACE('+\@colName+','''''+''''','''''+'''''+'''''+''''')+'''''+''''',''NULL'')+'',''+'
END
ELSE
if \@dataType in ('text','ntext') --if the datatype is text or something else
BEGIN
        SET \@stringData=\@stringData+'''''''''+isnull(REPLACE(cast('+\@colName+' as varchar(2000)),'''''+''''','''''+'''''+'''''+'''''),'''')+'''''',''+'
END
ELSE
IF \@dataType = 'money' --because money doesn't get converted from varchar implicitly
BEGIN
	SET \@stringData=\@stringData+'''convert(money,''''''+isnull(cast('+\@colName+' as varchar(200)),''0.0000'')+''''''),''+'
END
ELSE
IF \@dataType='datetime'
BEGIN
	SET \@stringData=\@stringData+'''convert(datetime,'+'''+isnull('''''+'''''+convert(varchar(200),'+\@colName+',121)+'''''+''''',''NULL'')+'',121),''+'
END
ELSE
IF \@dataType='image'
BEGIN
	SET \@stringData=\@stringData+'''''''''+isnull(cast(convert(varbinary,'+\@colName+') as varchar(6)),''0'')+'''''',''+'
END
ELSE --presuming the data type is int,bit,numeric,decimal
BEGIN
	--SET \@stringData=\@stringData+'''''''''+isnull(cast('+\@colName+' as varchar(200)),''0'')+'''''',''+'
	--SET \@stringData=\@stringData+'''convert(datetime,'+'''+isnull('''''+'''''+convert(varchar(200),'+\@colName+',121)+'''''+''''',''NULL'')+'',121),''+'
	SET \@stringData=\@stringData+''''+'''+isnull('''''+'''''+convert(varchar(200),'+\@colName+')+'''''+''''',''NULL'')+'',''+'
END
SET \@string=\@string+\@colName+','
FETCH NEXT FROM cursCol INTO \@colName,\@dataType
END
DECLARE \@Query nvarchar(4000)
SET \@query ='SELECT '''+substring(\@string,0,len(\@string)) + ') VALUES(''+ ' + substring(\@stringData,0,len(\@stringData)-2)+'''+'')'' Col1 FROM '+ '$2'
exec sp_executesql \@query
CLOSE cursCol
DEALLOCATE cursCol
SQLEND
# Run query. The return value is a reference to an array.
  my $result = $sqlsrv->sql($stmnt);
# Print results. Each array entry is a hash reference.
  foreach my $row (@$result) {
	print "$$row{Col1}\n";
  }
  if ($HasIdentityColumn == 1) {
	print "SET IDENTITY_INSERT $2 OFF\n";
  }
}
close MYDATA;

I then downloaded the source code, manually modified the MAKE file, fiddled with Visual Studio NMAKE, compiled it a few times, without success.

Google revealed the existence of Strawberry Perl. So I removed ActiveState Perl, installed Strawberry, ran ppm install DBI, ppm install DBD-mysql. And it worked.

Fast-forward a few weeks, I started playing with EPIC, a plugin for Eclipse that supposedly provides a nice IDE for Perl development and debugging. As a result, I started fiddling with PadWalker, a prerequisites for EPIC. I couldn’t remember all the details now due to frustration, but suffice it to say that I wrestled with EPIC, Eclipse, and Strawberry so much that I thought maybe I should gave ActiveState another look. Plus, I always wanted to verify for myself what the deal is with ActiveState Perl and DBD-mysql.

So I un-installed Strawberry Perl 5.10, reinstalled ActiveState Perl 5.10. And just like last time, DBD-mysql wouldn’t install. After un-installing ActiveState, reinstalling Strawberry, running ppm install DBD-mysql, I am back in business.

So, if you want to do Perl scripting with MySQL on Windows using DBD-mysql, pick Strawberry Perl, not ActiveState.

perl -MMyModule -le 'print "$MyModule::VERSION\n"'

For instance, to find out the version number of DBI installed on the machine, run

perl -MDBI -le 'print "$DBI::VERSION\n"'

To find out the version of DBD::mysql, run

perl -MDBD::mysql -le 'print "$DBD::mysql::VERSION\n"'

One caveat: this process does not seem to work on Windows, at least for my Strawberry Perl 5.10 installed on my Windows XP. I had to use the script below on Windows to get the version information. Here is the error message when I tried:

C:\junk>perl -MDBI -le ‘print “$DBI::VERSION\n”‘
Can’t find string terminator “‘” anywhere before EOF at -e line 1.

#!/usr/bin/perl
use DBI;
use DBD::mysql;
print "The version for DBI is $DBI::VERSION\n";
print "The version for DBD::mysql is $DBD::mysql::VERSION\n";

It seems converting character and text data to utf8 character set is a common task for MySQL, especially during an upgrade. In fact, I had trouble with it during server and WordPress database upgrade for this blog site. I wrote about it in this post, where I explained how to do it step-by-step using a command line tool such as mysql, taking advantage of some nifty code generation trick with information_schema.

One drawback of that method is that it’s a manual process, therefore time-consuming and error-prone.

I spent some time today to cook up a simple Perl script to automate this task. See below. Remember to change MyDatabase to your database name (there are 3 places that need to be changed), and adjust the user name and password accordingly. This script also prints out sql statements used during the process so you will know what has been done to the database.

I have tested this successfully on my own WordPress blog. Hopefully it will help somebody out there. Enjoy!

#!/usr/bin/perl
# MyUtf8Converter.pl - convert all character data to utf8 character set
use strict;
use warnings;
use DBI;
# data source name, username, password, connection attributes
my $dsn = "DBI:mysql:MyDatabase:localhost";
my $user_name = "MySQLUserName";
my $password = "MySQLPassword";
my %conn_attrs = (RaiseError => 1, PrintError => 0, AutoCommit => 1);
# connect to database
my $dbh = DBI->connect ($dsn, $user_name, $password, \%conn_attrs);
# Convert char types first. Use information_schema to generate alter table code to convert data to binary first.
my $sth = $dbh->prepare ("SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', REPLACE(column_type, 'char', 'binary')) FROM information_schema.columns WHERE table_schema = 'MyDatabase' and data_type LIKE '%char%'");
$sth->execute ();
# Run alter table stamement. First convert char types to binary types, then convert those to char types utf8 character set
while (my ($val) = $sth->fetchrow_array ())
    {
        print $val, "\n";
        $dbh->do ($val);
        $val =~ s/binary/char/;
        $val = $val . " CHARACTER SET utf8";
        print $val, "\n";
        $dbh->do ($val);
    }
$sth->finish ();
# Now we convert text data. Use information_schema to generate alter table code to convert data to blob first.
$sth = $dbh->prepare ("SELECT CONCAT('ALTER TABLE ', table_name, ' MODIFY ', column_name, ' ', REPLACE(column_type, 'text', 'blob')) FROM information_schema.columns WHERE table_schema = 'MyDatabase' and data_type LIKE '%text%'");
$sth->execute ();
# Run alter table stamement. First convert text types to blob types, then convert those to text types utf8 character set
while (my ($val) = $sth->fetchrow_array ())
    {
        print $val, "\n";
        $dbh->do ($val);
        $val =~ s/blob/text/;
        $val = $val . " CHARACTER SET utf8";
        print $val, "\n";
        $dbh->do ($val);
    }
$sth->finish ();
$dbh->disconnect ();

• Internet Explorer: Tools -> Internet Options-> LAN Settings -> “Use automatic configuration script”;
• Firefox 3: Tools -> Options -> Advanced -> Network -> Settings… -> Automatic proxy configuration URL:

Sometimes you need to know a proxy server name or IP address. For instance, if you use Perl and would like to do an automatic binary install of a module, then you will need to define a proxy for cpan to use. This is the command to do that:

set HTTP_proxy=http://my.proxy.server:8000/

So obviously you need to know a proxy server name or IP address. To find that out, you need to copy the location where the automatic configuration script is located, put that address into your browser and open it. It will prompt you to save it as a file. Do that, then open the file up to search for proxy server names or IP addresses.

As a side, I was not able to install DBD::mysql on a Windows XP machine with cpan. I also tried to do manual installation from source code, with no success. I use ActiveState Perl 5.10 and 5.1.30-community MySQL Community Server (GPL). I stumbled into Strawberry Perl while doing research for that issue. Maybe I will check it out in the future.

#Part 1
use File::Copy;
use File::stat;
use SQLDBA::Utility qw( dbaRunOsql );  # import the function
$dirname = shift or die "Please provide a directory to search for";
$timediff=0;
opendir DIR, "$dirname";
while (defined ($file = readdir(DIR)))
{
	if($file ne "." && $file ne "..")
	{
		$diff = time()-stat("$dirname/$file")->mtime;
		if($timediff == 0)
                       {
                       	$timediff=$diff;
                       	$newest=$file;
                       }
                if($diff< $timediff)
		       {
                       	$timediff=$diff;
                       	$newest=$file;
                       }
        }
}
print $newest,"\n";
copy("$dirname/$newest", "g:/$newest");
#Part 2
# Construct the SQL script
my $sql =<<__SQL__;
SET NOCOUNT ON
USE Master
DECLARE \@strSQL varchar(255)
PRINT 'Killing Users'
PRINT '-----------------'
CREATE table #tmpUsers(
spid smallint,
ecid smallint,
status varchar(30),
loginname varchar(128),
hostname varchar(128),
blk char(5),
dbname nchar(50),
cmd nchar(16))
INSERT INTO #tmpUsers EXEC SP_WHO
DECLARE LoginCursor CURSOR
READ_ONLY
FOR SELECT spid, dbname FROM #tmpUsers WHERE dbname = 'MyDatabase'
DECLARE \@spid varchar(10)
DECLARE \@dbname2 varchar(40)
OPEN LoginCursor
FETCH NEXT FROM LoginCursor INTO \@spid, \@dbname2
WHILE (\@\@fetch_status <> -1)
BEGIN
	IF (\@\@fetch_status <> -2)
	BEGIN
	PRINT 'Killing ' + \@spid
	SET \@strSQL = 'KILL ' + \@spid
	EXEC (\@strSQL)
	END
	FETCH NEXT FROM LoginCursor INTO  \@spid, \@dbname2
END
CLOSE LoginCursor
DEALLOCATE LoginCursor
DROP table #tmpUsers
PRINT 'Done'
-- The line below is the command to restrict access (behine EM) to dbo (schema owner) and sysadm (sys user):
--exec sp_dboption N\@dbname, N'dbo use', N'true'
--exec sp_dboption N\@dbname, N'dbo use', N'false'
EXEC master.dbo.xp_restore_database \@database='MyDatabase'
  , \@filename = 'G:\\$newest'
GO
__SQL__
# Prepare the osql command line options
my $optRef = {
      '-E' => undef,
      '-n' => undef,
      '-w' => '1024',
      '-d' => 'master',
      '-l' => '5'
   };
# Execute the SQL script on the APOLLO named instance on the local server
my $result = dbaRunOsql('MyServerName', $sql, $optRef);
if (!defined $result) {
   die "***Err: failed to run osql.exe.";
}
else {
   print $result;  # Print the results
}

Here is one way to do it. Note it uses sysaltfiles, a system table that might be deprecated in future releases. For Sql Server 2005, it is recommended that you use sys.master_file:

select sum(size)*8/1024 as 'TotalSizeInMeg', sum(size)*8/1024/1024 as 'TotalSizeInGig' from sysaltfiles

Another way to get that is to use sp_helpdb. Executing sp_helpdb without any parameters will give you total size for each database in megabytes. Adding them up will give you total databases size.

Since I am digging Perl, below is the Perl script to do that. You might notice the numbers you get from sysaltfiles and sp_helpdb are different, that’s due to rounding.

use strict;
use Win32::SqlServer;
# Get server name from the command line
my $ServerName = shift or die "Please enter server name as the first only parameter needed";
# Create object and login in one step with
# integrated security.
my $sqlsrv = sql_init($ServerName, undef, undef, 'master');
# Our SQL statement. In this case, this is a system stored procedure.
my $stmnt = < <SQLEND;
sp_helpdb
SQLEND
# Run query. The return value is a reference to an array.
my $result = $sqlsrv->sql_sp($stmnt);
my $TotalSize = 0;
# Print results. Each array entry is a hash reference.
foreach my $row (@$result) {
   print "$$row{name}: $$row{db_size}  \n";
   if ($$row{db_size} =~ /(\d+\.\d+)/ix) {
	$TotalSize = $TotalSize + $1;
   }
}
print "Total databases size is $TotalSize meg";