The Ji Village News » Windows http://www.haidongji.com 季庄新闻--Haidong Ji's Blog Mon, 30 Jan 2012 02:41:37 +0000 en hourly 1 http://wordpress.org/?v=3.1.2 Generating dimension data for dateshttp://www.haidongji.com/2011/07/30/generating-dimension-data-for-dates/ http://www.haidongji.com/2011/07/30/generating-dimension-data-for-dates/#comments Sun, 31 Jul 2011 04:12:06 +0000 Haidong Ji http://www.haidongji.com/?p=1238 Most analytical and BI databases have date dimension table(s). One frequently needs to generate and populate such data. I present a solution below for such data generation, written in Python. Please use different database drivers/modules to connect to your specific database server (MySQL, SQL Server, Oracle, etc.) for data population.

Notes:

1. It takes 2 parameters, start date and end date, in YYYYMMDD format, inclusive. Extensive error checking is built in, but let me know if you have comments/suggestions;

2. The script produce a Python dictionary (associated array) and print out its content;

3. The output includes dayNumber: a day’s position in a year. For example, 2011-02-01 is the 32ed day in 2011, therefore its dayNumber is 32;

4. The output includes weekNumber: a week’s position in a year. The week number in year is based on ISO standard. From documentation: the ISO year consists of 52 or 53 full weeks, where a week starts on a Monday and ends on a Sunday. The first week of an ISO year is the first (Gregorian) calendar week of a year containing a Thursday. This is called week number 1, and the ISO year of that Thursday is the same as its Gregorian year.

So, 2011-01-01 has the weekNumber 52, because it falls on a Saturday and belongs to the last week of 2010.

5. The output includes weekday information as well. 4 different variations are included:
Sunday 0, Monday 1, and so on
Sunday 1, Monday 2, and so on
Monday 0, Tuesday 1, and so on
Monday 1, Tuesday 2, and so on

6. The script requires the argparse module. It comes with Python 2.7. Python version prior to 2.7 does not have it by default, therefore you need to install it.

import argparse, sys, time
from datetime import date, timedelta
parser = argparse.ArgumentParser(description="Generating date dimension data")
parser.add_argument('-s', '--startDate', help='Start date in YYYYMMDD format', required=True, dest='startDate')
parser.add_argument('-e', '--endDate', help='end date in YYYYMMDD format', required=True, dest='endDate')
argList = parser.parse_args()
if (((not argList.startDate.isdigit()) or (not (len(argList.startDate) == 8))) or ((not argList.endDate.isdigit()) or (not (len(argList.endDate) == 8))) or (argList.startDate > argList.endDate)):
	print "Input(s) must be numeric in YYYYMMDD format and end date must not be earlier than start date"
	sys.exit (1)
try:
	startDate = date(int(argList.startDate[0:4]), int(argList.startDate[4:6]), int(argList.startDate[6:8]))
	endDate = date(int(argList.endDate[0:4]), int(argList.endDate[4:6]), int(argList.endDate[6:8]))
except ValueError:
	print "Input(s) must be valid date value in YYYYMMDD format"
	sys.exit (1)
start = time.time()
while startDate <= endDate:
	dateInfo = {'dateYYYYMMDD': startDate.strftime('%Y%m%d'), 'calDate': startDate.strftime('%Y-%m-%d'), 'calDay': startDate.day, 'calMonth': startDate.month, 'calYear': startDate.year}
	dateInfo['dayOfWeekSunday0Monday1'] = startDate.isoweekday() % 7
	dateInfo['dayOfWeekSunday1Monday2'] = startDate.isoweekday() % 7 + 1
	dateInfo['dayOfWeekSunday6Monday0'] = startDate.weekday()
	dateInfo['dayOfWeekSunday7Monday1'] = startDate.isoweekday()
	dateInfo['dayNumber'] = startDate.toordinal() - date(startDate.year - 1, 12, 31).toordinal()
	dateInfo['weekNumber'] = startDate.isocalendar()[1]
	print dateInfo
	startDate = startDate + timedelta(1)
]]> http://www.haidongji.com/2011/07/30/generating-dimension-data-for-dates/feed/ 1 Sysinternals and PALhttp://www.haidongji.com/2011/07/10/sysinternals-and-pal/ http://www.haidongji.com/2011/07/10/sysinternals-and-pal/#comments Sun, 10 Jul 2011 17:19:09 +0000 Haidong Ji http://www.haidongji.com/?p=1229 Sysinternals and PAL (Performance Analysis of Logs) are two fantastic tools for general server information gathering and troubleshooting on Windows.

Sysinternals suite is a set of tools that can be downloaded freely from Microsoft. One thing that is particularly attractive about them is that they can be run directly after downloading without special installation and all the footprints a typical installation leaves on the host machine (new directories under C:\Program Files\, registry entries, data files and what have you). I found them very valuable and handy.

In particular, psInfo provides good summary information of the server. For example, psinfo -s -h -d provides basic information about the system, software installed, Windows hot fixes installed, and disk volume information.

PAL: install PAL on your test/analysis/general purpose machine. Install the mschart control as it is a prerequisite of PAL. Here is how I used it:

1. Produce Perfmon data gathering template files using PAL. I exported 3 template files: overview, quick overview, and SQL Server 2005/2008;

Perfmon is the general-purpose data instrumentation tool on Windows. Through Perfmon you can gather system wide counters for things like CPU, memory, network, and disk IO. In addition, a lot of applications such as SQL Server, Exchange, and others, expose application level Instrumentation data such that you can collect them via Perfmon as well.

It is best to have a few handy data collection template, hence this step.

2. On the Windows server that I am interested in monitoring, import Perfmon counter template file produced above by opening a DOS prompt under Administrator and executing:

logman import -n templateNameIdefine -xml pathAndName2TemplateXmlFile

3. Open Perfmon, find the one you imported, and start collecting

4. After collection is done, copy the log file and use PAL for analysis. It will generate a very nice and intuitive report. Please don’t run PAL on the system you are diagnosing. Run it somewhere else. Be patient, as it will take a while for PAL to churn through the data (it took 2 hours on a Rackspace cloud server with 2 CPUs and 1 gig of RAM for a file about 30 meg)

]]> http://www.haidongji.com/2011/07/10/sysinternals-and-pal/feed/ 0 Installing Perl DBI and DBD-mysql on Windows 64 bithttp://www.haidongji.com/2011/06/20/installing-perl-dbi-and-dbd-mysql-on-windows-64-bit/ http://www.haidongji.com/2011/06/20/installing-perl-dbi-and-dbd-mysql-on-windows-64-bit/#comments Mon, 20 Jun 2011 21:28:33 +0000 Haidong Ji http://www.haidongji.com/?p=1214 I had trouble getting Perl DBI and DBD-mysql on Windows in the past. In addition, on Windows 64-bit, you sometimes see recommendations of using 32-bit Perl.

Today I got to test the latest 64-bit ActiveState Perl distro for Windows, version 5.12.3.1204. I tested it on Windows 2008 R2 64-bit. I am happy to report that it works. I am not categorically recommend FOR the installation of 64-bit Perl on Windows, though.

Here are the steps:
1. Get the ActiveState Perl 64-bit package for Windows and install it, following all the default options;
2. On command prompt, do:
cd c:\perl64\bin
ppm install DBI
ppm install DBD-mysql

I then tested against both Oracle’s MySQL 5.5 Community Server and MariaDb’s 5.2.7 on Windows with MaatKit’s mk-table-checksum to confirm. And it worked fine:

C:\Users\Administrator\Downloads\maatkit-7540\maatkit-7540\bin>c:\Perl64\bin\perl.exe mk-table-checksum –databases mysql h=localhost,u=root,p=password

]]> http://www.haidongji.com/2011/06/20/installing-perl-dbi-and-dbd-mysql-on-windows-64-bit/feed/ 0 Install MySQLdb module for Pythonhttp://www.haidongji.com/2011/04/04/install-mysqldb-module-for-python/ http://www.haidongji.com/2011/04/04/install-mysqldb-module-for-python/#comments Tue, 05 Apr 2011 03:06:22 +0000 Haidong Ji http://www.haidongji.com/?p=1130 Update:

Commenter MarkR made a great point: if possible, use some packaging tools, to try to maintain proper dependencies, to the extent that is possible. Install from the source should be Plan B. So, try yum install MySQL-python first.

This is mostly for my own future reference. It’ll be icing on the cake if it helps you!

This is geared for CentOS or Red Hat. Use apt-get or other packaging tools for different flavours of Linux.

1. Get Python module setuptools called easy_install. I love easy_install, by the way, sort of like CPAN for Perl modules;
2. To install MySQLdb package, you would think easy_install MySQLdb would do. But that is not the case. I hope the developer would fix that. Instead, you need:

easy_install MySQL-python

3. If you have build errors, you may need:
yum install python-devel or yum install gcc or both.

Update: one more tip. If you encounter:

gcc: error trying to exec ‘cc1plus’: execvp: No such file or directory, you probably need:

yum install gcc-c++

]]> http://www.haidongji.com/2011/04/04/install-mysqldb-module-for-python/feed/ 2 Windows disk throughput benchmark steps with sqliohttp://www.haidongji.com/2011/02/16/windows-disk-throughput-benchmark-steps-with-sqlio/ http://www.haidongji.com/2011/02/16/windows-disk-throughput-benchmark-steps-with-sqlio/#comments Thu, 17 Feb 2011 04:27:55 +0000 Haidong Ji http://www.haidongji.com/?p=1073 Credits to Linchi Shea, SQLServerPedia/Brent Ozar, and Jonathan Kehayias for their discussions on sqlio.

Update: The test below, against one drive, can run for more than 1 hour. That is a long time. I’ve tested running 2 sqlio batch scripts against 2 locally attached drives, similar to the ones listed below, and found the results to be consistent with running those 2 concurrently.

1. Download and install sqlio;
2. Modify param.txt in the same directory where sqlio is. Create a testing file that is 24 gig, or sufficiently large to beat hardware cache along the way. For example, to benchmark drive m, the entry should look like this:

m:\testfile.dat 2 0x0 20480

If you use mount point, it should look like this:

m:\mountPointDirectoryName\testfile.dat 2 0x0 20480

3. Make sure file in step 2 is saved. In DOS command line, under the sqlio installation directory, run the command below. It will create a file for use later:

sqlio -kW -s10 -fsequential -o8 -b8 -LS -Fparam.txt timeout /T 10

4. Create a batch file with the following line as content, let’s call it mDrive.bat. SQLServerPedia article lets each test run 120 seconds, 2 minutes. In my testing against local disks, I made it 30 seconds. Actually I ran both 120 and 30 and didn’t see much of a difference. So I will use 30 seconds.

Note: to test against Windows mount point, the lines below should be like this:

sqlio -kW -t2 -s30 -dM -o1 -frandom -b64 -BH -LS \mountPointDirectoryName\Testfile.dat

sqlio -kW -t2 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat

5. In DOS command line, under the sqlio installation directory, run the command below. It takes more than 1 hour, but less than 2 hours to finish:

mDrive.bat > mDriveSqlioResults.txt

6. Run this Python script to parse mDriveSqlioResults.txt into a CSV file;

7. Open the CSV file with Excel, create a new sheet, insert a column chart. Drag the chart to make it decent size. Right click on “Select Data…” or something like that, then go back to the first sheet to pick 2 columns of data for this chart. I found charts with MBPS, IOPS, avgLatencyInMS, and maxLatencyInMS are helpful.

]]> http://www.haidongji.com/2011/02/16/windows-disk-throughput-benchmark-steps-with-sqlio/feed/ 0 Parse sqlio log file with Python pyparsinghttp://www.haidongji.com/2011/02/15/parse-sqlio-log-file-with-python-pyparsing/ http://www.haidongji.com/2011/02/15/parse-sqlio-log-file-with-python-pyparsing/#comments Wed, 16 Feb 2011 04:44:44 +0000 Haidong Ji http://www.haidongji.com/?p=1066 A few weeks ago I posted some questions on 3 Python text processing modules: pyparsing, SimpleParse, and NLTK. Today I need to analyze a log file generated by sqlio. I decided to use pyparsing. I am pretty pleased with it.

I am aware that there are at least 2 utility scripts for this: one is in Perl and then other is in Windows PowerShell. But I wanted to write in Python.

Noteworthy points:

1. In analyzing and building up the grammar/structure of the text for Python, I opted for line by line analysis and coding. I think this brings clarity and is easier to read. It felt a bit tedious in building it up, but once it is done, it is pretty rewarding to see the end results;
2. After parsing, the script writes the results to a csv file. You can then open that in a spreadsheet program to create some charts. Seeing the results in a column diagram chart brings clarity and focus to the result. It worked pretty well for me.

All in all, a good and productive day. Below are the details. The top part is the sample log content to be parsed, followed by the program.

"""
c:\Program Files (x86)\SQLIO>sqlio -kW -t2 -s30 -dH -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio v1.5.SG
using system counter for latency timings, 2208037 counts per second
2 threads writing for 30 secs to file H:Testfile.dat
	using 64KB random IOs
	enabling multiple I/Os per thread with 1 outstanding
	buffering set to use hardware disk cache (but not file cache)
using current size: 24576 MB for file: H:Testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  6106.50
MBs/sec:   381.65
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 5
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 99  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
c:\Program Files (x86)\SQLIO>sqlio -kW -t2 -s30 -dH -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio v1.5.SG
using system counter for latency timings, 2208037 counts per second
2 threads writing for 30 secs to file H:Testfile.dat
	using 64KB random IOs
	enabling multiple I/Os per thread with 2 outstanding
	buffering set to use hardware disk cache (but not file cache)
using current size: 24576 MB for file: H:Testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  6951.19
MBs/sec:   434.44
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 6
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 87 12  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
"""
from pyparsing import *
output = open("sample.csv", "w")
input = open("hDriveSqlioResults.txt", "r")
data = input.read()
#------------------------------------------------------------------------
# Define Grammars
#------------------------------------------------------------------------
integer = Word(nums)
singleLetter = Word(alphas, max=1)
realNumber = Combine(Word(nums) + '.' + Word(nums))
reading = Literal("reading")
writing = Literal("writing")
readingOrWriting = reading | writing
random = Literal("random")
sequential = Literal("sequential")
randomOrSequential = random | sequential
#Line 1 is useful. We want to suck out the juicy parts.
sectionBeginning = CaselessLiteral("c:\\program files (x86)\\sqlio>sqlio ")
ioKind = "-k" + singleLetter("k")
numThreads = "-t" + integer("t")
numSeconds = "-s" + integer("s")
driveOrMountPath = "-d" + singleLetter("d")
outstandingRequests = "-o" + integer("o")
accessType = "-f" + Word(alphas)("f")
ioBlockSizeInKB = "-b" + integer("b")
#Line 2 can be safely ignored
line2 = "sqlio v" + restOfLine
#Line 3 can be safely ignored
line3 = "using system counter for latency timings" + restOfLine
#Line 4 can be safely ignored
line4 = integer + "threads" + readingOrWriting + restOfLine
#Line 5 can be safely ignored
line5 = "using" + integer + "KB" + randomOrSequential + "IOs" + restOfLine
#Line 6 can be safely ignored
line6 = "enabling multiple " + restOfLine
#Line 7 can be safely ignored
line7 = "buffering set to use" + restOfLine
#Line 8 we should get the file size out
line8 = "using current size: " + integer("fileSize") + Word(alphas)("fileSizeUnit") + restOfLine
#Line 9 can be safely ignored
line9 = "initialization done" + restOfLine
#Line 10 can be safely ignored
line10 = "CUMULATIVE" + restOfLine
#Line 11 can be safely ignored
line11 = "throughput" + restOfLine
#Line 12 we want IOPS
line12 = "IOs/sec:" + realNumber("IOPS") + restOfLine
#Line 13 we want MBPS
line13 = "MBs/sec:" + realNumber("MBPS") + restOfLine
#Line 14 can be safely ignored
line14 = "latency" + restOfLine
#Line 15 we need to get minLatency out
line15 = "Min_Latency(ms):" + integer("minLatency") + restOfLine
#Line 16 we need to get avgLatency out
line16 = "Avg_Latency(ms):" + integer("avgLatency") + restOfLine
#Line 17 we need to get maxLatency out
line17 = "Max_Latency(ms):" + integer("maxLatency") + restOfLine
#Line 18 can be safely ignored
line18 = "histogram" + restOfLine
#Line 19 can be safely ignored
line19 = "ms:" + restOfLine
#Line 20 can be safely ignored
line20 = "\%:" + restOfLine
extraStuff = ZeroOrMore(line14 | line15 | line16 | line17 | line18 | line19 | line20)
logEntry = sectionBeginning + ioKind + numThreads + numSeconds + driveOrMountPath + outstandingRequests + accessType + ioBlockSizeInKB + restOfLine + line2 + line3 + line4 + line5 + line6 + line7 + line8 + line9 + line10 + line11 + line12 + line13 + extraStuff
output.write("IO property, IOPS, MBPS, minLatencyInMS, avgLatencyInMS, maxLatencyInMS\n")
for tokens in logEntry.searchString(data):
	output.write("%(k)s%(t)s threads %(o)s queue %(f)s in %(b)s KB chunks,%(IOPS)s,%(MBPS)s,%(minLatency)s,%(avgLatency)s,%(maxLatency)s\n" % tokens)
]]> http://www.haidongji.com/2011/02/15/parse-sqlio-log-file-with-python-pyparsing/feed/ 5 Search objects in SQL Server databases with pyodbchttp://www.haidongji.com/2011/01/05/search-objects-in-sql-server-databases-with-pyodbc/ http://www.haidongji.com/2011/01/05/search-objects-in-sql-server-databases-with-pyodbc/#comments Thu, 06 Jan 2011 00:08:59 +0000 Haidong Ji http://www.haidongji.com/?p=1032 Command line utility to search for objects based on names. Search results will be displayed in well formatted tabular format, left-justified.

1. pyodbc installed;
2. Use trusted authentication by default. Find the relevant code to adjust to login/password as needed;
3. -w for wild card search. Default is exact name match, case insensitive in almost all cases, depending on your SQL Server configuration;
4. If -d is not specified, all databases in instance will be searched, minus the ones that one not accessible at the moment. To specify database(s) you want to search into, use -d followed by database names. Separate them with comma if more than one;
5. Use -S to specify instance;
6. Good sample code for command line parameters processing and pyodbc usage;
7. Won’t work with SQL Server 2000 or lower.

Example 1: python objectSearch.py myobject -S instance1
Result: Database objects that are named myobject in instance1 will be displayed

Example 2: python objectSearch.py -S server1 -w -d db1,db2 myobject
Result: Database objects that has “myobject” in db1 and db2 in server1 will be displayed

import pyodbc, argparse
def pp(cursor, data=None, check_row_lengths=True):
    if not data:
        data = cursor.fetchall( )
    names = [  ]
    lengths = [  ]
    rules = [  ]
    for col, field_description in enumerate(cursor.description):
        field_name = field_description[0]
        names.append(field_name)
        field_length = field_description[2] or 12
        field_length = max(field_length, len(field_name))
        if check_row_lengths:
            # double-check field length, if it's unreliable
            data_length = max([ len(str(row[col])) for row in data ])
            field_length = max(field_length, data_length)
        lengths.append(field_length)
        rules.append('-' * field_length)
    format = " ".join(["%%-%ss" % l for l in lengths])
    result = [ format % tuple(names), format % tuple(rules) ]
    for row in data:
        result.append(format % tuple(row))
    return "\n".join(result)
def objectMatch(wildcardSearch, objectNamePattern, dbName, cn):
	if wildcardSearch:
		sql = "select '%s' as dbName, s.name schema_name, o.name, o.type from %s.sys.objects o inner join %s.sys.schemas s on o.schema_id = s.schema_id where o.name like '%%%s%%\' order by dbname, schema_name, o.name" % (dbName, dbName, dbName, objectNamePattern)
		cursor = cn.cursor()
		cursor.execute(sql)
		rows = cursor.fetchall()
		if len(rows) > 0:
			print pp(cursor, rows, 1)
	else:
		sql = "select '%s' as dbName, s.name schema_name, o.name, o.type from %s.sys.objects o inner join %s.sys.schemas s on o.schema_id = s.schema_id where o.name = '%s' order by dbname, schema_name, o.name" % (dbName, dbName, dbName, objectNamePattern)
		cursor = cn.cursor()
		cursor.execute(sql)
		rows = cursor.fetchall()
		if len(rows) > 0:
			print pp(cursor, rows, 1)
parser = argparse.ArgumentParser(description='SQL Server object search. Case insensitive')
parser.add_argument('-S', '--server', help='Instance you wish to connect to. myInstance is the default if not specified', dest='instance', default='myInstance')
parser.add_argument('-w', help='Wildcard search indicator. If specified, LIKE clause will be used for pattern matching. Otherwise it will look for an exact match. Exact match is default', action="store_true", dest='wild', default=False)
parser.add_argument('-d', '--database', help='Database(s) you want to search in. If more than one, separate them by comma only. It will search all databases by default', action="store", dest="dbs")
parser.add_argument('objectNamePattern', help='Object name pattern you want to search for', action="store")
argList = parser.parse_args()
try:
	cn = pyodbc.connect("DRIVER={SQL Server};SERVER=%s;DATABASE=master;Trusted_Connection=yes" % argList.instance)
except:
	print "Couldn't connect to %s. It is down or you might have had a typo." % argList.instance
if argList.dbs is None:
	cursor = cn.cursor()
	cursor.execute("select name from sys.databases where state = 0")
	rows = cursor.fetchall()
	for row in rows:
		objectMatch(argList.wild, argList.objectNamePattern, row.name, cn)
else:
	dbs = argList.dbs.split(',')
	for db in dbs:
		objectMatch(argList.wild, argList.objectNamePattern, db, cn)
]]> http://www.haidongji.com/2011/01/05/search-objects-in-sql-server-databases-with-pyodbc/feed/ 0 Getting rid of “Welcome to Internet Explorer 8″ screenhttp://www.haidongji.com/2010/12/09/getting-rid-of-welcome-to-internet-explorer-8-screen/ http://www.haidongji.com/2010/12/09/getting-rid-of-welcome-to-internet-explorer-8-screen/#comments Thu, 09 Dec 2010 16:37:30 +0000 Haidong Ji http://www.haidongji.com/?p=1000 The lack of Vimperator function in Chrome, which exists in Firefox, prevents me from totally switching to it. Because I work in different environments in terms of different clients and operating systems, I have to use Internet Explorer sometimes.

And I found the “Welcome to Internet Explorer 8″ screen upon IE starting, if you have not followed Microsoft command to configure IE the first time you started, terribly irratitating. It has message like this in the window: “Internet Explorer 8 helps you use the Internet even faster than before. New features like search suggestions retrieve blah blah…”. Would you please respect the end user’s intelligence, get out of the way and leave him/her alone in peace, quite, and solitude? Sure, one can follow the wizard and set things up, but it feels like being violated. The ability to customize things is good, but not under your dictation.

Behold, there is a way! Come and follow my way, dear reader, for it leads to enlightenment and eternal happiness:

1. Start -> Run
2. gpedit.msc
3. Navigate to User Configuration -> Administrative Template -> Windows Components -> Internet Explorer -> Prevent performance of First Run Customize settings
4. Double click, then set it as Disabled.

]]> http://www.haidongji.com/2010/12/09/getting-rid-of-welcome-to-internet-explorer-8-screen/feed/ 2 Unattended install of SQL Server 2008 R2http://www.haidongji.com/2010/09/23/unattended-install-of-sql-server-2008-r2/ http://www.haidongji.com/2010/09/23/unattended-install-of-sql-server-2008-r2/#comments Thu, 23 Sep 2010 18:05:14 +0000 Haidong Ji http://www.haidongji.com/?p=929 I’ve written unattended / silent install before for SQL Server 2005. Unattended install is cool because you can automate things: it saves time and improves consistency.

Here is something to get you started. I used the command below to:

1. Install SQL Server 2008 R2 default instance, including Replication and FullText search;
2. Install client tools such as: SQL Server Management Studio, Business Intelligence Development Studio, sqlcmd, Configuration Manager, and such;
3. Install Books Online so help is available locally.

In other words, it sets things up quickly for a typical OLTP DBA so s/he can start getting dirty and productive with minimal delay and distraction.

Things to keep in mind:

1. I didn’t care much about startup account (SQLSVCACCOUNT and AGTSVCACCOUNT) stuff for my testing here. Modify that as necessary. You can also change the startup account after the install is done. So don’t hang up on that if you don’t know what the domain\userAccount should be. Let the learning process continue;
2. Features is a list of features seperated by comma only, no space in between;
3. /qs is nice if you want visual dialog windows popping up telling you where you are at. It is neat to have that visual confirmation during the first few runs. Once you are comfortable, a simple /q will do;
4. Don’t forget /SQLSYSADMINACCOUNTS. Remember to put that piece in so you can connect after the installation is complete. Kinda important, ya know.

E:\Installers\SQL>Setup.exe /qs /ACTION=Install /FEATURES=SQL,Tools /INSTANCENAME=MSSQLSERVER /SQLSVCACCOUNT="NT AUTHORITY\SYSTEM"
/SQLSYSADMINACCOUNTS=".\Administrator" /AGTSVCACCOUNT="NT AUTHORITY\Network Service" /IACCEPTSQLSERVERLICENSETERMS
]]> http://www.haidongji.com/2010/09/23/unattended-install-of-sql-server-2008-r2/feed/ 1 Testing Windows IO with SQLIO and SysBenchhttp://www.haidongji.com/2010/09/19/testing-windows-io-with-sqlio-and-sysbench/ http://www.haidongji.com/2010/09/19/testing-windows-io-with-sqlio-and-sysbench/#comments Mon, 20 Sep 2010 04:28:38 +0000 Haidong Ji http://www.haidongji.com/?p=915 To benchmark IO on Linux and MySQL transaction processing, SysBench is a popular choice that can do both. After poking around at the source code, it seems PostgreSQL and Oracle are also included for transaction processing testing if you have the proper header files, but I didn’t test those.

To benchmark IO on Windows and SQL Server transaction processing, Microsoft provides two tools, SQLIO and SQLIOSim. SQLIO is a misnomer in that it really doesn’t have much to do with SQL Server. It is a general purpose disk IO benchmark tool.

So today I was playing with SysBench and noticed that I can compile and build it on Windows as well. I decided I should run IO benchmark on a single machine with both tools (SQLIO and SysBench), and see if I could reconcile the results.

To make things simple, I thought I would just benchmark random read of 3G (orders of magnitude bigger than disk controller cache) files for 5 minutes (300 seconds) with a single thread using 16Kb block size, without adding any additional queue. I tested this on both my laptop and an Amazon EC2 instance. The commands for both tools are listed below, and they should perform the same thing, as far as I can tell. Let me know if you have any comments/pointers or if I missed anything.

SysBench commands:

sysbench --test=fileio --file-total-size=3G prepare
sysbench.exe --test=fileio --file-total-size=3G --file-test-mode=rndrd --max-time=300 run

Fro SQLIO, here is the line in param.txt and command used:

c:\testfile.dat 1 0x0 3072
sqlio -kR -s300 -dc -b16 -frandom -Fparam.txt

As this is a quick test, I ran the same test twice and took the average value for comparison purposes. The detailed output is pasted at the end of this post.

On my Windows XP Pro Service Pack 2 laptop with Intel X-25 SSD:

IO/SecondThroughput/Second
SQLIO3833.559.90Mb
SysBench3390.7752.98Mb

So on my laptop, SQLIO’s results are 13% higher than that of SysBench.

On Amazon EC2 ami-c3e40daa with EBS device running Windows Server 2008 Datacenter Edition Service Pack 2, whose results varied widely between my two runs:

IO/SecondThroughput/Second
SQLIO678.9110.61Mb
SysBench408.966.39Mb

On this machine, SQLIO results are 66% higher than that of SysBench.

Below is the gory details.

Here are the detailed output on my laptop:
SQLIO
C:\Program Files\SQLIO>sqlio -kR -s300 -dc -b16 -frandom -Fparam.txt
sqlio v1.5.SG
parameter file used: param.txt
file c:\testfile.dat with 1 thread (0) using mask 0×0 (0)
1 thread reading for 300 secs from file c:\testfile.dat
using 16KB random IOs
using specified size: 3072 MB for file: c:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 3835.39
MBs/sec: 59.92

C:\Program Files\SQLIO>sqlio -kR -s300 -dc -b16 -frandom -Fparam.txt
sqlio v1.5.SG
parameter file used: param.txt
file c:\testfile.dat with 1 thread (0) using mask 0×0 (0)
1 thread reading for 300 secs from file c:\testfile.dat
using 16KB random IOs
using specified size: 3072 MB for file: c:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 3832.00
MBs/sec: 59.87

SysBench
C:\MessAround\sysbench-0.4.12\sysbench-0.4.12\sysbench\RelWithDebInfo>sysbench.e
xe –test=fileio –file-total-size=3G –file-test-mode=rndrd –max-time=300 run
sysbench 0.4: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 0
128 files, 24Mb each
3Gb total file size
Block size 16Kb
Number of random requests for random IO: 10000
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
WARNING: Operation time (18446744073709226000.000000) is greater than maximal co
unted value, counting as 10000000000000.000000
WARNING: Percentile statistics will be inaccurate
Done.

Operations performed: 10000 Read, 0 Write, 0 Other = 10000 Total
Read 156.25Mb Written 0b Total transferred 156.25Mb (52.143Mb/sec)
3337.16 Requests/sec executed

Test execution summary:
total time: 2.9966s
total number of events: 10000
total time taken by event execution: 2.9343
per-request statistics:
min: 0.01ms
avg: 0.29ms
max: 18446744073709.47ms
approx. 95 percentile: 0.48ms

Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 2.9343/0.00

C:\MessAround\sysbench-0.4.12\sysbench-0.4.12\sysbench\RelWithDebInfo>sysbench.e
xe –test=fileio –file-total-size=3G –file-test-mode=rndrd –max-time=300 run
sysbench 0.4: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 0
128 files, 24Mb each
3Gb total file size
Block size 16Kb
Number of random requests for random IO: 10000
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
WARNING: Operation time (18446744073694841000.000000) is greater than maximal co
unted value, counting as 10000000000000.000000
WARNING: Percentile statistics will be inaccurate
Done.

Operations performed: 10000 Read, 0 Write, 0 Other = 10000 Total
Read 156.25Mb Written 0b Total transferred 156.25Mb (53.818Mb/sec)
3444.38 Requests/sec executed

Test execution summary:
total time: 2.9033s
total number of events: 10000
total time taken by event execution: 2.8777
per-request statistics:
min: 0.01ms
avg: 0.29ms
max: 18446744073696.34ms
approx. 95 percentile: 15.39ms

Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 2.8777/0.00

Here are the detailed output from Amazon EC2 ami-c3e40daa with EBS device:
SQLIO
c:\Program Files\SQLIO>sqlio -kR -t1 -s300 -dC -frandom -b16 -Fparam.txt -BH -LS

sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file c:\testfile.dat with 1 thread (0) using mask 0×0 (0)
1 thread reading for 300 secs from file c:\testfile.dat
using 16KB random IOs
buffering set to use hardware disk cache (but not file cache)
size of file c:\testfile.dat needs to be: 3221225472 bytes
current file size: 0 bytes
need to expand by: 3221225472 bytes
expanding c:\testfile.dat … done.
using specified size: 3072 MB for file: c:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 1230.94
MBs/sec: 19.23
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 204
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 98 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

c:\Program Files\SQLIO>sqlio -kR -t1 -s300 -dC -frandom -b16 -Fparam.txt -BH -LS

sqlio v1.5.SG
using system counter for latency timings, 3579545 counts per second
parameter file used: param.txt
file c:\testfile.dat with 1 thread (0) using mask 0×0 (0)
1 thread reading for 300 secs from file c:\testfile.dat
using 16KB random IOs
buffering set to use hardware disk cache (but not file cache)
using specified size: 3072 MB for file: c:\testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 126.88
MBs/sec: 1.98
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 7
Max_Latency(ms): 497
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 13 9 0 3 7 8 8 8 8 8 8 8 2 1 1 1 1 1 1 1 1 0 0 0 2

C:\Users\Administrator\Documents\sysbench-0.4.12\sysbench\RelWithDebInfo>sysbenc
h.exe –test=fileio –file-total-size=3G –file-test-mode=rndrd –max-time=300 r
un
sysbench 0.4: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 0
128 files, 24Mb each
3Gb total file size
Block size 16Kb
Number of random requests for random IO: 10000
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Done.

Operations performed: 10000 Read, 0 Write, 0 Other = 10000 Total
Read 156.25Mb Written 0b Total transferred 156.25Mb (10.64Mb/sec)
680.95 Requests/sec executed

Test execution summary:
total time: 14.6854s
total number of events: 10000
total time taken by event execution: 14.6048
per-request statistics:
min: 0.01ms
avg: 1.46ms
max: 150.29ms
approx. 95 percentile: 4.77ms

Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 14.6048/0.00

C:\Users\Administrator\Documents\sysbench-0.4.12\sysbench\RelWithDebInfo>sysbenc
h.exe –test=fileio –file-total-size=3G –file-test-mode=rndrd –max-time=300 r
un
sysbench 0.4: multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Extra file open flags: 0
128 files, 24Mb each
3Gb total file size
Block size 16Kb
Number of random requests for random IO: 10000
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Done.

Operations performed: 10000 Read, 0 Write, 0 Other = 10000 Total
Read 156.25Mb Written 0b Total transferred 156.25Mb (2.1371Mb/sec)
136.77 Requests/sec executed

Test execution summary:
total time: 73.1139s
total number of events: 10000
total time taken by event execution: 73.0284
per-request statistics:
min: 0.02ms
avg: 7.30ms
max: 728.84ms
approx. 95 percentile: 23.08ms

Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 73.0284/0.00

]]> http://www.haidongji.com/2010/09/19/testing-windows-io-with-sqlio-and-sysbench/feed/ 4