CREATE TABLE S
   ( SNO    VARCHAR(5)   NOT NULL ,
     SNAME  VARCHAR(25)  NOT NULL ,
     STATUS INTEGER      NOT NULL ,
     CITY   VARCHAR(20)  NOT NULL ,
     UNIQUE ( SNO ) ) ;

 CREATE TABLE P
   ( PNO    VARCHAR(6)   NOT NULL ,
     PNAME  VARCHAR(25)  NOT NULL ,
     COLOR  CHAR(10)     NOT NULL ,
     WEIGHT NUMERIC(5,1) NOT NULL ,
     CITY   VARCHAR(20)  NOT NULL ,
     UNIQUE ( PNO ) ) ;

 CREATE TABLE SP
   ( SNO    VARCHAR(5)   NOT NULL ,
     PNO    VARCHAR(6)   NOT NULL ,
     QTY    INTEGER      NOT NULL ,
     UNIQUE ( SNO , PNO ) ,
     FOREIGN KEY ( SNO )
        REFERENCES S ( SNO ) ,
     FOREIGN KEY ( PNO )
        REFERENCES P ( PNO ) ) ;

INSERT INTO S (SNO, SNAME, STATUS, CITY) VALUES ('S1', 'Smith', 20, 'London');
INSERT INTO S (SNO, SNAME, STATUS, CITY) VALUES ('S2', 'Jones', 10, 'Paris');
INSERT INTO S (SNO, SNAME, STATUS, CITY) VALUES ('S3', 'Blake', 30, 'Paris');
INSERT INTO S (SNO, SNAME, STATUS, CITY) VALUES ('S4', 'Clark', 20, 'London');
INSERT INTO S (SNO, SNAME, STATUS, CITY) VALUES ('S5', 'Adams', 30, 'Athens');

INSERT INTO P (PNO, PNAME, COLOR, WEIGHT, CITY) VALUES ('P1', 'Nut', 'Red', 12.0, 'London');
INSERT INTO P (PNO, PNAME, COLOR, WEIGHT, CITY) VALUES ('P2', 'Bolt', 'Green', 17.0, 'Paris');
INSERT INTO P (PNO, PNAME, COLOR, WEIGHT, CITY) VALUES ('P3', 'Screw', 'Blue', 17.0, 'Oslo');
INSERT INTO P (PNO, PNAME, COLOR, WEIGHT, CITY) VALUES ('P4', 'Screw', 'Red', 14.0, 'London');
INSERT INTO P (PNO, PNAME, COLOR, WEIGHT, CITY) VALUES ('P5', 'Cam', 'Blue', 12.0, 'Paris');
INSERT INTO P (PNO, PNAME, COLOR, WEIGHT, CITY) VALUES ('P6', 'Cog', 'Red', 19.0, 'London');

INSERT INTO SP (SNO, PNO, QTY) VALUES ('S1', 'P1', 300);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S1', 'P2', 200);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S1', 'P3', 400);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S1', 'P4', 200);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S1', 'P5', 100);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S1', 'P6', 100);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S2', 'P1', 300);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S2', 'P2', 400);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S3', 'P2', 200);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S4', 'P2', 200);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S4', 'P4', 300);
INSERT INTO SP (SNO, PNO, QTY) VALUES ('S4', 'P5', 400);

One method of masking data is through reshuffling, which shuffles the value in target column(s) that you want to protect randomly across different rows.

Another way of doing it is through data generation. For instance, for target column(s), we just replace its value with something else.

For reshuffling, obviously the data element is still meaningful. In other words, a reshuffled account number is still a valid account number, only now its original owner has been changed. Depending on how stringent the requirements are, this may or may not be enough.

For data generation, we have this question to consider: is the format of the generated data important to us? If yes, then obviously some intelligence needs to be built in so that the generated data follows the format we define. For instance, a valid credit card number is 16 digits long, has certain prefix and/or suffix, the nth digit has a certain meaning, so on and so forth.

Another example is people’s name. Do we replace the name with some random letters we concoct together, or do we want the name to be realistic? If we want realistic names, then we may have to supply a dictionary for the masking software to pull that data from.

In either case, we also have the unique and foreign key constraints to deal with, if there are any. In certain instances where more than one schema/database is involved, the complexity increases exponentially.

Regardless of the method being used, performance of data masking process is important to consider. If the volume of data to be masked is small, then it may not be a big deal. But, as is often the case, you may have a huge transaction table that has millions and millions of rows to mask, then performance is a definite concern.

One idea I am toying around with for data masking performance issue is through low-level data manipulation. For instance, in MySQL, maybe play with rowid. And for Sql Server, play around with fileid, pageid, and such.

Another way to get around that is to do masking through batches. In other words, divide a big task into smaller tasks and tackle them one at a time.

Personally, I like the idea of data reshuffling. On one hand, the data element is meaningful. I know I don’t want to work with randomly generated gibberish that does not make sense to me. On the other hand, if one wants to do performance testing in test or development environment, one would like to have the data distribution as close to production as possible. And data reshuffling can probably keep the data distribution pretty close to that of production.

In my next entry, I will share a simple C# program I wrote to reshuffle data inside a CSV file.

Let me share with you how I do it. Notice that I look at table/column statistics and indexes in almost all the steps below. Therefore I purposely left them out in the discussion.

1. I always talk to the original author or the current owner of the code, asking him/her to walk me through it. I listen mostly, trying to understand why s/he writes code this way. I may ask a few questions, just to help me understand. I almost never do any lecturing at this stage. If the code is a stored procedure, I ask the author or stakeholder to give me sample execution statements, with parameters filled with typical values for production load. Sometime it is not possible to talk to the original author, because s/he may have left the company;

2. After that, I will get the database in question and restore it on my own workstation, with the data as close to production as possible. I will start running the code, and gather IO statistics and execution plan, and save the results for benchmark comparison later on;

3. Here is the raw code analysis stage, where code is put into an editor for analysis. In my case, it is VI or VIM. I find myself using the * command on all variable names and tables (make sure you are doing case insensitive search for Sql Server code), to see where they are used. This is also where I weed out the historical garbage code that is there but never being used anymore, such as unused variables, tables, etc. In the case of Sql Server, I mostly look for cursors, temp tables, and user defined functions;

4. Code simplification stage. I found a lot of bad performing code is unwieldy and unnecessarily complex. At this stage, I look for redundant WHERE clauses, convoluted AND and/or OR operators. This can be a tedious and time-consuming process. At this point, it is important to talk to the author or owner of the code, as a good understanding of data and entity relationship is crucial. Also, since I have gained some understanding from the steps above, it is easier to have an intelligent conversation with the code owner, with fresh perspective and momentum. Usually something good will come out of the conversation, and we will have something concrete to work on next;

5. I will make the changes discussed in step 4, once again, collect statistics and execution plan for comparison;

6. Repeat Step 4 and 5 a few times, when necessary. I usually start talking about better database design, code/design refactoring, set based operations, etc., as hopefully I have some credibility and gained the customer’s trust.

So what do you do to break down complex, low performance SQL code? What methodology, tools, tips, tricks you can share with me?

I actually thought about writing a white paper or even a course on migrating from MySQL to Sql Server, but never got the time to do it. Sometimes a project doing similar things can serve as a launchpad for this endeavour, but that never came along, at least not yet. I am very interested in database interoperability field though. I’ve done MySQL and Oracle admin in the past and have published some MySQL and Oracle stuff in blogs. I have much better technical skills on Sql Server than any other RDBMS platforms, primarily because I’ve worked on it longer.

Here are some of my thoughts. I think most of it applies equally on migration from Oracle, DB2, Sybase, Postgresql, etc., to Sql Server, or the other way around. It might be slightly easier to migrate from Sybase to Sql Server, considering their common root.

1. It is not easy to migrate existing app, unless the app is a simple one. Even for that, there are enough quirks that can throw people off and cause enough frustration to derail the whole projects. I’ve seen that happening twice, having engaged in moving 2 apps from Sql Server to Oracle;

2. Therefore, the best way to migrating to a new database RDBMS, in my opinion, is to start from a new initiative, probably not big initially. When you start things from a clean slate, you don’t have the historical garbage to worry about. Furthermore, you will give the team enough time to learn the new platform, and prepare the team for future migration, if you choose to do so;

3. Having open-minded team members is crucial to a migration project’s success. Too often people have emotional attachments to the platform they are familiar with, possibly out of job security concerns and lack of general curiosity toward new things.

I generally adopt a platform agnostic attitude, and don’t get religious and too carried away on the platform I work on. Having said that, I think these are points that marketing people can spin for persuasion purposes:

1. MySQL has too many storage platforms: MyISAM, InnoDB, MaxDb, and the newly introduced Maria. This can be viewed as a plus, as it provides choice. The downside of it is that it causes confusion for end users;

2. MySQL’s support for relational model is fairly recent. For example, for a long time, MySQL didn’t support Stored Procedures, Views, Triggers, Foreign Keys, etc. One could argue that MySQL is not mature in this area since it is new for them, but I think it is difficult to find evidence to substantiate that claim. Also, running the risk of offending some people, I think the importance of relational model got overblown a bit;

3. Sql Server offers the CLR integration. This can be a great selling point;

4. Sql Server offers tight integration with Visual Studio, Windows network, and all other things Microsoft. This is a huge advantage.

5. Too many people find *nix environment intimidating. Although MySQL works on Windows, but the perception in the marketplace is MySQL works better on *nix.

As far as migrating MySQL to Sql Server in a hosting web environment, my honest opinion is Sql Server will be fighting an uphill battle, because MySQL excels in this arena, especially for small and medium-sized, or departmental organizations, with the proliferation of such LAMP app like blogs, wikies, discussion boards, etc. I believe Microsoft’s weapon of choice in this arena should be SharePoint. Given Microsoft’s clout, it is certainly a battle worth fighting.

This time, the first thing the installer noticed was that the swap file is not big enough. I fixed that by doing a total re-install of Fedora Core 2. Yeah, I know that was kind of lame, but I got a few machines with me, so I didn’t waste time waiting for it.

Started again, this time the install went through, but I got some error messages at the end. The message was “groupadd: unable to lock group file”. The problem was that there were passwd.lock and gshadow.lock files under /etc. I then removed those files.

I tried to reinstall the RPM package. The message I got back was that it was already setup. I then ran:

rpm -e oracle-xe-10.2.0.1-1.0.i386.rpm

to remove it.

Finally, I was able to install it correctly.

It was a good exercise. I’ve done some Oracle database administration stuff, but never installation and setup stuff. I would like to bring my Oracle skill to the next level, so this is a good way to get started. 11g will come out within a week or so, but hopefully the skills I learned here can be transferred. I will give 11g a run when it comes out.

A few things learned:

1. yum is worth looking into. It is probably a better way to install or update dependencies than compiling binary or doing manual rpm install;
2. Got to be careful when typing. During profile setup, when I tried to put . /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/bin/oracle_env.sh into /etc/profile, I missed the blank between the dot and forward slash. This shell script sets up $ORACLE_HOME and $ORACLE_SID for you;
3. Here is a good site for 10g Express setup on Linux.

Each of them wants you to convert your database to their platform. To that end, they provide some program to help you along. Oracle has a thing called Oracle Migration Workbench. Sql Server has something similar called Sql Server Migration Assistant.

More often than not, marketing people from those companies will tell you how great their migration program is. That it is easy to migrate, you will see performance improvement, etc. Don’t believe them.

A couple of months ago, I worked on a project to migrate a database from Sql Server to Oracle. This database has a lot of stored procedures, user defined functions, and linked server stuff. I talked to Oracle people, and they recommended their own Migration Workbench, with Sql Server plug-in. I downloaded them and started working.

Basically, the Migration Workbench tries to go through database code in T-Sql. For code that uses built-in T-Sql functions, the Migration Workbench creates functions with the same name in Oracle, and try to writes something similar in PL/Sql that makes an attempt to do what T-Sql function does. I didn’t look too deep into this, but I am suspicious at all of them. The reason I didn’t look too deep into them was that the whole effort was derailed by a bigger problem. Let me explain.

In T-Sql, all variable names start with @ symbol. In fact, in many places, variable names are simply column names prefixed with the @ symbol. This can be pretty easy to read. And it actually works very well.

However, PL/Sql variable names don’t follow that convention. So, as Migration Workbench goes through T-Sql stored procedures, it simply strips off the @ symbol from the variable names. This basically renders all the code useless.

This is just very, very dumb. Oracle is a multi-billion dollar company, and you would think they should know this. Instead of stripping off the @ symbol, it could replace it with some kind of prefix. But it does not do that.

Another problem is with the identity field. Once again, one would expect that the Migration Workbench converts it to Oracle sequence, but it doesn’t. It changes that to NUMBER.

Eventually, the project was canceled. The moral of the story: migration from one RDBMS to another is not as easy as it sounds. It is doable. Sometimes it is probably easier to just write everything from scratch.

I decided to load Linux on it. Neither Ubuntu 7.04 nor Ubuntu 6.01 worked. (They might work better for newer desktops, I just don’t know) I have Fedora Core 2, so I installed that. I know the latest Fedora Core is version 6. I am kinda old school in this regard. Anyway, Fedora Core 2 worked on this computer.

I then decided to load Oracle 10g R2 on it. Here is a brief note of my process. It failed, because this machine has 160 MB of memory, not reaching the 256 MB required. I think I will purchase some memory tomorrow and try again.

1. Verify dependencies

[root@localhost haidong]# rpm -i oracle-xe-10.2.0.1-1.0.i386.rpm  --test
error: Failed dependencies:
        libaio >= 0.3.96 is needed by oracle-xe-10.2.0.1-1.0

2. Downloaded libaio source. libaio is Oracle’s library for asynchronous IO. configure failed, therefore I couldn’t compile and install it. Below is the last output on the screen.

ecking for libaio >= 0.3... no
checking for libaio 0.1... no
configure: error:
*** libaio is required.

3. I didn’t bother to look through the configure log file. Instead, I searched for “rpm libaio fedora core 2″ and got this link:

http://download.fedora.redhat.com/pub/fedora/linux/core/2/i386/os/Fedora/RPMS/

Downloaded the package for libaio.

[root@localhost haidong]# rpm -ivh libaio-0.3.99-2.i386.rpm
warning: libaio-0.3.99-2.i386.rpm: V3 DSA signature: NOKEY, key ID 4f2a6fd2
Preparing...                ########################################### [100%]
   1:libaio                 ########################################### [100%]

4. Install. It failed. I will try again, if/when I get more memory for this old machine.

[root@localhost haidong]# rpm -ivh oracle-xe-10.2.0.1-1.0.i386.rpm
Preparing...                ########################################### [100%]
Oracle Database 10g Express Edition requires a minimum of 256 MB of physical
memory (RAM).  This system has 155 MB of RAM and does not meet minimum
requirements.
error: %pre(oracle-xe-10.2.0.1-1.0) scriptlet failed, exit status 1
error:   install: %pre scriptlet failed (2), skipping oracle-xe-10.2.0.1-1.0

On Unix/Linux/Solaris based systems, these folders are located under:

$ORACLE_BASE/admin/$ORACLE_SID/bdump, udump, and cdump

To find out your $ORACLE_BASE and $ORACLE_SID, you can do:

echo $ORACLE_BASE and echo $ORACLE_SID

There is also the alert log, usually in the format of alert_$ORACLE_SID.log format, under the bdump folder.

There can be numerous trace and log files in these folders. To troubleshoot and correlate events with some problems, you can do:

ls -ltr

to sort files in ascending order according to date and time, and open up the file that is closest to the time when problem occurred. Those trace and log files may hold telltale signs of what you need to examine further.

For Sql Server, by default, the error log file is at C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\LOG. And that is the only place you need to worry about. If you turn on certain trace, the trace info will also be logged into error log.

Don’t confuse Oracle’s .trc file with Sql Server’s .trc file. Oracle’s trace file is in text format, generated when certain trace flags are turned on. Sql Server’s trace file are in binary format, and can be read using Sql Server Profiler.

Here is a PL/SQL code snippet:

declare MyVariable varchar2(20);

Begin

select ColumnName into MyVariable from MyTable where MyID = SomeInteger;
dbms_output.put_line('Hello ' || MyVariable);

End

In the above example, a column value for a particular record is assigned to MyVariable and printed out.

How do you assign select results into T-Sql variables in Sql Server then? Here is a code sample that does the same thing above:

declare @MyVariable varchar(20)

select @MyVariable = ColumnName from MyTable where MyId = SomeInteger

print 'Hello ' + @MyVariable

create table T1 as select * from T1

This CREATE TABLE AS statement basically clones table T1 with its structure and data in T2. This can be pretty handy at times.

The equivalent of that in Sql Server is SELECT INTO. For example, you can do:

select * into T2 from T1

to achieve similar results.