Archive for February, 2011

我爸是美国

这两天读到CIA的雷蒙德-戴维斯枪杀两位巴基斯坦的打劫者,其中一位还是被追杀,身中数弹。这个畜生杀完人了,照完相了,可能有点害怕吧,就电话联系同伙过来帮忙壮胆。同伙开了越野车,横冲直撞,嚣张跋扈,轧死无辜路人一名。我只看到了戴维斯的相片,前美国特种兵,在那个臭名昭著的镖局里(前Blackwater,后来公司改名叫Xe)干过,现受雇于CIA,小平头,很干练的样子。倒没看到同伙的照片和越野车,就臆想下:是不是那几位有人带着墨镜,剃着板寸,嚼着口香糖,穿着潇洒,装备齐全且高级,肌肉发达,英俊和刚毅的外表下嘴角轻翘,皮笑肉不笑,摆出了“我爸是美国”的架势呢?但巴基斯坦知道戴维斯的老爸是谁,据说还给了蹲监狱的这个凶手葡萄酒喝!

警方抓获了这个凶手后,美国政府不高兴了,要巴政府交人,因他是“外交官”,应该有豁免权。连奥巴马总统都这么说,直接提出要求,并派了参议员克里到伊斯兰堡去游说(为什么不是其他政府相关人员或CIA的头头,还有这个“外交官”的签证和护照的猫腻,容我慢慢道来)。美方发言人严肃地指出戴维斯是领馆工作人员,但意识到“领馆”工作人员好像不享受外交豁免,很快改口说我说错了,他实际上是“使馆”工作人员。

美国相关宣传部门立即通知全国大小媒体,不准报道戴维斯的CIA关联。各大媒体,作为政府当权派的喉舌,一如既往地跟着主旋律走,集体噤声。但英国的《卫报》不买这个帐,把戴维斯的CIA身份捅了出来。我的美国同胞,Glenn Greenwald,我眼中的真正美国英雄,也不买这个帐,我力荐他的相关与此事的两篇分析: This week in winning hearts and mindsThe NYT’s journalistic obedience

呕,说起美国宣传手法,这里有一段很拙劣的五角大楼发言人的视频,关于wikileaks。读过奥威尔的《1984》和《动物农庄》的人应当会有点毛骨悚然的感觉:

如果我的读者哪位曾经获得过诺贝尔和平奖,我再次呼吁你帮个忙,给Glenn Greenwald和ESWN作者、张爱玲遗产执行人宋一朗先生提名。我觉得那个叫阿桑奇的白发魔男不够格:这哥们的一些行为不地道,表演欲太强。

——

上面说到了一些“外交官”签证护照的问题。很巧的是,我前几天刚读完Bob Woodward的Obama’s Wars这本书,中间提到过CIA进入巴基斯坦的问题。Bob Woodward这哥们总体上也跟着主旋律走,像他通过一些材料和对话对巴基斯坦和阿富汗的影射: 一无是处,很无能、可恶和可怕。像阿富汗总统卡尔扎伊,他本人到底怎样,暂且不论。但在小布什朝代初期,被媒体描绘成阿富汗的救世主。他的微笑、潇洒的袍子、和头顶的圆帽通过媒体的正面描述,深深地打动了美国人民。小布什有一次到国会发表国情咨文时,曾把他飞到华府,邀请他到上首去坐,可能还受到了参众议员的起立鼓掌欢迎。而现在的卡尔扎伊在美国的形象和当时的美国形象有天壤之别:无能、腐败、在“恐怖战争”上拖后腿,并且涉嫌贩毒。美国的参众议员们和电视台记者也开骂,说到动情处,声情并茂,有“打倒在地,再踏上一只脚”的味道。

另外Woodward说不会暴露美国情报部门在外国发布的信息、心理、和媒体宣传的烟幕弹。咳,没做笔记,忘了这个是哪一页提到的了,在亚马逊上也没搜到。但书中提到美施压要巴基斯坦给CIA的人员放行入境,就搜visa,居然搜到了,让我们可以从一个侧面来了解下这个外交官/间谍/外交护照/外交签证这些概念和身份的有趣关系,晒给大家看看,因为法庭判案和这些可能会很有关系。一个很有意思的问题会是:戴维斯的名字在不在美国要求巴基斯坦给CIA人员放行的名单上?

需要说明的是,这里提到的书页,是精装版(hardback)本的书页,平装版(paperback)还没出吧。

第287页:

The Pakistanis balked at the joint operations, but were soon granting visas for more CIA people to enter Pakistan. For example, a January 18, 2010, request for 36 CIA people was soon approved, and CIA deputy director Steve Kapps personally asked for 10 more visas on April, 19, 2010.

第364页:

Jones said that President Obama wanted four things: full intelligence sharing, more cooperation on counterterrorism, faster approval of visas for U.S. personnel, and, despite past refusals, the sharing of airline passenger data.

If, God forbid, Shahzad’s SUV had blown up in Times Square, we wouldn’t be having this conversation, Jones warned. The president would be forced to do things that Pakistan would not like.

“The President wants everyone in Pakistan to understand if such an attack connected to a Pakistani group is successful there are some things even he would not be able to stop. Just as there are political realities in Pakistan, there are political realities in the United States.

“No one will be able to stop the response and consequences. This is not a threat, just a statement of political fact.” 这一段多讽刺啊!

第368页:

The options for Obama would be significantly narrowed in the aftermath of an attack originating out of Pakistan. Before such as attack, however, he had more options, especially if there was a way for Pakistan to make good on his four requests. Some 150 visa applications for U.S. military and intelligence operatives were being withheld that were over six weeks old, and U.S. embassy personnel were now required to renew their visas every 90 days. The Pakistanis were dragging their feet on visas for U.S. personnel to carry out the transfer of equipment that the Pakistani military needed. It was insane, Jones thought.

第369页:

In the event of a terrorist attack, Lute worried that it would be hard for Obama to defend Pakistant because their leaders had refused to do what had been simple and easy, especially on the visas and passenger lists. If two near-misses were the leading edge of a trend, what would it take to wake the Pakistanis up?

呜呼,讽刺呀,讽刺!上面说到这次小奥的说客是克里参议员,因为以前的说客格兰姆(Lindsey Graham)参议员,拜登副总统,国家安全顾问,CIA的头等都和巴基斯坦的总统和情报部门谈过,肯定也在CIA人员入境的问题上施过压。这下出事了,无辜的人被杀死了,再派那些人去,再不要脸,恐怕也不行了吧!并且据说克里参议员对国务卿一职抱有浓厚的兴趣。

Comments

Windows disk throughput benchmark steps with sqlio

Credits to Linchi Shea, SQLServerPedia/Brent Ozar, and Jonathan Kehayias for their discussions on sqlio.

Update: The test below, against one drive, can run for more than 1 hour. That is a long time. I’ve tested running 2 sqlio batch scripts against 2 locally attached drives, similar to the ones listed below, and found the results to be consistent with running those 2 concurrently.

1. Download and install sqlio;
2. Modify param.txt in the same directory where sqlio is. Create a testing file that is 24 gig, or sufficiently large to beat hardware cache along the way. For example, to benchmark drive m, the entry should look like this:

m:\testfile.dat 2 0x0 20480

If you use mount point, it should look like this:

m:\mountPointDirectoryName\testfile.dat 2 0x0 20480

3. Make sure file in step 2 is saved. In DOS command line, under the sqlio installation directory, run the command below. It will create a file for use later:

sqlio -kW -s10 -fsequential -o8 -b8 -LS -Fparam.txt timeout /T 10

4. Create a batch file with the following line as content, let’s call it mDrive.bat. SQLServerPedia article lets each test run 120 seconds, 2 minutes. In my testing against local disks, I made it 30 seconds. Actually I ran both 120 and 30 and didn’t see much of a difference. So I will use 30 seconds.

Note: to test against Windows mount point, the lines below should be like this:

sqlio -kW -t2 -s30 -dM -o1 -frandom -b64 -BH -LS \mountPointDirectoryName\Testfile.dat
sqlio -kW -t2 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o4 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o8 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o16 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o32 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o64 -frandom -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o128 -frandom -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t2 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t4 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t8 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t16 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t32 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kW -t64 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t2 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t4 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t8 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t16 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t32 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o1 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o2 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o4 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o8 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o16 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o32 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o64 -fsequential -b64 -BH -LS Testfile.dat
sqlio -kR -t64 -s30 -dM -o128 -fsequential -b64 -BH -LS Testfile.dat

5. In DOS command line, under the sqlio installation directory, run the command below. It takes more than 1 hour, but less than 2 hours to finish:

mDrive.bat > mDriveSqlioResults.txt

6. Run this Python script to parse mDriveSqlioResults.txt into a CSV file;

7. Open the CSV file with Excel, create a new sheet, insert a column chart. Drag the chart to make it decent size. Right click on “Select Data…” or something like that, then go back to the first sheet to pick 2 columns of data for this chart. I found charts with MBPS, IOPS, avgLatencyInMS, and maxLatencyInMS are helpful.

Comments

Parse sqlio log file with Python pyparsing

A few weeks ago I posted some questions on 3 Python text processing modules: pyparsing, SimpleParse, and NLTK. Today I need to analyze a log file generated by sqlio. I decided to use pyparsing. I am pretty pleased with it.

I am aware that there are at least 2 utility scripts for this: one is in Perl and then other is in Windows PowerShell. But I wanted to write in Python.

Noteworthy points:

1. In analyzing and building up the grammar/structure of the text for Python, I opted for line by line analysis and coding. I think this brings clarity and is easier to read. It felt a bit tedious in building it up, but once it is done, it is pretty rewarding to see the end results;
2. After parsing, the script writes the results to a csv file. You can then open that in a spreadsheet program to create some charts. Seeing the results in a column diagram chart brings clarity and focus to the result. It worked pretty well for me.

All in all, a good and productive day. Below are the details. The top part is the sample log content to be parsed, followed by the program.

"""
c:\Program Files (x86)\SQLIO>sqlio -kW -t2 -s30 -dH -o1 -frandom -b64 -BH -LS Testfile.dat
sqlio v1.5.SG
using system counter for latency timings, 2208037 counts per second
2 threads writing for 30 secs to file H:Testfile.dat
	using 64KB random IOs
	enabling multiple I/Os per thread with 1 outstanding
	buffering set to use hardware disk cache (but not file cache)
using current size: 24576 MB for file: H:Testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  6106.50
MBs/sec:   381.65
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 5
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 99  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
c:\Program Files (x86)\SQLIO>sqlio -kW -t2 -s30 -dH -o2 -frandom -b64 -BH -LS Testfile.dat
sqlio v1.5.SG
using system counter for latency timings, 2208037 counts per second
2 threads writing for 30 secs to file H:Testfile.dat
	using 64KB random IOs
	enabling multiple I/Os per thread with 2 outstanding
	buffering set to use hardware disk cache (but not file cache)
using current size: 24576 MB for file: H:Testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec:  6951.19
MBs/sec:   434.44
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 0
Max_Latency(ms): 6
histogram:
ms: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 87 12  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
"""
from pyparsing import *
output = open("sample.csv", "w")
input = open("hDriveSqlioResults.txt", "r")
data = input.read()
#------------------------------------------------------------------------
# Define Grammars
#------------------------------------------------------------------------
integer = Word(nums)
singleLetter = Word(alphas, max=1)
realNumber = Combine(Word(nums) + '.' + Word(nums))
reading = Literal("reading")
writing = Literal("writing")
readingOrWriting = reading | writing
random = Literal("random")
sequential = Literal("sequential")
randomOrSequential = random | sequential
#Line 1 is useful. We want to suck out the juicy parts.
sectionBeginning = CaselessLiteral("c:\\program files (x86)\\sqlio>sqlio ")
ioKind = "-k" + singleLetter("k")
numThreads = "-t" + integer("t")
numSeconds = "-s" + integer("s")
driveOrMountPath = "-d" + singleLetter("d")
outstandingRequests = "-o" + integer("o")
accessType = "-f" + Word(alphas)("f")
ioBlockSizeInKB = "-b" + integer("b")
#Line 2 can be safely ignored
line2 = "sqlio v" + restOfLine
#Line 3 can be safely ignored
line3 = "using system counter for latency timings" + restOfLine
#Line 4 can be safely ignored
line4 = integer + "threads" + readingOrWriting + restOfLine
#Line 5 can be safely ignored
line5 = "using" + integer + "KB" + randomOrSequential + "IOs" + restOfLine
#Line 6 can be safely ignored
line6 = "enabling multiple " + restOfLine
#Line 7 can be safely ignored
line7 = "buffering set to use" + restOfLine
#Line 8 we should get the file size out
line8 = "using current size: " + integer("fileSize") + Word(alphas)("fileSizeUnit") + restOfLine
#Line 9 can be safely ignored
line9 = "initialization done" + restOfLine
#Line 10 can be safely ignored
line10 = "CUMULATIVE" + restOfLine
#Line 11 can be safely ignored
line11 = "throughput" + restOfLine
#Line 12 we want IOPS
line12 = "IOs/sec:" + realNumber("IOPS") + restOfLine
#Line 13 we want MBPS
line13 = "MBs/sec:" + realNumber("MBPS") + restOfLine
#Line 14 can be safely ignored
line14 = "latency" + restOfLine
#Line 15 we need to get minLatency out
line15 = "Min_Latency(ms):" + integer("minLatency") + restOfLine
#Line 16 we need to get avgLatency out
line16 = "Avg_Latency(ms):" + integer("avgLatency") + restOfLine
#Line 17 we need to get maxLatency out
line17 = "Max_Latency(ms):" + integer("maxLatency") + restOfLine
#Line 18 can be safely ignored
line18 = "histogram" + restOfLine
#Line 19 can be safely ignored
line19 = "ms:" + restOfLine
#Line 20 can be safely ignored
line20 = "\%:" + restOfLine
extraStuff = ZeroOrMore(line14 | line15 | line16 | line17 | line18 | line19 | line20)
logEntry = sectionBeginning + ioKind + numThreads + numSeconds + driveOrMountPath + outstandingRequests + accessType + ioBlockSizeInKB + restOfLine + line2 + line3 + line4 + line5 + line6 + line7 + line8 + line9 + line10 + line11 + line12 + line13 + extraStuff
output.write("IO property, IOPS, MBPS, minLatencyInMS, avgLatencyInMS, maxLatencyInMS\n")
for tokens in logEntry.searchString(data):
	output.write("%(k)s%(t)s threads %(o)s queue %(f)s in %(b)s KB chunks,%(IOPS)s,%(MBPS)s,%(minLatency)s,%(avgLatency)s,%(maxLatency)s\n" % tokens)

Comments (5)

Utility script for instance level objects comparison between 2 SQL Server instances

Useful for SQL Server upgrade and migration, instance consistency check in the enterprise, et cetera.

Known assumptions:
1. pyodbc and proper SQL Server database driver setup and tested;
2. Windows Active Directory trusted connection to both instances. Login/password can also be used with very simple modification of the code;
3. Proper permission is set on both instances.

Known limitations:
1. It is based on object name comparison. In many cases, this is “good enough”;
2. It does not compare End Points or Server level triggers, although that should be added easily.

Command line usage:
scriptName -s sourceInstance -t targetInstance

import pyodbc, argparse
def getAllRows(cn, sql):
	cursor = cn.cursor()
	cursor.execute(sql)
	return cursor.fetchall()
def rowCompare(objectType, rowsSource, rowsTarget):
	if rowsSource == rowsTarget:
		print "%s(s) appeared to have been migrated." % objectType
	else:
		print "%s(s) not migrated yet:" % objectType
		for row in rowsSource:
			if row not in rowsTarget:
				print row[0]
parser = argparse.ArgumentParser(description='SQL Server migration/upgrade comparison: databases')
parser.add_argument('-s', '--source-server', help='Source server/instance',  required=True, dest='sInstance')
parser.add_argument('-t', '--target-server', help='Target server/instance', required=True, dest='dInstance')
argList = parser.parse_args()
try:
	cnSource = pyodbc.connect("DRIVER={SQL Server};SERVER=%s;DATABASE=master;Trusted_Connection=yes" % argList.sInstance)
	cnTarget = pyodbc.connect("DRIVER={SQL Server};SERVER=%s;DATABASE=master;Trusted_Connection=yes" % argList.dInstance)
except:
	print "Couldn't connect to %s. It is down or you might have had a typo."
"""1. Compare if all databases on the source server are present at the target server"""
print "\n"
sql = "select name from master.sys.databases order by name"
rowsSource = getAllRows(cnSource, sql)
rowsTarget = getAllRows(cnTarget, sql)
rowCompare('Database', rowsSource, rowsTarget)
"""2. Compare if all logins on the source server are present at the target server"""
print "\n"
sql = "select name from master.dbo.syslogins where name not like '##%' order by name"
rowsSource = getAllRows(cnSource, sql)
rowsTarget = getAllRows(cnTarget, sql)
rowCompare('Login', rowsSource, rowsTarget)
"""3. Compare if all linked servers on the source server are present at the target server"""
print "\n"
sql = "select name from master.sys.servers where server_id > 0 order by name"
rowsSource = getAllRows(cnSource, sql)
rowsTarget = getAllRows(cnTarget, sql)
rowCompare('Linked server', rowsSource, rowsTarget)
"""4. Compare if all jobs on the source server are present at the target server"""
print "\n"
sql = "select name from msdb.dbo.sysjobs order by name"
rowsSource = getAllRows(cnSource, sql)
rowsTarget = getAllRows(cnTarget, sql)
rowCompare('Job', rowsSource, rowsTarget)
"""5. Compare if all credentials on the source server are present at the target server"""
print "\n"
sql = "select name from master.sys.credentials order by name"
rowsSource = getAllRows(cnSource, sql)
rowsTarget = getAllRows(cnTarget, sql)
rowCompare('Credential', rowsSource, rowsTarget)
"""6. Compare if all system stored procedures inside master that start with sp_ on the source server are present at the target server"""
print "\n"
sql = "select name from master.sys.all_objects where type = 'p' and name like 'sp_%' and is_ms_shipped = 0 order by name"
rowsSource = getAllRows(cnSource, sql)
rowsTarget = getAllRows(cnTarget, sql)
rowCompare('System stored procedure', rowsSource, rowsTarget)

Comments

Page optimized by WP Minify WordPress Plugin