Setting up ownCloud

I’ve spent some of my spare time lately researching on ways to have better control of my calendar, contacts, important files that I sync to different places, and such. My objective is to move those things away from big cloud providers like Google, Microsoft, Dropbox, Skype, and so on. I am happy to report that the open source solution ownCloud provides just what I needed. I was able to set it up today and the tests were successful. Here are some noteworthy points.

  • ownCloud provides clients for Windows, Mac, Linux, iPhone, and Android devices, which is really nice;
  • In my case, the only php component missing was the php GD module. After installation, I restarted my VM and then it worked fine. For database, I used MySQL;
  • Setup was easy. I did the manual install by following instructions here. My web server was Tengine, a fork of Nginx. I used the sample Nginx configuration in the instruction page as the starting point, and it worked just fine;
  • ownCloud recommends using SSL, which I concur. I spent some time researching on certificate options, and decided to use self-signed certificate. The reason is simple: the intended customer is me and my family, therefore I don’t need a Certificate Authority (CA) to tell me that I am using a site that I own 🙂 Not forking out the cash is added bonus. Note, though, that if you are running a commercial site that conducts transactions online, it is necessary to purchase certificates and enable SSL. That’s not the right time to be penny-pinching (or a 铁公鸡);
  • To generate my own self-signed certificate, I followed instructions here. I used the 2048 bit key. I followed the optional step 5 so that I can start Tengine without providing a password;
  • I think ownCloud will be great for companies and teams. If you have control to a web server, which most companies and teams do, then more than likely you’ve got the infrastructure to have your own cloud. For a company/team, I do recommend purchasing SSL certificates, unless your company/team is pretty small;
  • Don’t forget to renew your self-signed certificate as time moves closer to its expiration date. In my case, the validation period is one year. I’ve already set a reminder for me.
  • I’ve really enjoyed using dnsimple. Separating DNS and hosting service really gives me great flexibility in learning and testing out different web technology.

Hope this helps. Let me know if you have comments/suggestions.

Introducing JiMetrics

For the impatient, here is the link to JiMetrics web site, where you can get the scripts and get going. To learn why it is called JiMetrics, you will have to read all the way to the end.

During the last couple of years, I’ve been working with health care related BI data warehouse, running on dozens of SQL Server instances. The workflow is pretty typical: nightly loads, daily usage for reporting, analytical, and ad-hoc research questions, general administration, instance monitoring and upgrade, things of that nature. As a result, I do a lot of administrative and DevOps sort of work.

To support that, and for monitoring, planning, and provisioning, we need to collect and store metrics data for analysis. Here are a few examples of typical use cases:

  • Storage management and provisioning. We need total size and available space for all disk drives on a server. Not just disks with drive letters, but also mount points on Windows. This sounds silly, but I was surprised to learn that not all monitoring software does that for Windows;
  • Software/hardware asset management. What hardware does it run on (core count, CPU class, manufacturer)? What about memory size? Is it a VM or a physical server?
  • Software patching. What is service pack level of the operating system? What about its version number? Do we need to patch the server?
  • How many rows of records do we have for tables holding information on allergy reactions on day 1, day 5, day 23, and day 500? For that table, how much space does it take? Of that space, how much is taken by data and how much is taken by indexes? What do we need to do to anticipate, plan, and execute so that we are on top of things?
  • What about instance configuration? Has the max memory size of a SQL Server instance been changed in the last few weeks?
  • Etc…

I surveyed the landscape. There are a few commercial products that could work, but were not picked in the end due to licensing issues. Microsoft’s MAP Toolkit and SCCM can be helpful, but like so many Microsoft products, they are unnecessarily complex and cumbersome. They tend to try too hard to be cute but feel annoying in the end. So I started writing my own. In fact, I wrote two versions: one for general purposes that I worked on the side, and the other customized version based on internal requirements and instructions. The one that I worked on my own time is called JiMetrics.

When I started writing, it wasn’t a difficult decision to use SQL Server database for metrics data storage, because most servers are running Windows and SQL Server. I did think about using column-store databases for metrics data storage due to the nature of the data: insert only, no updates, and good compression. InfoBright provides a wonderful and free column-store database engine under MySQL, but I didn’t have enough time. I may work on that later. It looks that SQL Server 2014 will have true column-store tables (to be verified), instead of the column-store indexes introduced in SQL Server 2012. SQL Server 2012’s column-store index feels like a hack job that was put together quickly for marketing purposes, but I digress.

I struggled a bit with what language to use. I thought about Python, Perl, PowerShell, and Ruby. From personal taste and productivity point of view, I’d rather use Python or Perl. However, in the end I chose PowerShell because of its tight integration with Windows and various Microsoft server products, and the fact that it is bundled with Windows so it is always there. Besides, I use SQL Server Agent to schedule collection processes, which makes PowerShell even more compelling.

So check out JiMetrics and let me know what you think. If you are interested, I suggest you start collecting data early. It is light weight and won’t take up much space. Down the road, I think you will be happy that you start the collection earlier. Currently, I’ve coded processes that captures the following metrics:

  • Windows server host and storage metrics
  • SQL Server instance discovery and instance configuration info
  • Select SQL Server instance performance counters
  • All SQL Server instance user database file metrics
  • All SQL Server instance user database table stats

As to the JiMetrics name, I initially picked SysMetrics, but sysmetrics.com is taken (possibly due to domain squatting, because the site has an on-sale sign). I purchased sysmetrics.org and even set it up, but in the end decided a com domain is far easier for people to remember. Fortunately JiMetrics was available. I hope it sounds good and is easy enough to remember.

My .emacs file

I am using the Evil package, which essentially embeds the Vim editor inside Emacs. As I documented here, the behavior of ( and ) commands is inconsistent with that of Vim: in Vim, ( and ) move the cursor based on sentence, but in Evil, its move is based on paragraph. I think I can live with that, we’ll see.

Anoter thing that I miss is the super tab plugin that I like from Vim. It does completion based on text already in the buffer(s). This is very handy for composing emails and writing documentation. This is different from the auto-complete and yasnippert packages on Emacs. After some searching and hacking, I was able to get that to work.

Here is my .emacs config for now. A few highlights:

  • On Windows, the default .emacs location is at c:\users\username\appdata\roaming I am aware of the dotfiles practice and using the init.el file, but since this is on Windows, I opted for the simple, default solution.
  • I am using the MELPA package management. MELPA appears to be the most up to date and reliable Emacs package management based on my research so far. There are quite a bit of grumbling against Marmalade;
  • There is a section for automatic package installation: if Emacs detects a package I defined inside my-packages is not installed, it will install that automatically for me. However, for whatever reason, on Windows, if Emacs is invoked from the PowerShell console, the .emacs will not be triggered.
;; ---------------------------
;; -- Setup MELPA package management
;; ---------------------------
(require 'package)
(add-to-list 'package-archives
  '("melpa" . "http://melpa.milkbox.net/packages/") t)
(package-initialize)

;; ---------------------------
;; -- Install packages I need
;; ---------------------------
(defvar my-packages '(evil smart-tab yasnippet)
  "A list of packages to ensure are installed at launch.")

(dolist (p my-packages)
  (when (not (package-installed-p p))
    (package-install p)))

;; ---------------------------
;; -- Config packages I need
;; ---------------------------

(require 'evil)
(evil-mode 1)
(require 'smart-tab)
(require 'yasnippet)

(add-to-list 'hippie-expand-try-functions-list
'yas/hippie-try-expand) ;put yasnippet in hippie-expansion list

(setq smart-tab-using-hippie-expand t)
(global-smart-tab-mode t)

Evil Emacs, best of both worlds?

First, let’s get this out of the way. This is not a rant against Vim, or Emacs, for that matter. I am not interested in joining the epic religious battle between the two. In fact, if you are not using either one, I highly recommend learning one of them. Both have steep learning learning curves but the payoff down the road is tremendous. Like Baron Schwartz articulated so well in his post on Sublime: if you are not using a powerful text editor, get one. I’d suggest Vim, simply because Vim is a great *editor* and your hands/fingers will thank you by the time you retire 😉

With that said, let’s get to the subject of this post.

I’ve been a long time Vim user. I put in some real effort learning Vim around 2004, and it took some time before it really started to pay off. Even after I was comfortable with entering text and doing basic movements, it took another good few years to learn, appreciate, and then form a habit of using many of its powerful command combinations. In fact, the learning never ends. Just a few months ago, I learned a neat trick to comment out selection of code and the Pathogen plugin management.

To encourage and hopefully help a Vim beginner, here are the commands (the ones that come to mind now) that I use most often. I didn’t include the motion commands (w b e 0 $ g gg G and such) because those are the basics. But they are incredibly important because they can and should be combined with other commands. For a beginner, I suggest you print our a cheat sheet and put it next to your monitor so you can consult by rolling your eyeballs. If I miss certain super useful commands here, please comment away.

  • The dot (.) command, which repeats your last action. This sounds simple but is incredibly powerful after you get used to it. For instance, if the last action is dw (delete a word), simply pressing . deletes another one!
  • The f motion command. While in Normal mode, typing fa moves the cursor to the next occurrence of the letter a. Replace a with other keys you desire.
  • The c (change text) and d (delete text) command. They both start in Normal mode, but the former lands you in insert mode so you can start entering text, whereas the later keeps you inside Normal mode, ready for the next motion or command.
  • The power of combination. The whole is greater than the sum of its parts! I often combine c (or d) together with f (or t) commands. I also combine c (or d) with w very often. I often do something like c3w, which deletes (change) the next 3 words and change into Insert mode, ready for me to enter text at the current cursor location. I use cG or dG quite often as well.
  • Powerful search with the forward slash while in Normal mode. Just press forward slash and then start typing the term you are looking for. It can do regex search as well. After entering the search term, press the enter key and it will land at the first occurrence of the term after the present location. Press n to search forward, N to search backward for the same term. Another trick I use often is the * command. While in Normal mode, press * will search for the term that the cursor happens to be in, then press n/N to find the next/previous occurrence;
  • I use the sp (or vs) often, to see multiple source files (or even different section(s) of the same file). sp splits the current window horizontally, vs (vertical split) splits vertically.
  • I often use the o (or O) command to create a new line.
  • I am aware of the mark (mark the current spot in the file so you can jump back to that location quickly), but I haven’t used it often. I need to make a conscious effort to form a habit of doing that. This is how it is done: say you are at the bottom of your code file but you need to make a fix to a function at the beginning of the file before you can continue, you can do ma (can be mb, mc etc up to mz). Now go to the top of the file and make the change. When done, press ‘a and you will be back at where you were earlier.
  • Macros. I use this quite often and really enjoy this powerful function. Here is an example. Say I am composing an IN clause of a list of countries for a SQL statement. Say I copied a list of 26 countries from an email a friend sent me:
    Albania
    Belgium
    China
    Denmark

    Zambia

    Now I need to surround each country with single quote, put a comma after the closing quote (except the last one). I normally record a Vim macro for this. Macro is a recording of the sequence of my actions that I can apply to other text later. Here is how I do it:
    1. Make sure the cursor is in the first line, anywhere is fine, Normal mode;
    2. Start recording by pressing letter q. Then I will pick a letter, say a, which is my macro’s name that I can call later;
    3. Press I, which means (I)nsert at the beginning of the line, another thing I use very often;
    4. Enter ‘, the single quote, then press Esc so I am back to Normal and ready for next motion;
    5. Press A, which means (A)ppend at the end of the line, another thing I use very often;
    6. Enter ‘, the single quote, then press Esc so I am back to Normal;
    7. Press j, now I am at the line that says Belgium. I will press q again to indicate I am done recording the macro;
    8. We are still in Normal mode, now it is time to play back the actions I took for Albania and apply them to all other countries;
    9. Type 25@a, now the rest of the 25 countries will be quoted properly with a comma at the end. Just remember to remove the last comma for the last country, otherwise the SQL cannot parse. If you need to bring all the countries in one line, just press J (yes, upper case j).
    Now this looks like a lot of steps, but in reality it is pretty fast and very efficient once you get the hang of it.

Ok, now where are we? That’s a long detour. Let’s get back to the subject of this post, if it has one!

Right, although I have been and remain happy with Vim, I am aware of the editor war between Vim and Emacs and am curious about Emacs’s capabilities. When attending various conferences/user group meetings, I tried to find an Emacs user who could show me the capabilities of Emacs. Yeah there are articles and screencasts on the web, but talking to a real person is nice. I really didn’t come across many Emacs users in those meetings/conferences I attended. Therefore my understanding of Emacs remained shallow. My impression, which could be wrong, is that there are way more Vim users out there than Emacs users. Even if we combine Vim and Emacs users, I bet that it is still a tiny minority in the entire population who have easy access to a computer.

My long lasting curiosity toward Emacs was finally satisfied when I started taking Coursera courses, because two of them require the students to give Emacs a try. The first course is Dan Grossman’s Programming Languages, highly, highly recommended. The second one is Startup Engineering, still ongoing at this writing. Since I mentioned Coursera, why not take another detour. Let me bring navel-gazing to a whole new level while I am at it. Brownie points to whomever reached this far, entirely non-redeemable!

If you are interested in learning and improving yourself but haven’t heard of MOOCs, I highly recommend you checking them out. The big ones are Coursera, Udacity, and edX. Coursera is the one that gets the most buzz and has the most comprehensive offering of courses at the moment, but I heard people say there are good ones from Udacity and edX as well. So far I’ve registered 6 or 7 courses and finished 3. I am about to finish the fourth one, Introduction to Systematic Program Design – Part 1, another fantastic course. I hope the MOOCs are continuing to thrive. No doubt they will attract a lot of VC interests but I hope they can keep true to their original spirit: to bring high quality education to people all over the world free of charge (or very cheaply to local people).

The first part of Professor Dan Grossman’s Programming Languages introduces the students to ML. As a requirement, I loaded Emacs on my Windows machine and installed ML. One thing that I can really appreciate fairly quickly was the tight integration of the editor window with the SML REPL. That is really super convenient, effective, and efficient.

Then Google announced that it would cease the Google Reader service. Seeing the tight integration of Emacs with many external tools such as newsreader, IRC chat, email, and such, I kept Emacs on my workstation, and have played with various tools, but haven’t settled on anything yet. It is my intention to come back to it and get rid of my Feedly account after Google Reader was gone.

Then Coursera Startup Engineering course started, I’ve watched the video on the demonstration of coding JavaScript inside Emacs. Once again, the tight integration of the editor with the REPL impressed me. I will go back and re-watch that video to learn all those tips. I’ve tried using tmux in combination with Vim for similar effect, but in my limited experience, I think Emacs integration is superior. In fact, if you work in *nix environment, I would hazard a guess that the combination of Terminator and Vim works better in more cases than tmux and Vim.

After using Emacs here and there, it is clear that I really miss the editor power of Vim. Thankfully, Evil package does a fantastic job of emulating Vim editing capabilities. Even macro works! The only drawback I noticed so far is while in Normal mode, the ( and ) keys in Vim moves the cursor to the next/previous sentence. I use that often, but under Evil, the command moves a paragraph, not a sentence.

Another interesting note is gVim on Windows 7 (I haven’t tested it on Windows 8) cannot display Chinese input on an English version of Windows 7. I haven’t bothered to figure out why, but it works fine on Emacs.

So now I am using Evil on Emacs, and have been happy so far. My next goal is to check out Emacs integration with PowerShell, since that is the language I am using for a project I am working on. In writing this, I need to remind myself that there is a balance between spending time to get your tools right versus spending time producing things.

Setting up Windows development environment with VirtualBox

Over the years, I’ve used many virtualization software for testing, hacking, and development work. So far my favorite virtualization software is VirtualBox.

I’ve written about using Vagrant and Veewee to make that process easier. I’ve since changed my strategy slightly:

  • I don’t recommend using Veewee anymore, because silly Ruby compatibility issues. I like the idea of Veewee, but in my opinion jumping through hoops to get it to work with Vagrant is no longer worth it. It is often quicker to just build a VirtualBox vm and then create a Vagrant box out of it than taking the Veewee extra step and remembering those extra commands. I couldn’t find the link now, but I thought I read somewhere that Vagrant has plans to make VM box creation easier;
  • I don’t use Vagrant for managing Windows VMs anymore. I tried but my effort was not too successful so far. Again, I read it somewhere that Windows support will be provided in future release of Vagrant. We will see.

My use cases in the past mostly involves single VM. For networking among Linux VMs, the instruction in my Vagrant post still works. But what if you want to build a Windows VM environment with domain controller, database servers, web servers, and other application servers based on a single VM image. So far I’ve built an environment with a domain controller and a separate application server (I will soon install SQL Server on it). Here is my notes for reference. The host machine is Lenovo W520 running Linux Mint 14.

  1. Download Windows 2012 ISO here;
  2. Start creating a base Windows 2012 VM in VirtualBox. Use the vdi disk type to make it easier working with VirtualBox;
  3. I could have installed the Data Center eval edition, but chose the standard instead. I didn’t choose the Core edition. Using the Core edition might make more sense in a real production environment, but I’d like to get something going quickly and also want to use the same image for all future VM needs, so the standard non-Core install works the best for now. I may come back and play with the Core edition down the road;

    Note: There are sites that talks about the possibility of activating Windows with an official license key, so that future VMs based on this image no longer need activation. My advice is: don’t bother with it. Licensing issue aside, my reasoning is that if you decide the VMs are going to be part of a domain, they then need to have their unique SID, which requires a separate Windows key, I think. Plus, I’ve run eval edition of Windows that has past its validation date but it still functions, probably in some limited capacity, but so far it hasn’t affected me. So I am hopeful that for my purposes, the eval edition of Windows 2012 will satisfy my need.

  4. Build the base VM with NAT networking. It is convenient for the VM to have access to the web directly. We will add additional network card(s) when we clone this base image.

    Once the base VM is built, copy that single vdi file somewhere else. Then remove/delete this base VM from your system all together;

  5. Make the base vdi immutable by running this:
    VBoxManage modifyhd pathTo/myBase.vdi --type immutable
    
  6. Build a domain controller based on this image. When creating a VirtualBox hard disk, use the base vdi file. At this point, add in a second network adapter, which should be type “Internal Network”. Take the VirtualBox default value for the rest of the fields;
  7. Finish building the VM, change the VM to a proper name, such as DCServer or whatever you feel is appropriate;
  8. Shutdown the VM, then run this command against the vdi file in your new VM’s Snapshots folder:
    VBoxManage modifyhd ~/Virtual Machines/myNewVM/Snapshots/{someName}.vdi --autoreset false
    

    This step is necessary, otherwise VirtualBox resets the VM to the base image after every reboot;

  9. Now install the appropriate software and set this VM as a domain controller. Follow instructions here.
  10. Configure the network card for the “Internal Network” by assigning a fixed IP address. If you’ve followed the instructions, the second network card should be called Ethernet 2. I didn’t bother with IPv6, and gave my VM the fixed IP address of 10.10.21.1. I left everything else to its default value;
  11. Turn off all Windows firewalls. In addition, open up “Group Policy Management”, expand to Domains->MyDomain->Default Domain Policy. Right click and choose “Edit…”. Group Policy Management Editor opens up. Under “Computer Configuration”, expand to Policies->Windows Settings->Security Settings->Account Policies. Change password policy accordingly. Remember to make sure the policy is enforced by right clicking and picking Enforce.
  12. Take a snapshot of the machine. Mark every disk file under the Snapshot folder so its autoreset is false.
  13. Now build another VM. Add the second network adapter for “Internal Network”. When done, shut it down and change its vdi disk file’s autoreset property, described above;
  14. Because this VM and the domain controller VM use the same base image, so by default they will have the same SID, which will prevent them to be in the same domain. Fix that by starting this new VM, then run sysprep. Follow the instructions here, if you need it.
  15. Reboot the new VM again, configure its “Ethernet 2” before it can join the domain we created earlier. Here is my configuration. Once again I didn’t touch IPv6:
    IP address: 10.10.21.11
    Subnet mask: 255.0.0.0
    Default gateway: 10.10.21.1
    Preferred DNS server: 10.10.21.1
  16. Rename the VM to an appropriate name, for example, sql1. Then join the domain we created earlier. Obviously the other domain controller VM needs to be up and you’ve run sysprep on this second VM. If you haven’t assigned a different SID for this new VM, you will receive error message like “the domain join cannot be completed because the SID of the domain you attempted to join was identical to the SID of this machine …”
  17. Repeat the process of building other VMs to join the domain, if you have the need.

Any comments/suggestions welcome. Good luck!