Ubuntu更新、Python和R软件包安装、和Firefox下载插件演示

视频演示:
1. 怎么更新Ubuntu Linux;
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential
2. 怎么安装Python包;
sudo apt-get install python-pip
sudo apt-get install python-dev
sudo pip install numpy
sudo pip install ggplot…
3. 怎么安装R和R包;
sudo apt-get install r-base
sudo apt-get install openjdk-7-jdk
sudo R
install.packages(“xlsx”)
4. 如何方便快捷下载视频:
Firefox, 插件DownThemAll

建立中文版Linux虚拟机

最近在几个QQ和微信IT群里灌水,注意到不少同学在大学或工作中没接触过Linux。而很多IT项目如大数据、机器学习、服务器等都需要Linux技能,所以很多人想开始接触、学习Linux。我就动手做了以下视频给初学者,希望能有帮助。这是我第一次做screencast,很希望能听到你的批评和建议。

更新:视频上传到优酷后,效果并不理想。我又尝试了其它视频分享网站如乐视、QQ视频、新浪视频、和土豆。乐视的上传网页没有上传渠道,或许因为我的IP地址在国外?我在新浪视频网页也碰到同样问题。QQ视频倒是允许上传,但最终告诉我“您的视频可能包含有相关主管机关明确规定不能出现的违规内容,因此无法通过审核。请修改后再重新上传。”,真是令人匪夷所思。

最后上传到土豆,效果还可以。我是把几个小视频合并到一个文件,但由于操作错误,我没有把一开始介绍VirtualBox软件的那部分合并进去。你只要记得VirtualBox在Windows、Linux、Mac上都可以免费运行并去下载安装就可以啦。

Using rsync to backup remote n00 files

I had trouble rsync remote Linux 600 files (rw——-) today. I knew that I came across this issue before but couldn’t remember how I resolved it. Therefore I had to waste time looking for and verifying a solution. Hence this blog post.

This is the problem I had earlier:

rsync -zr userA@remoteServer:/var/www/website/ /home/user/Documents/webSiteBackup/website/www/
rsync: send_files failed to open "/var/www/website/wp-config.php": Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1655) [generator=3.1.0]

So the issue is that wp-config.php is a 600 file, meaning only root can read and write it. Although userA@remoteServer has sudo privilege, I still need to run visudo so that it won’t ask for password when this user runs rsync.

Here is the line I added into visudo:

userA    ALL=(root) NOPASSWD: /usr/bin/rsync

And here is the slightly modified bash command to run:

rsync --rsync-path="sudo rsync" -zr userA@remoteServer:/var/www/website/ /home/user/Documents/webSiteBackup/website/www/

Hope it helps you as well.

sed tricks

I helped a charity to rebuild a MySQL server and to restore a database with a lot of data of longblob type in the last two days. Fortunately there was a dump backup file for the database in question.

However, tables with longblob column(s) were not defined with “ROW_FORMAT=COMPRESSED”. I’d like to restore that database with row compression before inserting the data. Therefore I need to modify the dump sql file. The problem is that the file is 2.5 GB and the server only has 4 GB memory. So editing it is a challenge. Fortunately, Linux has sed to save the day. Don’t you love open source free software?

I am power Vi/Vim user, so I am familiar with sed and have used it in the past. But there are still a few things that I searched around for quick answers. So I’ll record noteworthy points here. I couldn’t remember how many times my own past blog entries helped me over the years. And I hope you’ll find this helpful too!

  • The -n switch is important. sed is a stream editor. In many cases you’d like to supress data streaming to stdout, and -n does that. This was especially important in my case, because a) the file is large, b) it contains blob that may or may not “fit to print”;
  • To see a particular line, say line a, use the p (print) command: sed -n 'ap' file
  • To see all lines between line a and b: sed -n 'a,bp' file
  • To see multiple, non-adjacent lines, say line a, e, g: sed -n 'ap;ep;gp' file
  • To edit big files, you’d like to make in-place changes. Therefore the -i switch. For example, to put in InnoDB row compression, this is the command I used: sed -i 's/CHARSET=utf8;/CHARSET=utf8 ROW_FORMAT=COMPRESSED;/' file
  • Similarly, to delete line a: sed -i 'ad' file You can also do range delete as well

By the way, when restore InnoDB database with a lot of blob data, it makes a lot of sense to enable the following settings in my.cnf, if they are not enabled already. It’ll make administration much easier down the road:
innodb_file_per_table
innodb_file_format = Barracuda

You may also need to tweak the max_allowed_packet and innodb_log_file_size parameters for successful restore.

Something else to pay attention to:
If you use:

mysql -uuser -p database < dump.sql

to restore the database back, the program may report the wrong line where it had loading problems. In most cases, you need to search the surrounding lines to find where the problem is.

Additionally, if you are in a hurry and want to load data in without troubleshooting loading issues, you can try adding -f switch to the command above so the restore ignores errors it encountered and jump to the next line.

WordPress post update services and Nginx upstream time out

Starting around last Friday, I think, this site became irresponsive. Checking Tengine/Nginx error logs, here is a sample of what I saw:

2014/03/xx 08:43:40 [error] 3837#0: *6149 connect() to unix:/var/run/php-fastcgi/php-fastcgi.socket failed (11: Resource temporarily unavailable) while connecting to upstream, client: xxx.xx.xxx.xxx, server: haidongji.com, request: "POST /xmlrpc.php HTTP/1.0", upstream: "fastcgi://unix:/var/run/php-fastcgi/php-fastcgi.socket:", host: "www.haidongji.com"
2014/03/xx 08:45:13 [error] 3837#0: *6151 upstream timed out (110: Connection timed out) while reading response header from upstream, client: x.xxx.xx.xxx, server: haidongji.com, request: "GET /wiki/doku.php?id=wiki:dokuwiki HTTP/1.0", upstream: "fastcgi://unix:/var/run/php-fastcgi/php-fastcgi.socket", host: "www.haidongji.com", referrer: "http://www.haidongji.com/"
2014/03/xx 08:46:00 [error] 3837#0: *6159 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xx.xxx.xx.xxx, server: haidongji.com, request: "GET /category/technology/mysql/feed/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fastcgi/php-fastcgi.socket", host: "www.haidongji.com"

So I started researching. A number of web pages, including StackOverflow, recommended increasing values for some or all of the following Tengine/Nginx parameters, so time out wouldn’t occur:

  • fastcgi_read_timeout
  • proxy_connect_timeout
  • proxy_read_timeout
  • proxy_send_timeout

Those suggestion appeared to be reasonable. So I tried raising those parameter values one at a time. Their default value is 60 seconds. I raised them to 600 just to test.

In my case, the problem persisted. The only difference was that the site suffered a slower death and the few minutes it was able to stay alive, the site was terribly slow and unacceptable.

At this point I’d spent hours debugging this. I decided to give myself a break and come back to it the next day. Before falling asleep, I wrote down a note of checking some reputable forums the search engines may not necessarily able to get, such as Linode’s forums. Failing that, I am determined to read up on more documentation and code to get to the bottom of this.

It turned out that Linode’s forum provided the necessary clue to finally resolve this. The problem lies in the WordPress blog publishing update service. The idea behind it was to notify blog aggregators like Technorati/Feedburner (remember them?) as soon as you publish a blog. One of the aggregator sites listed in my WordPress setting decided to not accepting those notifications anymore. Not knowing this, WordPress still tries to send notification out in vain, which caused the whole mess. After removing that URL, everything went back to normal.

On your WordPress admin page, you can find which URL your blog pings to by going to: Settings -> Writing -> Update Services. Update: The same problem happened again. So I removed pingomatic.com also. Now my WordPress does not ping or update anybody. So far the site has been stable.

Hope this helps!

Lesson learned/reinforced:
After working on a problem for a long time and you are stuck, take a break, go for a walk, find a friend to talk to and describe the issue and what you’ve done thus far. You’d be surprised at how effective this can be.