Web front for my code

Awhile ago I set up my own git server. I’ve been hacking happily using that and Eclipse. In my spare time, I’ve been taking UCSD’s wonderful algorithm course: Algorithmic Design and Techniques. The course provides plenty of programming challenges! I chose the paid version so my code can be evaluated against all tests in the grading server. I solved the programming challenges in 3 languages: Java, Python, and C++. It has been so much fun!

Anyway, I thought it’d be nice to enable a web front end to my own code hosted on my own server. There are a lot of choices: gitweb, gitlab, gitea, etc. A couple of days ago I came across cgit. It’s pretty lightweight, written in C. That’s what I chose for that task. It’s setup now already, please go to https://code.haidongji.com to check it out!

It takes a bit of work to get the clean subdomain working. My environment is Debian 9 running apache2. Below were steps taken to make my environment work. Hope it helps somebody!

  • sudo apt install cgit
    The install creates /etc/cgitrc file and /etc/apache2/conf-available/cgit.conf
  • Here is the content of my apache2 site conf file
    <VirtualHost *:80>
            ServerName code.haidongji.com
            ServerAdmin emailAddress
            DocumentRoot /usr/share/cgit/
    	<Directory "/usr/share/cgit/">
    		AllowOverride None
    		Options ExecCGI
    		Order allow,deny
    		Allow from all
    	</Directory>
    	Alias /cgit-css/cgit.css /usr/share/cgit/cgit.css
    	Alias /cgit-css/cgit.png /usr/share/cgit/cgit.png
    	ScriptAlias / /usr/lib/cgit/cgit.cgi/
    
            ErrorLog ${APACHE_LOG_DIR}/codehaidongji.log
            CustomLog ${APACHE_LOG_DIR}/codehaidongji.log combined
    
    </VirtualHost>
  • sudo apt install python3-pygments for syntax highlighting
  • Here is the content of the /etc/cgitrc file
    css=/cgit-css/cgit.css
    logo=/cgit-css/cgit.png
    
    # enable Pygments syntax highlighting. Must be above scan-path, otherwise it would not work!
    source-filter=/usr/lib/cgit/filters/syntax-highlighting.py
    scan-path=/srv/git
  • sudo a2enmod cgid
    sudo a2enconf cgit
  • sudo systemctl restart apache2

Note that in /etc/cgitrc, syntax highlighting line MUST BE ABOVE the scan-path line, otherwise syntax highlighting will not work!

Cheers and happy coding!

Getting Eclipse’s EGit to work with my own git server

I’ve been thinking about setting up my own git server for a while, and finally got it up and running last week. Since I do a lot of hacking with Eclipse, I naturally want Eclipse’s EGit to work with my own git server. Here are a couple of noteworthy points:

  • As of this writing, if you want ssh authentication with EGit, ecdsa type key is NOT going to work. rsa ssh key works. I found this out the hard way, since I’m trying to transition from rsa to ecdsa. Therefore I initially didn’t generate rsa keys on my computers. I had to run “ssh-keygen -t rsa -b 4096” and made sure that Eclipse – Preferences – General – Network connections – SSH2 – General tab – Private keys has id_rsa entered;
  • I followed instructions from here to get a few bare repo that I need initialized. Initially I had the misunderstanding that once I push code to the server, I should be able to see the files on the remote server’s repo directory. That turns out to be NOT the case. There is some magic going on under the objects directory that make things happen, such that although the code directories and files are not visible to the naked eye, I could push/fetch/pull to the remote server without problems. When I have time, I need to dig deeper and have a better idea how it all works. For now, I’m happy that I have it working that I can work with the same code base on different dev computers.
  • I followed exercise here for testing and found it helpful.

Happy coding y’all.

Mathematical modeling of fake news

I took Mathematical Modelling Basics course during the last couple of months. It was produced by Delft University of Technology, offered for free on edX. Thanks TUDelft and edX!

It is a great course introducing mathematical modeling. I like the fact in this short course, 3 important areas are covered with good practices: mathematics, computer programming, and technical writing.

  • The math involved is modeling with system of ordinary differential equations. Both analytical and numeric solutions to said model are introduced and practiced with good exercises;
  • The programming part is using Python to solve the system of ordinary differential equations numerically, with NumPy, using Euler’s method. Plotting is also introduced and practiced with Matplotlib;
  • Finally the course asks students to write a technical report using LaTex.

I also like the fact that the course encourages students to form a team and work together. I was very fortunate to be able to work with Zeus Garyulo, an Argentine currently working in Finland. Zeus is a wonderful teammate, very smart and has a much better grasp of math involved than I have. He broke down problems into actionable items quickly and provided the majority of the modeling, analysis, and validation work. Thanks Zeus!

The problem we chose to tackle is the spread of fake news. Without further ado, below is our report.
FakeNewsMathModel_Z

Automatically mount USB 3 attached to my wifi router

As discussed here, I have Seagate Ultra Slim USB 3 attached to my wifi router as a NAS (Network-attached storage). It provides a good backup/sharing storage for my home network that’s accessible from all my home computing devices, which was nice.

Accessing it from Windows is easy to set up, all I needed to do was to map a drive. Accessing it from Linux needs some additional work, but it turned out to be not too bad either.

Initially I created an entry in /etc/fstab. It uses the cifs protocol. Some cursory research shows that cifs is old and oboselete, but I’m not sure of other ways of accessing it. As I mentioned in that post, I had to put in the ver=1.0 option for it to work, something I don’t like.

Anyway, the drawback of /etc/fstab entry is that my Linux system would try to mount it before establishing network access, which doesn’t work for obvious reasons. I had to “sudo mount -a” afterwards. That is manual and gets tedious real fast.

So I did some research. One solution was to use _netdev option in the fstab entry, which didn’t work for me. The other was to use the auto option, which didn’t work either.

In the end, what worked for me was using systemd. Here are two articles that I relied to get mine going.

Tomáš Tomeček, Automatic mounts with systemd
Michael Albert, Systemd: (Auto-) Mount cifs shares This article has good examples for cifs options. Yes, I had to use the ver=1.0.

Thanks guys!

fail2ban installation and configuration notes

A couple of days ago one web site I volunteer to manage was under DDOS attack. I installed and configured fail2ban to protect us from future similar attacks. Here are some notes. The server is the RedHat/Fedora/CentOS variety, as you can tell from commands listed below. Please translate them to your distro’s corresponding commands as needed.

  • Installation is easy:
    sudo yum install fail2ban

    To make fail2ban starts automatically after a reboot, run this:

    sudo systemctl enable fail2ban

  • Configuring is relatively easy. It’s recommended that you create your own jail configuration file, using the jail.conf from the installation as a starting point. Three things are noteworthy from my experience:
    1. Make sure that you provide the correct log file. For web server, there are typically one access log file and one error log file. Ensure that you feed the right log file when using a particular filter;
    2. On this server, fail2ban didn’t properly expand the log and file names when I put wildcard characters in them. I got around that by listing them one by one.
    3. In the jail.conf file, no default banaction was defined. I added the following line:
    banaction = iptables-multiport
  • To write your own custom filter, make sure you put a sample log entry inside the filter file as a comment. Use the following command to debug your filter:
    sudo fail2ban-regex /path2testLogfile/test.log /etc/fail2ban/filter.d/my-filter.conf
    Here is a filter that I wrote:
    [Definition]

    failregex = ^ -.*”POST \/component\/mailto\/\?tmpl=component\&link=aHR0cHM6.*”$

    ignoreregex =

  • After getting your jail.local ready, run the following command to debug any potential issues. I’ve found that if you have issues with your jail or filter files, “sudo systemctl start fail2ban” doesn’t always give you a good enough error message. Use this instead:

    sudo /usr/bin/fail2ban-client -x start

    You may need to start/stop a couple of times. To stop, run

    sudo /usr/bin/fail2ban-client -x stop

  • After debugging, before you finally start fail2ban service, it’s better to search the current access/error log and see if there is a match to the filter you defined. If yes, then take a note of its IP address and the last time it appears in the log file. Then start fail2ban by running
    sudo systemctl enable fail2ban
  • To verify that it works, run iptables -S and if it catches one offender and puts it in jail, you should see it in the output. Now go back to the access/error log and ensure there is no entry from that IP address since the last timestamp.

Good luck in protecting your servers!