Python program template

For every simple python script you need same simple parts:

  • command line arguments parsing
  • logging
  • [optional] submodules
  • [optional] building to a binary

I’ve prepared a simple python project with all these parts to use as template. It is available on github.

Python sorting objects of user defined class

The most common way of sorting collections of custom objects in Python is to provide key function that is used to extract a comparison key from each element:

sorted("Case insensitive Sorting is here".split(), key=str.lower)

Install Jupyter Notebook and Scientific Environment in Ubuntu 14.04 With Python 3

Several changes occured since my last note about installing python scientific environment:

Scan images from command line in Linux

Console utilities in Linuxes are cool: mcabber for jabber, mutt for mail, mocp for music, vim to rule them all! This note is about scaning from the console (I assume your scanner is already set-up). Aim is to scan document with good quality into pdf with reasonable size.

Animations from scientific data using Paraview 4.3 and python

Create frames

This note describes usage of Paraview 4.3 (contrary to previous note with Paraview before 4.1). Some changes were made:

  • command line arguments were added to make code more reusable
  • it works with Paraview extracted from the archive (4.3.1)
  • white background must be specified manually (SetViewProperties(view, Background = [1, 1, 1]) in the code)

Install CUDA 6.5 on clean Ubuntu 14.04

Lately NVIDIA added repository for Ubuntu 12.04 and Ubuntu 14.04. Recently I hit problem with missing dependencies for libcheese-gtk23 and libcheese7 libraries while installing latest CUDA 6.5 on clean Ubuntu 14.04. Remedy to this can be found on askubuntu. So, the complete set of commands is below.

Research Papers Management With Zotero and Yandex.Disk

Zotero is an open source plugin for Firefox for managing research papers collections. You can use online zotero storage (pretty small) or use third-party sync application. I use Yandex Disk for this purpose: 10 Gb it quite enough for a moderate research library. zotero can automatically fetch article metadata (author, year, etc) from the web (like Mendeley and similar software do).

SSH config for multiple hops in Linux

Often some hosts in corporate network are available only from several gateway machines. The scheme looks like that: you connect to gateway ssh -A gateway.public.net and then to desired host from gateway machines ssh -A host.private.net. But the problem arises when you want to copy or download something from host.private.net. You have to do it in two hops, because host.private.net is not available from your machine directly. The solution is ssh forwarding.

Example of CMake file for CUDA+CPP code

Makefiles are quite straightforward and easy to write (in reasonable situations). But GNU Make is not crossplafrom. CMake is cross-platform, cross-application (it can generate projects for different IDEs and Makefile itself).

throw() in function signature in C++

There is base class exception in standard library (header <exception>). It has default constructor, copy constructor, copy operator and destructor and virtual what() function. Latter returns string with additional information about exception.

Sort strings from file in C++

The very simple and common test program is to read bunch of strings from input file (let it be input.txt), sort them and write down to another file (output.txt). There is an implementation with small bug: it adds extra empty line. I’ve modified original code a bit, so now it works correctly (note: if have last empty line in the input, you will have an empty line in the output). Fixed code is provided below.

Randomized quicksort implementation in C++

Quicksort has $O(N\log(N))$ computational complexity in best and average cases, $O(N^{2})$ for bad case. Extremely bad cases may be avoided by using randomized Quicksort.

Heapsort implementation in C++

Heapsort is one of the fastest sorting algorithms. The best and the worst cases for heapsort have same $O(n\log(n))$ performance.

Command line arguments in C and C++ with gengetopt

Gengetopt is a tool for parsing command line arguments in C and C++ applications. It generates cmdline.c and cmdline.h files in pure C, which contain parser, helper functions, error handling and structure to store argument values. Arguments to be included in auto-generated file are defined in config. I usually gengetopt config file as cmdline.ggo. In this note I provide an example of gengetopt config file and arguments usage in code.

Hybrid symmetric-asymmetric encryption for large files

Assymetric encryption is useful if you want to encrypt data at remote server via script. In case you use bash script for symmetric encryption job, you have to store password inside this script. So, if remote server is compromized, you will loose both the encrypted data and the key to decrypt it. Assymetric encryption allows you not to store decryption key at remote machine. So, data will be in safe even if server is hacked.

Creation of paper-ready plots with matlotlib

I used to utilize gnuplot for creation of plots with arbitrary data. The main problem was to postprocess data before plotting: I had to integrate postprocess into calculation code itself or to write additional script to do so. I’d like to provide an example of good-looking plotting with Python and matplotlib. Python has bataries included, so there will be no problem with preparation of data to be plotted.

Email notification if IP is changed with Python and cron

This Python script retrieves computer’s external IP address (using internet.yandex.ru) and emails message with new IP if IP has been changed since last check. BeautifulSoup module is needed, so install it if you have no installed version:

sudo pip install beautifulsoup

Universal simple Makefile

I often work on my netbook, so I prefer to use Sublime Text with Makefiles instead of full-featured IDEs. To automate build process I’ve constructed (with help of Vasily Picard and examples from the Internet) universal Makefile. It assumes the following structure of files.

Setting up Joomla 3.1.5 with PostgreSQL on Ubuntu 13.04

Installing dependencies Firstly you need to install webserver, php-related stuff and database server (we use PostgreSQL). apt-get install -y php5-fpm nginx postgresql php5-pgsql Now download and install Joomla (currently 3.1.5 is the latest). Please, update following lines with latest version if nescessary: cd /srv wget http://joomlacode.org/gf/download/frsrelease/18622/83487/Joomla_3.1.5-Stable-Full_Package.zip unzip Joomla_3.1.5-Stable-Full_Package.zip -d joomla chown -R www-data:www-data joomla Database setup The only two things you need to do is to create user (joomla in example) and database (joomla in example).

CudaSafeCall application snippet

It is a good tone to check CUDA API errors while calling cudaMalloc() and other functions. It also helps to find floating bugs caused by hardware (lack of memory, etc). I provide below an adapted version of CudaSafeCall I found many weeks ago in the Internet. Simply remove #define CUDA_ERROR_CHECK in production if unneeded.

Configuring Ubuntu 12.04 in OpenVZ on CentOS 6.4

This howto is based on two sources: Official wiki and Amutu blog entry. In short, the main difficulty is in bridge network configuration: correct network configuration doesn’t work in CentOS 6.4 while NetworkManager is running. I compiled official documentation and fix found in Amutu blog here in one entry.

Ubuntu Server 12.04 + Slurm 2.5.7 fatnodes

SLURM (Simple Linux Utility for Resource Management) — job scheduler and resource manager usually installed on supercomputers. For example, it runs on Lomonosov supercomputer in MSU, Moscow, Russia.

Tools for tracing a pcap file in linux bash

Pcap files are pocket capture files. They usually contain raw log of network connections and packets. It is the most popular format and it available in almost all network analysis tool.

Show N biggest directories in linux bash

du (disk usage) utility in linux bash summarizes disk usage of each directory or file. By default, it outputs only directories. Just enter some dir and type

$ du ./

Toggling string case in linux bash

It’s quite an academic task, but anyway useful sometimes. I’ve collected different ways to do it in terminal in linux. Some of them work with UTF-8 characters (some it will toggle case for “й”, “ё” and so on. It will not in general handle special ligatures, such as “ß”” and “fi”.)

Setting up RhodeCode on Ubuntu 12.04

This howto is based on a great guide with modifications. All settings are made for server domain hg.kenarius.org. You can replace it with your one easily.

AWK: a powerful tool for programmer

AWK is an acronym of first letters of its authors (Aho, Weinberger and Kernighan). It is a data-manipulating scripting language with huge possibilities. There are several implementations of it: awk is a canonical one, nawk (new awk), mawk (default in Ubuntu 12.04), gawk is GNU awk. I recommend latter one, because it works correct with unicode symbols in example:

$ echo юникод | gawk "{res = toupper(\$1); print res;}"
ЮНИКОД

Daily backups (files + mysql) to yandex.disk in Ubuntu 12.04

We all do backups. This note is about doing them on Ubuntu 12.04.

Tasks:

  • archive important folders
  • dump mysql databases
  • upload all to yandex.disk (online web storage with 10GB space)
  • delete old backups

I’ll save only backups for last five days.

Fixing missing icons in PCManFM

After moving to i3 I’ve installed PCManFM as GUI file manager. To fix missing icons one have to define icon theme. These instructions are tested on Ubuntu 12.04, but it must work with other distros.

Montage png into single one

Glue several png’s into one png [1]+[2]+[3]+[4] = [[1][2][3][4]] with montage from ImageMagick:

montage -geometry +1+1 -tile 4x1 image*.png final.png

Blender and Python molecules visualization

Blender is a free software for 3D graphics. In old virsions many operations were binded to hotkeys, so it was believed that Blender is very hard to master. Now it has more userfriendly interface with button in addition to hotkeys.