For every simple python script you need same simple parts:
I’ve prepared a simple python project with all these parts to use as template. It is available on github.
The most common way of sorting collections of custom objects in Python is to provide key function that is used to extract a comparison key from each element:
sorted("Case insensitive Sorting is here".split(), key=str.lower)LXC containers are great for application isolation and safe deployment. Running unprivileged containers is the safest way to run containers in a production environment. This note contains enough information to create your own safe container.
Ubuntu has a very detailed documentation about OpenVPN server configuration. The key feature of this note is script for automatic generation of single-file client configurations (keys and certs are embedded) and sending them by email. Looks interesting? Read below!
Several changes occured since my last note about installing python scientific environment:
This note describes usage of Paraview 4.3 (contrary to previous note with Paraview before 4.1). Some changes were made:
SetViewProperties(view, Background = [1, 1, 1]) in the code)Lately NVIDIA added repository for Ubuntu 12.04 and Ubuntu 14.04. Recently I hit problem with missing dependencies for libcheese-gtk23 and libcheese7 libraries while installing latest CUDA 6.5 on clean Ubuntu 14.04. Remedy to this can be found on askubuntu. So, the complete set of commands is below.
See updated note.
I’ve completed setting up my comfort set of LXC containers and want to save some points I spent a lot of time for or things for copy-paste.
I used the official Ubuntu Server Guide but note at Digital Ocean knowledge base also helped me a lot. I created 3 containers: one for vpn, one for site on Drupal and last one as www proxy for second one with nginx only.
Note: I’ve created an updated page with most recent instructions for Jupyter Notebook.
Zotero is an open source plugin for Firefox for managing research papers collections. You can use online zotero storage (pretty small) or use third-party sync application. I use Yandex Disk for this purpose: 10 Gb it quite enough for a moderate research library. zotero can automatically fetch article metadata (author, year, etc) from the web (like Mendeley and similar software do).
Often some hosts in corporate network are available only from several gateway machines. The scheme looks like that: you connect to gateway ssh -A gateway.public.net and then to desired host from gateway machines ssh -A host.private.net. But the problem arises when you want to copy or download something from host.private.net. You have to do it in two hops, because host.private.net is not available from your machine directly. The solution is ssh forwarding.
Makefiles are quite straightforward and easy to write (in reasonable situations). But GNU Make is not crossplafrom. CMake is cross-platform, cross-application (it can generate projects for different IDEs and Makefile itself).
The very simple and common test program is to read bunch of strings from input file (let it be input.txt), sort them and write down to another file (output.txt). There is an implementation with small bug: it adds extra empty line. I’ve modified original code a bit, so now it works correctly (note: if have last empty line in the input, you will have an empty line in the output). Fixed code is provided below.
Quicksort has $O(N\log(N))$ computational complexity in best and average cases, $O(N^{2})$ for bad case. Extremely bad cases may be avoided by using randomized Quicksort.
Heapsort is one of the fastest sorting algorithms. The best and the worst cases for heapsort have same $O(n\log(n))$ performance.
Previously I wrote about gengetopt tool to parse command line arguments. There was a line about application version.
version "13.10.25"
It is important to keep this line up to date every time you compile and deploy binary.
Gengetopt is a tool for parsing command line arguments in C and C++ applications. It generates cmdline.c and cmdline.h files in pure C, which contain parser, helper functions, error handling and structure to store argument values. Arguments to be included in auto-generated file are defined in config. I usually gengetopt config file as cmdline.ggo. In this note I provide an example of gengetopt config file and arguments usage in code.
Assymetric encryption is useful if you want to encrypt data at remote server via script. In case you use bash script for symmetric encryption job, you have to store password inside this script. So, if remote server is compromized, you will loose both the encrypted data and the key to decrypt it. Assymetric encryption allows you not to store decryption key at remote machine. So, data will be in safe even if server is hacked.
I used to utilize gnuplot for creation of plots with arbitrary data. The main problem was to postprocess data before plotting: I had to integrate postprocess into calculation code itself or to write additional script to do so. I’d like to provide an example of good-looking plotting with Python and matplotlib. Python has bataries included, so there will be no problem with preparation of data to be plotted.
I’d like to show how to use HPC part written on C++ with CUDA in Python code. So, every heavy part may be done on GPU with CUDA, all gluing tasks (with beautiful matplotlib plots) are done on CPU with Python.
This Python script retrieves computer’s external IP address (using internet.yandex.ru) and emails message with new IP if IP has been changed since last check. BeautifulSoup module is needed, so install it if you have no installed version:
sudo pip install beautifulsoupI often work on my netbook, so I prefer to use Sublime Text with Makefiles instead of full-featured IDEs. To automate build process I’ve constructed (with help of Vasily Picard and examples from the Internet) universal Makefile. It assumes the following structure of files.
This note describes installation of Bittorrent Sync on Ubuntu. Tutorial was tested on Ubuntu 12.04, but it must work for all Debian-based systems. Read full btsync manual for advanced information.
It is a good tone to check CUDA API errors while calling cudaMalloc() and other functions. It also helps to find floating bugs caused by hardware (lack of memory, etc). I provide below an adapted version of CudaSafeCall I found many weeks ago in the Internet. Simply remove #define CUDA_ERROR_CHECK in production if unneeded.
To enable tun device in OpenVZ containers tun module - needs to be loaded in the host system (in my case CentOS 6.4) - containter (in my case Ubuntu Server 12.04) must be allowed to use tun
This howto is based on two sources: Official wiki and Amutu blog entry. In short, the main difficulty is in bridge network configuration: correct network configuration doesn’t work in CentOS 6.4 while NetworkManager is running. I compiled official documentation and fix found in Amutu blog here in one entry.
Depends on server configuration, each OpenVPN client needs configuration file (client.conf for *nix and client.ovpn for windows), certificate authority (ca.crt), [optional] tls auth file (ta.key), user crt and key.
It is possible to save configuration of ssh connection for later usage. For example, one had to type
ssh <username>@<ip address> -p <port> -i <path to key file>every time one wants to connect to server.
SLURM (Simple Linux Utility for Resource Management) — job scheduler and resource manager usually installed on supercomputers. For example, it runs on Lomonosov supercomputer in MSU, Moscow, Russia.
Default Drupal package in Ubuntu installs Apache, but I prefer nginx as a webserver. In this note I’ll describe steps necessary to get Drupal 7 work.
You can record your screen using only command line in Linux. Workflow will look like: Create a bunch of screenshots -> Crop screenshots -> Make a movie from images.
In i3 window manager by default Nautilus actually opens two windows: Desktop window and Nautilus itself. The former can’t be closed by the default shortcut $Mod+Shift+Q.
Pcap files are pocket capture files. They usually contain raw log of network connections and packets. It is the most popular format and it available in almost all network analysis tool.
du (disk usage) utility in linux bash summarizes disk usage of each directory or file. By default, it outputs only directories. Just enter some dir and type
$ du ./It’s quite an academic task, but anyway useful sometimes. I’ve collected different ways to do it in terminal in linux. Some of them work with UTF-8 characters (some it will toggle case for “й”, “ё” and so on. It will not in general handle special ligatures, such as “ß”” and “fi”.)
This howto is based on a great guide with modifications. All settings are made for server domain hg.kenarius.org. You can replace it with your one easily.
AWK is an acronym of first letters of its authors (Aho, Weinberger and Kernighan). It is a data-manipulating scripting language with huge possibilities. There are several implementations of it: awk is a canonical one, nawk (new awk), mawk (default in Ubuntu 12.04), gawk is GNU awk. I recommend latter one, because it works correct with unicode symbols in example:
$ echo юникод | gawk "{res = toupper(\$1); print res;}"
ЮНИКОДWe all do backups. This note is about doing them on Ubuntu 12.04.
Tasks:
I’ll save only backups for last five days.
After moving to i3 I’ve installed PCManFM as GUI file manager. To fix missing icons one have to define icon theme. These instructions are tested on Ubuntu 12.04, but it must work with other distros.
Glue several png’s into one png [1]+[2]+[3]+[4] = [[1][2][3][4]] with montage from ImageMagick:
montage -geometry +1+1 -tile 4x1 image*.png final.pngBlender is a free software for 3D graphics. In old virsions many operations were binded to hotkeys, so it was believed that Blender is very hard to master. Now it has more userfriendly interface with button in addition to hotkeys.