Passing an external variable to AWK

9 02 2010

Confronted with a heap of colon separated text files which had to be merged and cleaned of unrelated lines and columns, i tryed my luck inside Excel and spend a lot of time doing it manually, but finally got fed up.

So I decided to use AWK on the task.

A FOR-loop lists the files in the folder into the UNIX pipe.

AWK selects the non-empty observations and adds the name of the file as a classifier to the beginning of the line (the result is a repeated measure dataset).

This is the code:

for CSV in `ls`
do
cat $CSV | awk -F ";" '{
if ($2 ~ /[0-9]+/) {print CSV , FS , $0;}}'
done

Remark: -F ";" option specifies how to distinguish the columns/fields of the lines/records in the file(default is ” ” or empty space).

BUT: The variable CSV gets not passed to AWK by default it has to be fed into AWK.

Solution:
The
-v CATEGORY=$CSV
option feeds the external variable CSV into the AWK-variable CATEGORY.

This gives:
for CSV in `ls`
do
cat $CSV | awk -F ";" -v CATEGORY=$CSV '{
if ($2 ~ /[0-9]+/) {print CATEGORY , FS , $0;}}'
done
.. and works :)

Hat tip:fpmurphy


Comments : Leave a Comment »
Tags: , ,
Categories : Ubuntu

HowTo: Download YouTube videos (the easy way)

25 01 2010

I asked and googled this questions a lot of times. I even used time on awkward Python and even C scripts to extract the download location from the html/php code of the youtube webpage. It turns out to be WAAAY easier and I remember doing this ages ago with Netscape and some flashmovies somewhere (Youtube, even Google did not exist):

I am just quoting FRuMMaGe on Ubuntuforums

Many people have been asking how to download youtube videos, and kept being pointed to useless software which will do it for them.

Here is the easy way:

Step 1: Find the video you want to download and wait for it to completely finish loading.

Step 2: Minimize your browser (do NOT close it, some browsers delete the temp directory upon closing), and navigate to the directory /tmp

Step 3: Drag the flash file onto your desktop. Voila!

This definitely works with firefox, and probably most other browsers aswell. Enjoy!


Comments : 1 Comment »
Tags: ,
Categories : Ubuntu, misc

Xubuntu Desktop Disappeared

21 01 2010

The other day my desktop disappeared in Xubuntu Karmic (9.10). First I did not notice since the background was still visible, but no icons showed up (which I rarely use, appart from removable media) and the very useful middleclick showing all processes on all desktops did not work anymore.

Took quite some googling, but Ubuntuforums had the answer:

Alt-F2 to open the application starter and
xfdesktop
started the desktop process again…

The desktop process must have failed the other day and after exiting the session this configuration was safed… even restarting did not change it.


Comments : Leave a Comment »
Tags: , ,
Categories : Ubuntu

Xrandr – Manipulating Monitor Setting via Command Line

11 11 2009

Usually Ubuntu and all of its derivatives would recognize an external monitor and set it up reasonable automatically. At least all the flavours I tried of Ubuntu, Xubuntu, Linux Mint did that – with Linux Mint most ellegantly – in my opinion.

I read some warnings that one should be prepared for “regressions” in connection with some development issues in Karmic Koala.

So:

Second (external) display does not work as used to anymore. After Xubuntu Karmic Koala fresh install I found just “Display1″ in the Display settings. I could set it to the (higher) resolution 1280×1024 of the external display, but then my Lenovo X61s laptop display (1024×768) would turn black.

A workaround was not so hard though:
xrandr
in the command line showed that both the laptop display (“LVDS1″) and the external display (“VGA1″) were detected and running; just that the external display was running at laptop resolution too.

Xrandr did even detect the optimal settings for the external display.

Thus
xrandr --output VGA1 --auto --above LVDS1
set the resolution and refresh rate of the external monitor correctly and put it as an extended Desktop “above” the laptop screen. Exactly the way I did like it in Linux Mint 7 before…

Xrandr is commandline interface “to set the size, orientation and/or reflection of the outputs for a screen” (from the man page).

In Ubuntu
man xrandr gives a short description with examples. I derived my workaround from one of the examples given.


Comments : Leave a Comment »
Tags: , , , ,
Categories : Ubuntu

Managing Amazon S3 Online Storage with S3sync

1 11 2009

After trying to use Amazon’s S3 web service to backup files and to get a reliable download area for R functions and stuff which is not allowed to be uploaded to wordpress.com I ended up with some experimental “buckets” (= S3 online directory) and some 100 MB of files in them.

It turned out that it is not possible to delete a non-empty bucket from S3, so one is to required to recurse into the directories and delete all files one by one!

Eric Cheng and other blogs appearing after a google search pointed out S3sync as a suitable tool to remove a non-empty bucket.

So first one has to get Ruby and then also the OpenSSL interface for Ruby: sudo aptitude install ruby libopenssl-ruby

Then download s3sync (to your /home/yourself folder in this case) and unpack it: cd $HOME/
wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz tar xvzf s3sync.tar.gz
rm s3sync.tar.gz
This creates a s3sync folder containing the ruby code.

The package ca-certificates includes PEM files of CA certificates to allow SSL-based applications to check for the authenticity of SSL connections. It is needed to have the S3 connection secure via SSL and part of the default Ubuntu installation (at least included in my Xubuntu Karmic Koala. If not: sudo aptitude install ca-certificates

Before using s3sync get your access key and secret access key from Amazon. It has to be included in a file “s3config.yml” which is located in your home folder inside the directory “.s3conf” which has to be created. So:

mkdir $HOME/.s3conf
to create the directory.

Open your favorite text editor and create a plain textfile called s3config.yml inside the “.s3conf” folder which contains: aws_access_key_id: YourS3AccessKeyFromAmazon
aws_secret_access_key: YourS3SecretAccessKeyFromAmazon
SSL_CERT_DIR: /etc/ssl/certs

Prevent others from reading the configuration file containing your confidential access codes by chmod 700 $HOME/.s3conf/s3config.yml

Now you can start to use s3sync and s3cmd to manipulate your S3 storage space with e.g.: ruby $HOME/s3cmd.rb listbuckets

This was the first time I managed to manipulate successfully my S3 account. Ok, Djungledisk under Mac OS-X worked, but it is proprietary, though not expensive.

John Eberly’s blog was an inspiration to get started. Follow the link to his excellent blog post.


Comments : 1 Comment »
Tags: , , , ,
Categories : Backup, Ubuntu

Mount a CD Image on Harddisk

24 10 2009

A copy of a CD containing e.g. installation software or a game (so called iso image) can be mounted directly from the harddrive. This might be useful to mimic a CD without inserting the CD into the (very slow and power consuming) CD/DVD-drive.

1. Create a mountpoint e.g. in the /media directory:
sudo mkdir /media/iso

2. Mount the image MyCDimage.iso as a CD:
sudo mount -t iso9660 -o loop MyCDimage.iso /media/iso

3. Display all filesystems to check its mounted:
df -h

In order to play e.g. Windows games with wine the mountpoint /media/iso has to be added as a drive under wine.
winecfg
choose the drive tab and add /media/iso


Comments : Leave a Comment »
Tags: , , ,
Categories : Ubuntu

Mount NTFS-partition under Ubuntu

23 09 2009
  1. Open up Terminal (Applications – Accessories – Terminal) and type the following:
    sudo fdisk -l

    you will be asked for the root password (password you chose at setup) and will be presented with a list that looks similar to this:

    Disk /dev/sdb: 320.0 GB, 320072933376 bytes
    255 heads, 63 sectors/track, 38913 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes
    Disk identifier: 0x000e4abf

    Device Boot Start End Blocks Id System /dev/sdb2 * 1 38913 312568641 7 HPFS/NTFS

    this tells us that the drive is 320GB and is located at /dev/sdb2, you will need to remember the location of the drive.

  2. In terminal again, type
    sudo gedit /etc/fstab

    You will be presented with your fstab file, which basically tells Ubuntu where to mount the drives listed.

  3. At the bottom of the fstab file paste the following:
    /dev/sdb2 /media/wimpy ntfs-3g defaults 0 0

    You will need to adjust 2 things: the name /dev/sdb2 to the drive that is specific to your computer, and the name /media/wimpy. The name I chose “wimpy” the directory where the drive will be mounted. Save the file

  4. to create the mount point, open Terminal and type:
    sudo mkdir /media/wimpy

    where “wimpy” matches the name you used in the fstab file.

  5. To test that it works, type:
    sudo mount -a

    This will mount all the drives listed in the fstab file.


Comments : 2 Comments »
Tags: , ,
Categories : Ubuntu

Slow Filetransfer under Linux

1 09 2009

After all Bill Gates has also an issue to prey on: Linux seems to be quite unefficient in transfering files from one partition (or harddisk or USB-disk) to another.

I asked for an explanation on the Linux Mint Forum.

‘optimize me’ came up with an interessting answer, which I just quote:

There is some long standing problems with how file transfers work. I never had a problem with it until kernel 2.6.24-19 came out for Ubuntu Hardy Heron back in June of 2008. At first, all I noticed is that file transfers to USB devices (flash drives, external HDDs) started moving slower than molasses in winter. CPU usage stayed below 15% or so, but transfers never went faster than 5MB/s and the speed always degraded to tens of K/s.

Research I did shows that this has been a problem on various distros going as far back as 2005. Over time, I’ve discovered that this bug is not limited to just moving files to USB devices, but also affects moving files between partitions on the same hard disk, and also affects network file transfers (both SMB & NFS).

If you look at your system monitor in the panel (I assume that’s what you’re using), and change the color scheme of your CPU monitor(s) to highly contrasting colors (Red, Blue, Yellow & Green, for example), you’ll see that what’s termed as “I/O Wait Time” is what’s eating up all your power. I’ve been up and down a million forums, been in contact with kernel and module developers, and spent countless hours researching the problem.

I’ve got zilch.

All I can tell you is that the problem doesn’t effect everyone – only, it seems, a small minority – so it’s not at all a priority for the developers to fix. You can do a google search for “slow+usb+linux” or “slow+usb+ubuntu” and you’ll probably find all my forums posts and all the same info I found. I’m not a programmer, I’m not a developer, and I don’t know jack about where to even begin tackling this problem. That’s where that stands.

I inserted some linebreaks for readability. Thanks a lot ‘optimize me’.


Comments : Leave a Comment »
Tags: ,
Categories : Ubuntu

No Sound on HP EliteBook with Ubuntu Jaunty

22 08 2009

Sound did not work on a HP EliteBook 2730p. It seemed the sound did not reach the speakers. I found the solution at linlap.com for the EliteBook 2530p and some Linux distribution. It worked on the 2730p as well:

Open alsa-base.conf with any texteditor (here ‘gedit’) by typing
sudo gedit /etc/modprobe.d/alsa-base.conf
in the terminal.

Then add a line
options snd-hda-intel enable_msi=1 single_cmd=1 model=laptop
at the end of the file.

Save it and restart the computer.

PS: This is one-of-those-Newbee-with-Linux-problems which get solved by semi-competent replication of some geeky fiddling with foo.conf files in /etc or /boot or /bar folders … I have no Idea what alsa-base.conf is and why/how this works. I just found the fix with google at the site cited above and it works…


Comments : Leave a Comment »
Tags: , , , , ,
Categories : Ubuntu

Cannot find sprng 2.0 header file

10 08 2009

Trying to
install.packages("rsprng")
fails because
Cannot find sprng 2.0 header file.

The development package of the SPRNG (Scalable Parallel Random Number Generator) library which provides several RNGs that are suitable for use in parallel computing are missing:

sudo aptitude install libsprng2-dev

fixes it.


Comments : Leave a Comment »
Tags: , , ,
Categories : Ubuntu, Use R!

« Previous Entries