Blogging in the wind

What is QE?

Esteban Manchado and I begun working on this cartoon almost one year ago. Our intention was to make software quality engineering appealing to developers. We spent a good deal of time trying to structure it and discarded many, many ideas. The original goal seemed impossible to achieve in just one page, so we tried to explain what is software quality engineering. Hope you enjoy it! (Click on the image for the full version).

The comic was created using open source tools like Inskcape and has a CC BY 4.0 license. Free to share!

PS: Esteban Manchado wrote a post explaining the creative process behind this comic. Check it out! Making the Quality Engineering comic.

Using PyCuda to boost FITS image processing times

One of my hobbies is amateur astronomy. Years ago I bought a fairly good CCD and a transportable telescope to do some serious imaging, for scientific purposes. One key difference between photography and astro-imaging is that CCD cameras provide just raw images that need post-processing. From phones to DSRL cameras, those provide pretty images processed to look great. But to aim for scientific results, the crude numbers are needed.

Imagine that we are trying to discover an exoplanet orbiting in a distant star. What we want is the exact count of photons that come from that star, to measure its brightness with very high precision. Any error, and the eclipse produced by the exoplanet won't be detected. CCDs just do that, they provide the number of photons detected. But even that count has some noise, including unwanted deviations produced by dust, heat and the analog-to-digital converter. For that, special images are taken, called bias, dark frame and flat field. Those are combined with the raw image to provide a calibrated image that has scientific value.

All these image operations consume lot of computer power. Not because they are inherently difficult (in other manipulations, they can), but because images today tend to be large and usually many images are taken in an observing session. To process images, amateur astronomers use good programs like IRIS, Nebulosity or PixInsight, that offer both good GUI's and batch options. Astrophysicists more often use CLI environments, like IRAF. These tools are commonly programmed in C/C++, and are optimized.

However, even C/C++ are slow compared to new computing options. Nowadays, computers come with specialized number-crunching hardware also known as graphics cards. These cards can be used not only to display windows in a monitor and to play 3D games, but to do calculations. One key difference between the CPU of our computer and the GPU of our graphic card is that GPU's are designed to do parallel computation. And graphic card makers are increasingly providing good SDK and tools to do that.

Below I provide a very simple example on how to use PyFITS to read an astronomical image, calculate the log of each pixel, and to display the result. Python is a increasingly popular language among the astronomical community. PyFITS is a library to read FITS, the standard image format of the field (Hubble Space Telescope uses it!). And usually, when manipulating images with PyFITS, the fast NumPy library is also used, because we want C speed and not Python slowness. But, what if C is also the slower option?

The example below uses PyCuda, which is a Python wrapper to the CUDA tools. These tools are provided by NVIDIA, the company that makes the popular graphics cards. As can be seen in the code, we provide a function in a C-like language as a string, with is then interpreted by CUDA and cached. We then pass the image array, which is transferred from the computer memory to the graphics card memory. And then, magic happens, because the GPU modifies each pixel value in parallel, much faster than using the CPU. Finally, the result is returned to the computer memory and the log10 image is available to display it.

#!/usr/bin/python3

import astropy.io.fits as pyfits   # Library to read FITS
import pycuda.autoinit             # PyCuda autoinit
import pycuda.driver as cuda       # PyCuda In, Out helpers
import matplotlib.pyplot as plot   # Library to plot
import matplotlib.cm as colormap   # Library to plot
import numpy                       # Fast math library
import time

from pycuda.compiler import SourceModule

# Open FITS image
fits = pyfits.open('gsc-2108-1564-750mm-020s_001_c_r.fit')
# Get the image data as array
data = fits[0].data.astype(numpy.float32)
# Read image metadata
header = fits[0].header
# Show all metadata
print(header.keys)

# PyCuda: Create kernel
kernel = SourceModule("""
__global__ void log_fast(float *result, float *source) {
    const int i = blockIdx.x * blockDim.x + threadIdx.x;
    result[i] = log10(source[i]);
}
""")

# PyCuda: Get function
log_fast = kernel.get_function("log_fast")

# PyCuda: Calculate block size
BLOCK_SIZE = 1024
block = (BLOCK_SIZE, 1, 1)
grid = (int(data.shape[0] * data.shape[1] / BLOCK_SIZE) + 1, 1, 1)

# PyCuda: Call the function to transform image
data_result = numpy.zeros_like(data)
start_time = time.time()
log_fast(cuda.Out(data_result), cuda.In(data), block=block, grid=grid)
end_time = time.time()
cuda_time = end_time - start_time
print("PyCuda log10 {0:.4f} seconds".format(cuda_time))

# NumPy
start_time = time.time()
np_data_result = numpy.log10(data)
end_time = time.time()
numpy_time = end_time - start_time
print("NumPy log10 {0:.4f} seconds ".format(numpy_time))

print("PyCuda is {0:.1f} times faster than NumPy".format(numpy_time / cuda_time))

# Show image using matplotlib
figure = plot.figure()
figure.add_subplot(3,1,1)
plot.imshow(data, cmap = colormap.Greys_r)
figure.add_subplot(3,1,2)
plot.imshow(data_result, cmap = colormap.Greys_r)
figure.add_subplot(3,1,3)
plot.imshow(np_data_result, cmap = colormap.Greys_r)
plot.show()

 

Of course, the question is, how much faster is CUDA than NumPy?

PyCuda log10 0.0243 seconds
NumPy log10 0.0402 seconds
PyCuda is 1.7 times faster than NumPy

Almost 2 times faster. Amazing, isn't it? In your case, it will depend on the calculations that you want to do. The difference can be much higher.

So, if your computer has a NVIDIA graphics card, take a look to PyCuda. It will save lots of time. For other graphics, take a look to OpenCL and alternative to PyCuda which is vendor-neutral.

And finally, another processing that I've done using with PyCuda to average images of the galaxy NGC 253 that I took a couple of years ago. The goal is to increase the signal-to-noise ratio by averaging many images.

Visualino: How to measure voltage with Arduino

Inspired by  Crea un voltímetro tan solo con tu placa Arduino y un par de cables (How to create a multimeter with an Arduino board and a couple of wires), I've created the following tutorial to show how easy and fun is to program with Visualino. Here you'll learn how to measure the voltage a battery with Arduino. For that, a multimeter is often used, but multimeters aren't smart. Arduinos are!

Arduino boards have two set of pins. The digital pins are the ones we usually use to blink a LED. Digital means they only have two states: ON and OFF. But we also have the analog pins, which are able to measure current and convert it from a voltage to a number, that can be read. To build the circuit these components are needed:

• Arduino Uno or Nano (or any other one).
• Two resistors 1K Ohms.
• Some wires.
• A battery: in my case 9V.

Next, place the components like this:

Now, we are ready to program the Arduino board using Visualino. You can see how to do that in the following video. First, we use the blocks to create the program. The program do these:

1. It reads a number from the analog pin #0 and stores it in the "read" variable.
2. It re-scales that number from 0-1023 to 0-900 and stores the result in "voltage". My battery is 9V. If your battery has max 3 volts, then use the appropriate max value (e.g. 300). 
3. The measured voltage is printed.
4. After half a second, the measure is repeated.

After the program has been created, we build it and transfer it to the Arduino board.

As we can see in the video, Visualino has a Serial monitor. There we can see the voltage readings. That was easy, uh? :) But to make things easier, Visualino is able to convert those numbers in the Serial monitor to neat real-time chart! 

 

And that's it! Stay tuned as other awesome features will be coming soon to Visualino. Enjoy!

Visualino: Visual programming for Arduino

To celebrate the 10th anniversary of Arduino, and the Arduino Day, today I am proud to present Visualino. What is it? It's a visual programming environment for Arduino, a project that I begun last year and has been actively developing in the last months, with the help of my friends at Arduino Gran Canaria.

Arduino is a microcontroller board that allows to connect to sensors and other electronic components. It has a companion program called the Arduino IDE, which makes really easy to program the microcontroller. The language is based in C/C++ but the functions are quite easy to learn. This easiness is part of the revolution. Making LEDs blink and moving robots with Arduino is easy and fun. But it can be easier! Kids and adults who don't know programming often struggle with C/C++ coding strictness: commas and brackets must be correctly placed, or the program won't run. How to make it even more intuitive? Visual programming to the rescue!

Scratch is a popular visual programming environment for kits, developed at MIT. Instead of keyboards and codes, kids use the mouse and blocks to create games like a puzzle. And there is an extension called Scratch for Arduino that allows to control the board from Scratch. However, the program runs in Scratch, so the Arduino board must be always connected to the PC.

So, what does Visualino do? It's a Scratch-like program: it allows to create programs for Arduino like a puzzle. But it directly programs the Arduino board, and the PC connection is no longer needed for it to run. Also it generates the code in real time, so the user knows what's happening. The environment is very similar to Arduino IDE, with the same main options: Verify, Build, Save, Load and Monitor. Visualino can be seen at work in this screencast:

Visualino is based in Google Blockly and bq's bitbloqs. It is open source, multiplatform and multilanguage. It just requires Arduino 1.6, which is the actual engine used to program Arduino boards. You can download the beta version right now for Ubuntu, Mac and Windows. The code is available at github.com/vrruiz/visualino. Right now it works out of the box. It needs some documentation and translations to Catalan, Italian and Portuguese will be welcomed. 

• 
• 

Visualino was presented this week to a group of educators at an Arduino Workshop, and next month, we'll have a three-hour session to teach how to use it. So I hope it will be used soon at schools here at home.

So, go to download and use it. Feedback is welcome. And stay tuned, as there are some niceties coming very soon :)

How to migrate your contacts from iPhone to Ubuntu Touch

In the last years I've been using an iPhone and I recently wanted to transfer the contact list to my phone with Ubuntu Touch. These are the steps I followed.

What is required:

• iPhone.
• Mac OSX.
• Ubuntu Desktop.
• Ubuntu Touch device.

How to do it:

1. Export Contacts from the iPhone.
2. Transfer the contact file to Ubuntu Touch.
3. Import the contact file.

Let's go step by step.

Exporting the Contact list

There are different ways to export the Contact list from the iPhone to elsewhere. One option is to install an application like My Contacts Backup, and then to transfer the file containing the contacts to the computer. This option is preferred because avoids the need of a Mac desktop.

In my case, I already had my iPhone contacts synchronized with iCloud (option that I don't remember to have activated) and had the MacBook Air at hand. So these are the steps that I followed. First, the iCloud account must be setup to synchronize Contacts. Open Contacts in OSX, go to application menu and press to Accounts...

Then select iCloud and introduce your account credentials.

Finally, enable Contacts synchronization.

After sync is done, the contacts are available in the application. Next step is to export all the list to a vCard file. To do that, select all contacts.

And export them.

Give a name to the file, and we are done in OSX.

The /Users/<user>/Documents/phone-contacts.vcf is a text file in vCard format that contains all the details of our contacts. Use a USB key to copy that file and copy it to an Ubuntu desktop.

Re-formatting

Once the file is in Ubuntu desktop, we need to tweak it a little bit, since the import tool in Ubuntu Touch doesn't like the vcf file as exported directly by OSX. The problem is that each contact entry in the file is not separated by a new line, so it must be added afterward. Open a Terminal and type this command:

In text:

$ cat phone-contacts.vcf | sed -e "s/END:VCARD/END:VCARD\n/" > phone-contacts-touch.vcf

A new (and correct) file will be created, phone-contacts-touch.vcf.

Transfer to Ubuntu Touch

Now connect the Ubuntu Touch device to Ubuntu desktop and install some packages:

$ sudo apt-get install phablet-tools android-tools-adb

In the Ubuntu Touch phone, go to System Settings > About this phone > Developer Mode and  enable the developer mode. Connect the phone to Ubuntu via USB and type this command to transfer the file to Ubuntu Touch:

$ adb push phone-contacts-touch.vcf /home/phablet/Documents

And log into Ubuntu Touch:

$ phablet-shell

Importing the contacts

And now, the final step that will import the contacts. That will be done using SyncEvolution tool that comes by default in Ubuntu Touch.

$ syncevolution --import ~/Documents/phone-contacts-touch.vcf backend=evolution-contacts database=Personal

And that's it. Now, all the contacts are imported and available in the Address Book app in Ubuntu Touch. Enjoy!

Wrap up

Of course, with some development effort this task could be much easier. A path that I don't know whether it is possible are direct calls to iCloud API's from Ubuntu Touch, but I doubt it. I've took a quick look to  libimobiledevice library. Many years ago there was a tool, python-idevicesync, that was based on it and provided a contact export feature. It's now out of sync with the library API. Another good step would be an Ubuntu Touch app that could read vCard files, and avoid the need to call SyncEvolution using the command line. I'm sure the community will be looking into this features soon.

Mobile and Free Software: The System of the World

For Mobile World Capital's blog, I was asked to write an article about the impact of free software in the history of mobile. This is it:

«Today Apple and Google dominate the market of mobile operating systems. In a few years, these companies have gained privileged positions in the telecommunications market, overshadowing the terminal manufacturers. Motorola Mobility is now in the hands of Google. But, ironically, the computer industry owes much to the phone industry. Unintentionally, AT&T created the operating system that is today ruling the world, from mobile phones to supercomputers. Or, if you will, the key programming language of the last 40 years. While doing so, they also planted the seed of the free software».

Continue at  Mobile and open source: how Bell Labs (AT&T) created the system of the world.

PS: This is the English translation, it's also available in Catalan and the original, in Spanish.

Migrating a Linux system to a SSD on a Mac

In my daily work at Canonical I use VM's quite often to test Ubuntu Web Apps and new browser releases in different environments. I use a MacBook Pro 15" (mid 2012) as the main computer, currently running Ubuntu 13.04. This computer had rEFIt to boot and the default OSX system. Of the 500 GB, just 75 were dedicated to Linux, so I was forced to delete VM's or backup them in an external hard disk to re-use them. Finally, I decided to buy a SSD, which are quite cheap nowadays.

The SSD I bought was a Samsung 840 250 GB (not Pro version, which has some additional features). It costs 179 €.

This are the steps I followed to move my Ubuntu setup from a HD to a SSD.

1. Burn a DVD with Ubuntu. I reused an old 12.04 disk, the -mac version.
2. Boot Ubuntu from the DVD.
3. Attach an external USB drive. This is used to (backup and) copy the partition from the HD to the SSD.
4. Run gparted as root to copy the Linux partition (in my case, an ext4).

To copy the partition, I resized the USB drive partitions to live room to the HD's Linux partition, which is 75 GB. Then, in gparted I selected the partition from the HD, selected "Copy" from the partition menu and then "Paste" it in the spare space in the USB drive. This step took +40 minutes. At this point, the Linux partition is available in the external disk.

After that, I switched off the computer and replaced the HD with the SSD. Follow the link to see how. Now, the second part: to move and setup the system to the solid state disk. The SSD disk was blank, so it needed proper configuration.

1. Boot Ubuntu from the DVD.
2. Attach the external USB drive.
3. Run gparted.
4. Create a partition table in the SSD. I used GUID Partition Table (gpt) format, the one the original HD uses.
5. Partition the SDD.
   • Create an EFI partition. The first partition has FAT32 format, 200 MB in size, "EFI" as label and "grub_boot" flag. 
   • Create the swap partition. At the end of the disk, I created a 10 GB "linux-swap" partition.
   • The rest of the disk will be available for the main Linux partition.
6. Copy the Linux partition to the SSD. In gparted, select the Linux partition in the USB drive, copy and paste it in the SSD. This takes +45 minutes.
7. Resize the Linux partition to fill the entire disk and flag it as "boot".

Congratulations! The Linux partition is now copied bit-by-bit in the SSD. However, it cannot boot. The reasons are: rEFIt (the bootloader) is not installed; the Linux partitions  are not properly configured. To do that, we need to modify the file /etc/fstab in the Linux partition (SSD).

And this is the final third step: setup the system to properly boot. In my SSD, /dev/sda1 is the EFI partition, /dev/sda2 the Linux partition and /dev/sda3 the swap partition. You may have different setup. /etc/fstab must be changed to reflect this addresses. From a terminal:

$ sudo mkdir /media/root 

$ sudo mount /dev/sda2 /media/root

The Linux partition is accesible in the directory /media/root/. The file /etc/fstab/ of the Linux partition can be edited now at /media/root/etc/fstab/

$ gksu gedit /media/root/etc/fstab

This is the original content:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda5 during installation
UUID=24acabd4-2fcb-49aa-9fb3-ce9e657d4465 / ext4 errors=remount-ro,user_xattr 0 1
# swap was on /dev/sda7 during installation
UUID=c40532ab-5716-47bc-9b95-3672b834c6a2 none swap sw 0 0

Here, the modifications:

# / was on /dev/sda5 during installation
/dev/sda2 / ext4 discard,errors=remount-ro,user_xattr 0 1
# swap was on /dev/sda7 during installation
/dev/sda3 none swap sw 0 0

The blkid command can be run as root to find out the UUID strings of the devices and use them instead.

Then, I downloaded the compiled version of rEFInd, which is a fork of the rEFIt bootloader and able to run from Linux, Mac and Windows.

$ cd /media/root/root/

$ sudo wget http://downloads.sourceforge.net/project/refind/0.6.11/refind-bin-0.6.11.zip

This is almost done. We now "log" into the Linux partition (SSD) to apply the changes and setup the bootloaders. In order to do that successfully, the /proc and /dev from the live DVD are mounted to the Linux partition and the EFI partition (this is needed by rEFInd).

$ sudo mkdir /media/root/boot/efi

$ sudo mount -B /proc /media/root/proc

$ sudo mount -B /dev /media/root/dev

$ sudo mount /dev/sda1 /media/root/boot/efi

$ sudo chroot /media/root/

Now we're "logged" in the Linux partition as the root user. This is when GRUB and rEFInd are installed in the SSD to be able to boot Linux from the Mac.

# mount -t sysfs sysfs /sys

# cd /root/

# unzip refind-bin-0.6.11.zip

# cd refind-bin-0.6.11

# ./install-bin.sh --esp --alldrivers

# grub-install /dev/sda

# update-grub

# exit

And that's all. The system should be able to boot the Linux partition from the SSD.

Some caveats and open questions:

• I installed rEFInd first, without GRUB. rEFInd and the system wasn't able to boot.
• For some reasons, rEFInd in my system is much slower than rEFIt. It takes around 40 seconds to show up.
• After installing GRUB, I'm not able to mount the EFI partition. Now has an unkown partition format.
• Does GRUB really needs rEFInd?

References

• Ask Ubuntu: How to move Ubuntu to an SSD.
• Move your Linux installation to a new Solid State Drive - even a smaller one.

Happy 32,013!

Look carefully at the picture. More than 1,000 generations separate us from that hand. This person lived 30,000 years ago in southern France. The prehistoric painters of Chauvet Cave transport us to a distant and different time. The ice glaciers covered Europe and Neanderthals were close to extinction. Represented on the walls of the cave are other species of large mammals, also extinct. In light of the embers, skillfull hands like that drew rhinos in combat, megaloceros, cave bears, aurochs and mammoths. A landslide sealed and preserved this cave in pristine condition, until its recent discovery.

Look at the palm of your hand. Look at the picture again. It is a human hand. Troglodytes are often portraited as a crude and inferior species. But when we watch what they achieved, this prejudice is impossible to sustain. Despite the time that separates us, the drawings demonstrate great expertise, imagination and abstraction, and an artistic quality recognizeable as distinctly human. Anyone, from any culture, is able to see himself reflected in the drawings. We are descendants of these cave painters. Genetically, a baby born today would be almost identical to a baby born at that time. What differentiates us is not in the genes.

The world has changed since then. Glaciers have retreated from Europe. The climate is warmer. Animals and plants were domesticated. Hands like those created bricks. With bricks, towns were build. In towns, organized societies fluorished. Social complexity exceeded individual capacity. To account for large quantities became a need. And art gave way to knowledge: 5,500 years ago, marks were made on clay tablets to represent numbers. From there, and in a few hundred years, writing was invented. Thanks to the written symbols, accurate information could be transmitted to the future. And this is what made us modern. Our civilization is supported by the slow accumulation of written knowledge. How to make a building. How to cure a disease. How the universe began. The knowledge to be meaningful, to be valuable, must be shared.

Now put your hand over the image. Think about what you share with her. The length of the fingers. The chin. The sense of warmth. And now think all the possibilities that yours has, that that hand never had. To play the piano. To solve an equation. To write a poem.

My wishes for 2013: Learn. Teach. Create. Share. Transcend. Be human.

HOWTO: Upgrade the BIOS in Aspire One (Linux)

Transoceanic flights can be boring, even with an iPad to watch movies and play Angry Birds. In my last trip to Boston for a meeting at Canonical's offices, I missed a lot a small computer to play with in the airplane. The 15 inches of the MacBook Pro are good to be productive, but not suitable for the small airline seat. I own a somewhat outdated 9" Acer Aspire One, one of the first netbooks to be released. Unfortunately, it was not working properly.

I bought this Aspire One AOA110 near three years ago. I used it primarly as a portable computer and at hme as the device of choice to surf and read documentation (then, it came the iPad). Some months ago I upgraded the aging operating system, the rpm-based Linux distro called Linpus with Ubuntu Lucid. However, I discovered an issue with the battery: it won't work. The netbook could only be used with with the power cord. I thought it was a hopeless case.

However, it seems that Aspire One battery issue is a common and known problem, which only requires a BIOS update. Acer provides the firmware update in its site, but the programs to execute are for DOS/Windows. Here are the instructions to upgrade it using Linux:

1. Download an up-to-date BIOS firmware.
2. Uncompress the downloaded ZIP file. 
3. Download an ISO image of FreeDOS, an open source version of MS-DOS. I recommend this FreeDOS 1.1 image.
4. Burn the ISO image to a USB key (beware to backup the USB key contents, burning hte ISO image will delete its contents). I used "dmesg" to find the USB key device (/dev/sdb) and then typed the command below (again, be careful to identify the correct /dev/ or you can destroy your data).
   
   $ sudo dd if=FreeDOS-1.1-USB-Boot.img of=/dev/sdb
   
   
5. Unplug and plug again the USB key, now it will be mounted.
6. Reboot and you're done.

This way, I rescued the netbook. Of course, it feels slow compared to more recent computers, but its days may not be over yet :)

The life of links

Fernando Tricas always has interesting things to say. In a recent post he talks about The life of links and digital content (Spanish):

«We tend to assume that digital [content] is forever. But anyone who accumulates enough information also knows that sometimes its difficult to find it, in other cases it breaks and, of course, there is a non-zero probability that things go wrong when hosted by third-party services. It is an old topic here, remember Will we have all this information in the future? . The topic resurfaces as news in the light of Currently charged by the article that can be read at A Year After the Egyptian Revolution, 10% of Its Social Media Documentation Is Already Gone».

In the comments, Anónima said: «Given a time t and an interval Δt, the larger Δt, the more likely is that all information in a time t-Δt you want to find is gone». This sounded like an statement to check, Thus, I decided to do an experiment with del.icio.us' bookmarks.

In delicious.com/rvr I have archived around 4000 links from 2004. So, I downloaded the backup file, an HTML file with all links and metadata (date, title, tags). I developed a python script to process this file: go through the links and save its current status (whether the link is alive or not). With another script, the status were processed to generate the statistics. These are the results:

As can be seen, there is a correlation between the age of the links and the probability of being dead. For the 10% who cited the Egyptian revolution, in the case of my delicious, we must go back three years ago (2009). But at 6 years from now, a quarter of the links are now defunct. Of course, the sample is very small shouldn't be representative. It would be interesting to compare it with other accounts and to extend the time span: How many links are still alive after 10 or 15 years? Is it the same with information stored in other media? Are all this death links resting in peace in a forgotten Google's cache disk?

I imagine that sometime in the future, librarians will begin to worry not only to digitize remote past documents, but also to preserve those of the present.

In case you are interested, the code to generate such data is available at github.com/vrruiz/delicious-death-links. The spreadsheet is also available in Google Docs .